Federal medicine multi-modal Hash retrieval method, system and equipment based on prototype and medium

Abstract

The invention discloses a prototype-based federated medicine multi-modal Hash retrieval method, system and equipment and a medium, belongs to the technical field of federated learning and multi-modal Hash retrieval, and aims to solve the technical problem of how to effectively guarantee the privacy and security of medical data in a multi-modal Hash retrieval process. The technical scheme adopted by the invention is as follows: the method comprises the following steps of: constructing a medical multi-modal data set; constructing a federal medicine multi-modal Hash retrieval model based on the prototype: constructing the federal medicine multi-modal Hash retrieval model based on the prototype by utilizing a neural network and a deep learning technology; model training: training a federal medicine multi-modal Hash retrieval model based on a prototype by utilizing the final data set; the method for constructing the federal medicine multi-modal Hash retrieval model based on the prototype comprises the following specific steps: performing medical multi-modal fusion; carrying out prototype parameter reinforcement learning; a hash retrieve query.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning and multimodal hash retrieval, and in particular to a prototype-based federated medical multimodal hash retrieval method, system, device and medium. Background Art

[0002] With the continuous increase of various data types such as images, texts and audios, information retrieval technology is no longer limited to a single mode. Therefore, in the face of massive multimodal data, improving retrieval speed and reducing data storage consumption have become two important challenges that need to be solved in the current information retrieval field. Through hashing technology, high-dimensional multimodal data can be encoded into hash codes, which significantly reduces the consumption of storage space and speeds up the retrieval process while retaining the semantic similarity between different modal data. Therefore, hash retrieval has become a key technology to support large-scale information retrieval. In addition, considering the inherent low-level representation differences and high-level semantic consistency characteristics of multimodal data, that is, the same object can be presented in multiple ways and forms of expression, this diverse form of expression can characterize the same data instance from different angles, make up for the possible information loss of each modality, thereby promoting each other and enriching people's understanding of modal semantic information. Therefore, multimodal retrieval must be able to cross the semantic barriers between different modal data to achieve accurate and efficient retrieval.

[0003] Different from the traditional centralized processing method, federated learning has attracted widespread attention in recent years as an emerging distributed machine learning paradigm. Federated learning allows model collaboration by training the model on the local device and sharing only the model parameters without directly sharing the original data. This method has significant advantages in protecting user privacy and data security, and is especially suitable for multimodal data retrieval scenarios. In actual scenarios, multimodal data sets are often highly sensitive and multi-source heterogeneous, which makes it difficult to centrally apply multimodal data in practical applications. For example, there are a large number of highly sensitive diagnostic reports in hospitals that are distributed in different departments and are difficult to use centrally. Therefore, federated hash retrieval can achieve efficient retrieval while ensuring the privacy and security of multimodal data.

[0004] In recent years, federated learning architecture models have been applied to hash retrieval systems, but they are all federated cross-modal hash retrieval. Cross-modal hash retrieval aims to query modal instances with semantically related modalities from one modality through a series of hash functions. In contrast, multimodal hashing uses the complementary and consistent correlations between multimodal data to learn hash codes for retrieval, which is more practical. In addition, the existing federated hash retrieval tasks ignore the imbalance problem of class imbalance in client data.

[0005] Therefore, how to effectively protect the privacy and security of medical data in the process of multimodal hash retrieval and avoid defects caused by data imbalance is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The technical task of the present invention is to provide a prototype-based federated medical multimodal hash retrieval method, system, device and medium to solve the problem of how to effectively protect the privacy and security of medical data in the multimodal hash retrieval process and avoid the imbalance defects caused by data imbalance.

[0007] The technical task of the present invention is achieved in the following way: a prototype-based federated medical multimodal hash retrieval method, the method is as follows:

[0008] Construct a medical multimodal dataset: obtain image modality and text modality datasets, standardize the image modality dataset, standardize the medical terminology of the text modality dataset, and add the patient's clinical background information to enrich the context preprocessing operation, then use the pre-trained fine-tuning model CLIP and the pre-trained fine-tuning model GPT2 to extract deep features from the image modality and text modality datasets respectively, obtain deep image features and deep text features, and divide the dataset evenly according to category and sample number to construct the final dataset containing training set, test set and retrieval set;

[0009] Construct a prototype-based federated medical multimodal hash retrieval model: Use neural networks and deep learning technology to construct a prototype-based federated medical multimodal hash retrieval model;

[0010] Training model: Use the final dataset to train the prototype-based federated medical multimodal hash retrieval model;

[0011] The construction of a prototype-based federated medical multimodal hash retrieval model is as follows:

[0012] Medical multimodal fusion: After feature alignment, deep image features and deep text features are concatenated and fused to generate a feature fusion matrix, and the hash code is output through the hash mapping part;

[0013] Prototype parameter reinforcement learning: The feature fusion matrix in the client is used to generate a personalized prototype through prototype calculation. The server collects the personalized prototypes of each client to generate a centralized prototype, identifies and fills in the category prototypes missing in the client-specific prototype, and ensures that the personalized prototype generated by the client sent contains all categories;

[0014] Hash retrieval query: Hash codes are used for similarity retrieval through Hamming distance, and the average precision is calculated after multiple iterations. The average precision is used to evaluate the retrieval performance.

[0015] As a preferred method, the medical multimodal dataset is constructed as follows:

[0016] Obtain public datasets: Obtain and download public datasets containing image modalities and text modalities from public network resources as original datasets; public datasets include IU-XRay datasets and CheXpert datasets. IU-XRay datasets and CheXpert datasets contain rich clinical backgrounds to ensure the professionalism and integrity of the datasets.

[0017] The downloaded original dataset is divided into two parts: a set proportion of image-text pairs are randomly selected from the IU-XRay dataset and the CheXpert dataset as the test set to ensure that common and rare cases are covered, so as to comprehensively evaluate the model performance; the image-text pair data of the IU-XRay dataset and the CheXpert dataset are excluded from the retest set, and part of the remaining data is extracted as the retrieval set to ensure the diversity and representativeness of different diseases; in the retrieval dataset, the unselected image-text pairs are divided into the training set to ensure the richness of the training data;

[0018] Extract deep features of image modality: Use the pre-trained fine-tuning model CLIP as the image encoder to extract deep features of the image modality and obtain deep features of the image modality;

[0019] Extract deep features of text modality: Use the pre-trained fine-tuned model GPT2 as the text encoder to extract deep features of the text modality and obtain deep features of the text modality.

[0020] Preferably, the image modality depth features are extracted as follows:

[0021] Preprocess the image modality data: uniformly crop and reshape the input image to H×H pixel size, normalize the medical image according to the medical image processing standard, make the pixel value range of the medical image between [0,1], and eliminate the brightness and contrast differences caused by different imaging devices; then use the BM3D medical image-specific denoising algorithm, such as non-local means (NLM) or BM3D, to denoise the noise unique to medical images;

[0022] Pre-trained fine-tuned model CLIP: Load the pre-trained CLIP model, use the contrast loss function to optimize the pre-trained CLIP model, obtain the pre-trained fine-tuned model CLIP model, and use the pre-trained fine-tuned model CLIP model to optimize the similarity between images and texts, so that the similarity between similar image-text pairs is as high as possible, and the similarity between dissimilar image-text pairs is as low as possible; the contrast loss function is as follows:

[0023]

[0024] Where τ represents the temperature hyperparameter; N represents the number of samples; sim(I i ,T i ) represents the image-text similarity;

[0025] Extract deep image features: Load the pre-trained fine-tuned model CLIP to adapt to medical image feature extraction, ensure that the pre-trained fine-tuned model CLIP can identify pathological features in the image, use the pre-trained fine-tuned model CLIP to extract deep image features, capture subtle changes in the image, such as tumor morphology, lesion area, etc., and record deep image features for subsequent analysis;

[0026] The specific steps for extracting text modality deep features are as follows:

[0027] Preprocessing text modal data: Perform preprocessing operations such as text cleaning, text segmentation, and text vectorization on the input text, and map the text information to text vectors;

[0028] Pre-trained fine-tuned model GPT2: Load the pre-trained model GPT2, use the pre-trained model GPT2 as a text encoder, and fine-tune the pre-trained model GPT2 through medical records and medical literature-related data sets to improve the pre-trained model GPT2's ability to understand medical content. A supervised learning method is used to use the cross-entropy loss function to calculate the error between the pre-trained model GPT2 output and the target text, and then obtain the pre-trained fine-tuned GPT2; the cross-entropy loss function is as follows:

[0029]

[0030] Among them, w t represents the target word; P(w t |w ＜t ; θ) represents the probability of predicting the current word given the previous context; θ represents the model parameters;

[0031] Extract deep text features: Extract deep text features through pre-training fine-tuning model GPT2 to capture information about disease descriptions and treatment plans, ensure that deep text features can reflect the complexity of medical content, and standardize deep text features to ensure consistent feature values ​​for easy model learning and analysis.

[0032] As a preferred embodiment, medical multimodal fusion is as follows:

[0033] Feature alignment: The feature aligner composed of multiple linear layers and activation functions obtains the aligned deep image features and aligned deep text features. The formula is as follows:

[0034]

[0035] Where X and Y represent deep image features and deep text features respectively; T x and T y denote the image feature aligner and text feature aligner respectively; θ dx and θ dy represents the training parameters; and They represent aligned deep image features and aligned deep text features respectively;

[0036] Feature splicing and fusion: The aligned deep image features and aligned deep text features are processed through the feature splicing and fusion function to generate a feature fusion matrix. The formula is as follows:

[0037]

[0038] Among them, Integrate represents the feature concatenation and fusion function, and H represents the feature fusion matrix; and denotes the aligned deep image features and aligned deep text features respectively; θ 3 represents training parameters; ⊕ represents concatenation operation;

[0039] Hash mapping: The feature fusion matrix is ​​passed through the hash mapping function to generate a loose hash code, and then through the sign function to generate a hash code. The formula is as follows:

[0040] b i =HashEncoder(H i θ 4 );

[0041] B i =sign(b i );

[0042] Among them, HashEncoder represents the hash mapping function; b i Represents a loose hash code; B i represents the hash code; H represents the feature fusion matrix; θ 4 Represents training parameters.

[0043] Preferably, prototype parameter reinforcement learning is as follows:

[0044] Personalized prototype learning of the client: The feature fusion matrix is ​​averaged according to the label category to obtain the personalized prototype. The formula is as follows:

[0045]

[0046] in, represents the personalized prototype of the jth class in the i-th client; H represents the feature fusion matrix; m i represents the number of samples in the i-th client; l ij Indicates a label; represents the indicator function, which determines whether the label belongs to the jth category;

[0047] Server prototype aggregation: The server receives personalized prototypes from all clients and obtains a centralized prototype through prototype aggregation. The formula is as follows:

[0048]

[0049] in, represents the jth concentrated prototype in the server; represents the personalized prototype of the jth class in the i-th client; represents the client containing the j-th type of sample; l ij Indicates a label; represents the indicator function, which determines whether the label belongs to the jth category;

[0050] Prototype enhancement strategy: The server distributes the centralized prototypes to the client through the prototype enhancement strategy, and a systematic comparison is made between the centralized prototype and the client's personalized prototype. Through this comparison, the server identifies any missing category prototypes and performs prototype enhancement. The formula is as follows:

[0051]

[0052] in, represents the jth concentrated prototype in the server; represents the personalized prototype of the jth class in the i-th client; represents the client containing the j-th type of sample;

[0053] The hash search query is as follows:

[0054] Calculate the average precision using the following formula:

[0055]

[0056] Where N represents the total number of samples associated with the retrieval sample; P(m) represents the accuracy of the first M related instances, when the mth sample is related to the retrieval sample, δ(m) = 1, otherwise δ(m) = 0;

[0057] Calculate the mean average precision, the formula is as follows:

[0058]

[0059] Where Q represents the number of retrieval sets; AP(i) represents the i-th AP value;

[0060] The MAP value is obtained by calculating the mean average precision under different recall rates and averaging the mean average precision.

[0061] Preferably, the training model is as follows:

[0062] Construct loss function: The goal of the prototype-based federated medical multimodal hash retrieval model is to use the feature fusion matrix with personalized prototypes and centralized prototypes to train the federated learning process, and to train the efficient multimodal hash process through loose hash codes and hash codes, and finally achieve a more efficient federated multimodal hash retrieval, using the feature fusion matrix H, personalized prototype P, centralized prototype Loose hash code b i and hash code B i Construct loss function to train prototype-based federated medical multimodal hash retrieval model;

[0063] Optimization model: The Adam algorithm is used as the optimization function of the prototype-based federated medical multimodal hash retrieval model; the learning rate parameter is set to 0.001, and other hyperparameters use the default values ​​in PyTorch;

[0064] Among them, the loss function is constructed as follows:

[0065] Construct the global contrast loss, the formula is as follows:

[0066]

[0067] in, represents the centralized prototype; H represents the fusion matrix; τ represents the temperature coefficient;

[0068] Construct the local contrast loss, the formula is as follows:

[0069]

[0070] in, represents the multimodal data in the i-th client; m i represents the number of samples in the i-th client; P represents the personalized prototype; τ represents the temperature coefficient;

[0071] Construct the hash loss, the formula is as follows:

[0072]

[0073] Among them, α and β represent optimization parameters; b i Represents a loose hash code; B i represents a hash code; l represents a label.

[0074] A prototype-based federated medical multimodal hash retrieval system, the system comprising:

[0075] The medical multimodal dataset construction unit is used to obtain image modality and text modality datasets, standardize the image modality dataset, standardize the medical terminology of the text modality dataset, and add the patient's clinical background information to enrich the context preprocessing operation, and then use the pre-trained fine-tuning model CLIP and the pre-trained fine-tuning model GPT2 to extract deep features from the image modality and text modality datasets respectively to obtain deep image features and deep text features, and evenly divide the dataset according to category and sample number to construct the final dataset including training set, test set and retrieval set;

[0076] A prototype-based federated medical multimodal hash retrieval model construction unit, which is used to construct a prototype-based federated medical multimodal hash retrieval model using neural networks and deep learning technologies;

[0077] A model training unit, used to train a prototype-based federated medical multimodal hash retrieval model using the final dataset;

[0078] Among them, the prototype-based federated medical multimodal hash retrieval model construction unit includes:

[0079] The medical multimodal fusion module is used to align the deep image features and the deep text features, then concatenate and fuse the deep image features and the deep text features to generate a feature fusion matrix, and output the hash code through the hash mapping part;

[0080] The prototype parameter reinforcement learning module is used to generate a personalized prototype by calculating the feature fusion matrix in the client through the prototype. The server collects the personalized prototypes of each client to generate a centralized prototype, identifies and fills in the missing category prototypes in the client-specific prototype, and ensures that the personalized prototype generated by the client sent contains all categories.

[0081] The hash retrieval query module is used to perform similarity retrieval based on the hash code through the Hamming distance, and the average precision is calculated through multiple iterations. The average precision is used to evaluate the retrieval performance.

[0082] Preferably, the medical multimodal fusion module obtains aligned deep image features and aligned deep text features through a feature aligner, and then generates a feature fusion matrix through a feature splicing fusion function for the aligned deep image features and aligned deep text features, and then generates a loose hash code through a hash mapping function for the feature fusion matrix, and finally outputs the hash code through a sign function;

[0083] The prototype parameter reinforcement learning module generates a personalized prototype by learning the feature fusion matrix in the client through the personalized prototype of the client. The server collects the personalized prototypes of each client and generates a centralized prototype through server prototype aggregation. The prototype reinforcement strategy is then used to identify and fill in the missing category prototypes in the client-specific prototype to ensure that the personalized prototype generated by the client sent contains all categories and the client obtains an updated personalized prototype.

[0084] The hash retrieval query module calculates the Hamming distance through the hash code and performs similarity retrieval. It calculates the average precision through multiple iterations, and then calculates the average precision mean. The final MAP value is used to evaluate the retrieval performance.

[0085] An electronic device comprising: a memory and at least one processor;

[0086] Wherein, the memory stores a computer program;

[0087] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the prototype-based federated medical multimodal hash retrieval method as described above.

[0088] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the prototype-based federated medical multimodal hash retrieval method as described above.

[0089] The prototype-based federated medical multimodal hash retrieval method, system, device and medium of the present invention have the following advantages:

[0090] (I) The present invention solves the problem of performing safe and efficient multimodal hash retrieval while ensuring data privacy and security, and implements a neural network structure for federated medical multimodal hash retrieval based on a prototype, ensuring the generation of high-quality hash codes while ensuring security and privacy, and effectively improving the accuracy of federated multimodal hash retrieval;

[0091] (ii) The present invention uses CLIP and GPT2 as modal deep feature extraction encoders, which not only enriches the semantic information of multimodal data, but also more accurately captures the underlying semantic information of images and texts, thereby further enriching the feature semantic information of multimodal data and improving the accuracy and efficiency of retrieval;

[0092] (III) The present invention proposes a neural network structure for prototype federated medical multimodal hash retrieval, which can align the semantics of two modalities, effectively solving the problem of poor fusion efficiency of medical data sets, and laying the foundation for generating high-quality hash codes through the medical multimodal fusion module, ultimately achieving efficient federated multimodal hashing;

[0093] (IV) The prototype-based federated medical multimodal hash retrieval proposed by the present invention effectively improves the accuracy and efficiency of multimodal hash retrieval, provides a more superior solution for practical applications in fields such as secure multimodal retrieval, and effectively meets the requirement of strong privacy for medical data;

[0094] (V) The present invention securely transmits prototypes in federated medical multimodal hash retrieval, which further increases the security of information without directly transmitting parameters, and effectively solves the problems of category imbalance and statistical heterogeneity in multimodal data, thereby improving the generalization ability of the model;

[0095] (VI) The present invention proposes an innovative prototype enhancement strategy, which effectively fills in the missing category prototypes and achieves a balanced category distribution by systematically comparing and supplementing the personalized prototypes submitted by the client on the server side. This strategy not only ensures that the prototype set of each client covers the prototype information of all categories, but also strengthens the connection between prototypes and categories and fills the semantic gap by enhancing the semantic information of the model during the training process.

[0096] (VII) The present invention implements a server-guided aggregation and enhancement mechanism, which promotes the sharing of sample feature information across the entire network by integrating prototype sets from all clients;

[0097] (VIII) The present invention improves the quality and distinguishing ability of hash codes by combining global contrast loss, local contrast loss and hash optimization loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The present invention is further described below in conjunction with the accompanying drawings.

[0099] Attached Figure 1 This is a flowchart of the prototype-based federated medical multimodal hash retrieval method;

[0100] Attached Figure 2 A flowchart for building a medical multimodal dataset;

[0101] Attached Figure 3 A flowchart for building a prototype-based federated medical multimodal hash retrieval model;

[0102] Attached Figure 4 This is a flowchart of medical multimodal fusion;

[0103] Attached Figure 5 A flowchart for prototype parameter reinforcement learning;

[0104] Attached Figure 6 A flowchart for hash retrieval query;

[0105] Attached Figure 7 This is a flowchart for training the model. DETAILED DESCRIPTION

[0106] The prototype-based federated medical multimodal hash retrieval method, system, device and medium of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.

[0107] Embodiment 1:

[0108] As attached Figure 1 As shown, this embodiment provides a prototype-based federated medical multimodal hash retrieval method, which is specifically as follows:

[0109] S1. Construct a medical multimodal dataset: obtain image modality and text modality datasets, standardize the image modality dataset, including resizing and unifying contrast and brightness to eliminate differences caused by different devices, standardize medical terminology for the text modality dataset, and add clinical background information of patients to enrich the context preprocessing operations, then use the pre-trained fine-tuning model CLIP and the pre-trained fine-tuning model GPT2 to perform deep feature extraction on the image modality and text modality datasets respectively, obtain deep image features and deep text features, and divide the dataset evenly according to categories and sample numbers to construct the final dataset containing training set, test set and retrieval set;

[0110] S2. Construct a prototype-based federated medical multimodal hash retrieval model: Use neural network and deep learning technology to construct a prototype-based federated medical multimodal hash retrieval model;

[0111] S3. Training model: Use the final dataset to train the prototype-based federated medical multimodal hash retrieval model;

[0112] As attached Figure 3 As shown, the construction of the prototype-based federated medical multimodal hash retrieval model in step S2 of this embodiment is specifically as follows:

[0113] S201, medical multimodal fusion: after feature alignment of the deep image features and the deep text features, the deep image features and the deep text features are concatenated and fused to generate a feature fusion matrix, and a hash code is output through the hash mapping part;

[0114] S202, prototype parameter enhancement learning: The feature fusion matrix in the client is used to generate a personalized prototype through prototype calculation. The server collects the personalized prototypes of each client to generate a centralized prototype, identifies and fills in the category prototypes missing in the client-specific prototype, and ensures that the personalized prototype generated by the client sent contains all categories;

[0115] S203, hash search query: The hash code performs similarity search through Hamming distance, and the average precision is calculated through multiple iterations. The average precision is used to evaluate the search performance.

[0116] As attached Figure 2 As shown, the construction of the medical multimodal dataset in step S1 of this embodiment is specifically as follows:

[0117] S101. Obtain public datasets: Obtain and download public datasets containing image modalities and text modalities from public network resources as original datasets; the public datasets include the IU-XRay dataset and the CheXpert dataset, which contain rich clinical backgrounds to ensure the professionalism and integrity of the datasets;

[0118] S102. Divide the downloaded original data set: randomly select a set proportion of image-text pairs from the IU-XRay data set and the CheXpert data set as the test set to ensure that common and rare cases are covered, so as to comprehensively evaluate the model performance; exclude the image-text pair data from the IU-XRay data set and the CheXpert data set and retest the data, and extract part of the remaining data as the retrieval set to ensure the diversity and representativeness of different diseases; and then divide the unselected image-text pairs in the retrieval data set into the training set to ensure the richness of the training data;

[0119] S103, extracting image modality deep features: using the pre-trained fine-tuning model CLIP as an image encoder to extract deep features of the image modality and obtain image modality deep features;

[0120] S104. Extract deep features of text modality: Use the pre-trained fine-tuning model GPT2 as the text encoder to extract deep features of the text modality and obtain deep features of the text modality.

[0121] The extraction of image modality depth features in step S103 of this embodiment is specifically as follows:

[0122] S10301. Preprocess image modality data: uniformly crop and reshape the input image to H×H pixel size, normalize the medical image according to the medical image processing standard, make the pixel value range of the medical image between [0,1], and eliminate the brightness and contrast differences caused by different imaging devices; then use the BM3D medical image dedicated denoising algorithm, such as non-local means (NLM) or BM3D, to denoise the noise unique to medical images;

[0123] S10302, pre-trained fine-tuning model CLIP: load the pre-trained CLIP model, use the contrast loss function to optimize the pre-trained CLIP model, obtain the pre-trained fine-tuning model CLIP model, and optimize the similarity between the image and the text through the pre-trained fine-tuning model CLIP model, so that the similarity between similar image-text pairs is as high as possible, and the similarity between dissimilar image-text pairs is as low as possible; wherein the contrast loss function is as follows:

[0124]

[0125] Where τ represents the temperature hyperparameter; N represents the number of samples; sim(I i ,T i ) represents the image-text similarity;

[0126] S10303. Extract deep image features: Load the pre-trained fine-tuned model CLIP to adapt to medical image feature extraction, ensure that the pre-trained fine-tuned model CLIP can identify pathological features in the image, use the pre-trained fine-tuned model CLIP to extract deep image features, capture subtle changes in the image, such as tumor morphology, lesion area, etc., and record the deep image features for subsequent analysis.

[0127] For example: In the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0128]

[0129] Among them, feature represents the extracted deep image features.

[0130] The extraction of text modality depth features in step S104 of this embodiment is specifically as follows:

[0131] S10401, preprocessing text modal data: performing preprocessing operations of text cleaning, text segmentation and text vectorization on the input text, and mapping the text information to a text vector;

[0132] S10402, pre-trained fine-tuned model GPT2: load the pre-trained model GPT2, use the pre-trained model GPT2 as a text encoder, fine-tune the pre-trained model GPT2 through medical records and medical literature-related data sets in the medical field, improve the pre-trained model GPT2's ability to understand medical content, and use a supervised learning method to use a cross-entropy loss function to calculate the error between the pre-trained model GPT2 output and the target text, thereby obtaining the pre-trained fine-tuned GPT2; wherein, the cross-entropy loss function is as follows:

[0133]

[0134] Among them, w t represents the target word; P(w t |w ＜t ; θ) represents the probability of predicting the current word given the previous context; θ represents the model parameters;

[0135] S10403. Extract deep text features: Extract deep text features through pre-training fine-tuning model GPT2 to capture information about disease descriptions and treatment plans, ensure that deep text features can reflect the complexity of medical content, and standardize deep text features to ensure consistent feature values ​​for easy model learning and analysis.

[0136] For example: In the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0137]

[0138]

[0139] Among them, text_features represents the extracted deep text features.

[0140] As attached Figure 4 As shown, the medical multimodal fusion in step S201 of this embodiment is specifically as follows:

[0141] S20101. Feature alignment: The feature aligner composed of multiple linear layers and activation functions is used to obtain the aligned deep image features and aligned deep text features. The formula is as follows:

[0142]

[0143] Where X and Y represent deep image features and deep text features respectively; T x and T y denote the image feature aligner and text feature aligner respectively; θ dx and θ dy represents the training parameters; and They represent aligned deep image features and aligned deep text features respectively;

[0144] S20102, feature splicing and fusion: The aligned deep image features and aligned deep text features are subjected to the feature splicing and fusion function to generate a feature fusion matrix. The formula is as follows:

[0145]

[0146] Among them, Integrate represents the feature concatenation and fusion function, and H represents the feature fusion matrix; and denotes the aligned deep image features and aligned deep text features respectively; θ 3 represents training parameters; ⊕ represents concatenation operation;

[0147] S20103, hash mapping: The feature fusion matrix is ​​passed through the hash mapping function to generate a loose hash code, and then through the sign function to generate a hash code. The formula is as follows:

[0148] b i =HashEncoder(H i θ 4 );

[0149] B i =sign(b i );

[0150] Among them, HashEncoder represents the hash mapping function; b i Represents a loose hash code; B i represents the hash code; H represents the feature fusion matrix; θ 4 Represents training parameters.

[0151] For example: In the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0152]

[0153] Among them, nn.Linear represents the linear layer; torch.sign represents the sign activation function; FeatureAligner represents feature alignment; Integrate represents feature concatenation and fusion; HashEncoder represents the hash mapping function.

[0154] As attached Figure 5 As shown, the prototype parameter enhancement learning in step S202 of this embodiment is specifically as follows:

[0155] S20201. Personalized prototype learning of the client: The feature fusion matrix is ​​averaged according to the label category to obtain a personalized prototype. The formula is as follows:

[0156]

[0157]

[0158] in, represents the personalized prototype of the jth class in the i-th client; H represents the feature fusion matrix; m i represents the number of samples in the i-th client; l ij Indicates a label; represents the indicator function, which determines whether the label belongs to the jth category;

[0159] S20202, server prototype aggregation: The server receives personalized prototypes of all clients and obtains a centralized prototype through prototype aggregation. The formula is as follows:

[0160]

[0161] in, represents the jth concentrated prototype in the server; represents the personalized prototype of the jth class in the i-th client; represents the client containing the j-th type of sample; l ij Indicates a label; represents the indicator function, which determines whether the label belongs to the jth category;

[0162] S20203, prototype enhancement strategy: The server distributes the centralized prototype to the client through the prototype enhancement strategy, and a systematic comparison is made between the centralized prototype and the client's personalized prototype. Through this comparison, the server identifies any missing category prototypes and performs prototype enhancement. The formula is as follows:

[0163]

[0164] in, represents the jth concentrated prototype in the server; represents the personalized prototype of the jth class in the i-th client; represents the client containing the j-th type of samples.

[0165] For example: In the PyTorch framework, the code implementation corresponding to the above description is as follows:

[0166]

[0167]

[0168] Among them, compute_personalized_prototypes represents the personalized prototype learning of the S20201 client; aggregate_prototypes represents the S20202 server prototype aggregation; enhance_prototypes represents the S20203 prototype enhancement strategy.

[0169] As attached Figure 6 As shown, the hash search query in step S203 of this embodiment is specifically as follows:

[0170] S20301. Calculate the average precision using the following formula:

[0171]

[0172] Where N represents the total number of samples associated with the retrieval sample; P(m) represents the accuracy of the first M related instances, when the mth sample is related to the retrieval sample, δ(m) = 1, otherwise δ(m) = 0;

[0173] S20302. Calculate the average precision. The formula is as follows:

[0174]

[0175] Where Q represents the number of retrieval sets; AP(i) represents the i-th AP value;

[0176] S20303. Calculate the mean average precision under different recall rates and average the mean average precision to obtain the MAP value.

[0177] For example, in the PyTorch framework, the code implementation corresponding to the hash search query described above is as follows:

[0178]

[0179] Among them, compute_ap is the S20301 average precision (AP) calculation; compute_map is the S20302 average precision mean (MAP) calculation; torch.cdist is the calculation of Hamming distance.

[0180] As attached Figure 7 As shown, the training model in step S3 of this embodiment is specifically as follows:

[0181] S301. Construct loss function: The goal of the prototype-based federated medical multimodal hash retrieval model is to use the feature fusion matrix with personalized prototypes and centralized prototypes to train the federated learning process, and to train the efficient multimodal hash process through loose hash codes and hash codes, and finally achieve a more efficient federated multimodal hash retrieval, using the feature fusion matrix H, personalized prototype P, centralized prototype Loose hash code b i and hash code B i Construct loss function to train prototype-based federated medical multimodal hash retrieval model;

[0182] S302. Optimize model: Use the Adam algorithm as the optimization function of the prototype-based federated medical multimodal hash retrieval model; the learning rate parameter is set to 0.001, and other hyperparameters use the default values ​​in PyTorch.

[0183] In this embodiment, the MAP value is selected as the evaluation index.

[0184] For example, the optimization function described above is expressed in PyTorch using the following code:

[0185] optimizer=torch.optim.Adam(model.parameters(),lr=learning_rate)

[0186] Among them, torch.optim.Adam is the Adam optimization function embedded in pytorch; model.parameters() is the parameter set for model training; learning_rate is the learning rate.

[0187] The loss function constructed in step S301 of this embodiment is specifically as follows:

[0188] S30101. Construct global contrast loss. The formula is as follows:

[0189]

[0190] in, represents the centralized prototype; H represents the fusion matrix; τ represents the temperature coefficient;

[0191] S30102. Construct local contrast loss. The formula is as follows:

[0192]

[0193] in, represents the multimodal data in the i-th client; m i represents the number of samples in the i-th client; P represents the personalized prototype; τ represents the temperature coefficient;

[0194] S30103. Construct hash loss, the formula is as follows:

[0195]

[0196] Among them, α and β represent optimization parameters; b i Represents a loose hash code; B i represents a hash code; l represents a label.

[0197] For example, the loss function described above is expressed in PyTorch using the code:

[0198]

[0199] Among them, global_contrastive_loss represents S30101 global contrast loss; local_contrastive_loss represents S30102 local contrast loss; hash_loss represents S30103 hash loss.

[0200] The prototype-based federated medical multimodal hash retrieval model of this embodiment is experimented on two public datasets, the IU-XRay dataset and the CheXpert dataset. The experiment shows that the model of this embodiment is better than the existing model. The comparison of the experimental results MAP is shown in the following table:

[0201]

[0202]

[0203] The model of this embodiment is compared with the existing model, and it can be seen that the performance of this embodiment is the best compared with other methods.

[0204] Embodiment 2:

[0205] This embodiment provides a prototype-based federated medical multimodal hash retrieval system, which includes:

[0206] The medical multimodal dataset construction unit is used to obtain image modality and text modality datasets, standardize the image modality dataset, standardize the medical terminology of the text modality dataset, and add the patient's clinical background information to enrich the context preprocessing operation, and then use the pre-trained fine-tuning model CLIP and the pre-trained fine-tuning model GPT2 to extract deep features from the image modality and text modality datasets respectively to obtain deep image features and deep text features, and evenly divide the dataset according to category and sample number to construct the final dataset including training set, test set and retrieval set;

[0207] A prototype-based federated medical multimodal hash retrieval model construction unit, which is used to construct a prototype-based federated medical multimodal hash retrieval model using neural networks and deep learning technologies;

[0208] A model training unit, used to train a prototype-based federated medical multimodal hash retrieval model using the final dataset;

[0209] Among them, the prototype-based federated medical multimodal hash retrieval model construction unit includes:

[0210] The medical multimodal fusion module is used to align the deep image features and the deep text features, then concatenate and fuse the deep image features and the deep text features to generate a feature fusion matrix, and output the hash code through the hash mapping part;

[0211] The prototype parameter reinforcement learning module is used to generate a personalized prototype by calculating the feature fusion matrix in the client through the prototype. The server collects the personalized prototypes of each client to generate a centralized prototype, identifies and fills in the missing category prototypes in the client-specific prototype, and ensures that the personalized prototype generated by the client sent contains all categories.

[0212] The hash retrieval query module is used to perform similarity retrieval based on the hash code through the Hamming distance, and the average precision is calculated through multiple iterations. The average precision is used to evaluate the retrieval performance.

[0213] The medical multimodal fusion module in this embodiment passes the deep image features and the deep text features through the feature aligner to obtain the aligned deep image features and the aligned deep text features, then passes the aligned deep image features and the aligned deep text features through the feature splicing and fusion function to generate a feature fusion matrix, then passes the feature fusion matrix through the hash mapping function to generate a loose hash code, and finally outputs the hash code through the sign function.

[0214] The prototype parameter enhancement learning module in this embodiment generates a personalized prototype through the personalized prototype learning of the client through the feature fusion matrix in the client, and then collects the personalized prototypes of each client on the server and generates a centralized prototype through server prototype aggregation. The prototype enhancement strategy is then used to identify and fill in the missing category prototypes in the client-specific prototype to ensure that the personalized prototype generated by the client sent contains all categories, and the client obtains an updated personalized prototype.

[0215] The hash search query module in this embodiment calculates the Hamming distance through the hash code and performs similarity search, calculates the average precision through multiple iterations, and then calculates the average precision mean, and finally obtains the MAP value to evaluate the search performance.

[0216] Embodiment 3:

[0217] This embodiment also provides an electronic device, including: a memory and a processor;

[0218] Wherein, the memory stores computer-executable instructions;

[0219] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the prototype-based federated medical multimodal hash retrieval method in any embodiment of the present invention.

[0220] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0221] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0222] Embodiment 4:

[0223] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor, so that the processor executes the prototype-based federated medical multimodal hash retrieval method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0224] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0225] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0226] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0227] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A prototype-based federated medical multimodal hash retrieval method, characterized in that: The method is as follows: Construct a medical multimodal dataset: obtain image modality and text modality datasets, standardize the image modality dataset, standardize the medical terminology of the text modality dataset, and add the patient's clinical background information to enrich the context preprocessing operation, then use the pre-trained fine-tuning model CLIP and the pre-trained fine-tuning model GPT2 to extract deep features from the image modality and text modality datasets respectively, obtain deep image features and deep text features, and divide the dataset evenly according to category and sample number to construct the final dataset containing training set, test set and retrieval set; Construct a prototype-based federated medical multimodal hash retrieval model: Use neural networks and deep learning technology to construct a prototype-based federated medical multimodal hash retrieval model; Training model: Use the final dataset to train the prototype-based federated medical multimodal hash retrieval model; The construction of a prototype-based federated medical multimodal hash retrieval model is as follows: Medical multimodal fusion: After feature alignment, deep image features and deep text features are concatenated and fused to generate a feature fusion matrix, and the hash code is output through the hash mapping part; Prototype parameter reinforcement learning: The feature fusion matrix in the client is used to generate a personalized prototype through prototype calculation. The server collects the personalized prototypes of each client to generate a centralized prototype, identifies and fills in the category prototypes missing in the client-specific prototype, and ensures that the personalized prototype generated by the client sent contains all categories; Hash retrieval query: Hash codes are used for similarity retrieval through Hamming distance, and the average precision is calculated after multiple iterations. The average precision is used to evaluate the retrieval performance.

2. The prototype-based federated medical multimodal hash retrieval method according to claim 1, characterized in that: The construction of a medical multimodal dataset is as follows: Obtain public datasets: Obtain and download public datasets containing image modalities and text modalities from public network resources as original datasets; public datasets include IU-XRay datasets and CheXpert datasets. IU-XRay datasets and CheXpert datasets contain rich clinical backgrounds to ensure the professionalism and integrity of the datasets. The downloaded original dataset is divided into two parts: a set proportion of image-text pairs are randomly selected from the IU-XRay dataset and the CheXpert dataset as the test set; the image-text pairs from the IU-XRay dataset and the CheXpert dataset are excluded from the retest set, and part of the remaining data is extracted as the retrieval set to ensure the diversity and representativeness of different diseases; in the retrieval dataset, the unselected image-text pairs are divided into the training set to ensure the richness of the training data; Extract deep features of image modality: Use the pre-trained fine-tuning model CLIP as the image encoder to extract deep features of the image modality and obtain deep features of the image modality; Extract deep features of text modality: Use the pre-trained fine-tuned model GPT2 as the text encoder to extract deep features of the text modality and obtain deep features of the text modality.

3. The prototype-based federated medical multimodal hash retrieval method according to claim 2, characterized in that: The specific steps for extracting image modality depth features are as follows: Preprocess image modality data: uniformly crop and reshape the input image to H×H pixel size, normalize the medical image according to the medical image processing standard, make the pixel value range of the medical image between [0, 1], and eliminate the brightness and contrast differences caused by different imaging devices; then use the BM3D medical image dedicated denoising algorithm to denoise the noise unique to medical images; Pre-trained fine-tuned model CLIP: Load the pre-trained CLIP model, use the contrast loss function to optimize the pre-trained CLIP model, obtain the pre-trained fine-tuned model CLIP model, and use the pre-trained fine-tuned model CLIP model to optimize the similarity between images and texts, so that the similarity between similar image-text pairs is high, and the similarity between dissimilar image-text pairs is low; the contrast loss function is as follows: Where τ represents the temperature hyperparameter; N represents the number of samples; sim(I i , T i ) represents the image-text similarity; Extract deep image features: Load the pre-trained fine-tuned model CLIP to adapt to medical image feature extraction, ensure that the pre-trained fine-tuned model CLIP can identify pathological features in the image, use the pre-trained fine-tuned model CLIP to extract deep image features, capture subtle changes in the image, and record deep image features for subsequent analysis; The specific steps for extracting text modality deep features are as follows: Preprocessing text modal data: Perform preprocessing operations such as text cleaning, text segmentation, and text vectorization on the input text, and map the text information to text vectors; Pre-trained fine-tuned model GPT2: Load the pre-trained model GPT2, use the pre-trained model GPT2 as a text encoder, and fine-tune the pre-trained model GPT2 through medical records and medical literature-related data sets to improve the pre-trained model GPT2's ability to understand medical content. Use a supervised learning method to use the cross-entropy loss function to calculate the error between the pre-trained model GPT2 output and the target text, and then obtain the pre-trained fine-tuned GPT2; the cross-entropy loss function is as follows: Among them, w t represents the target word; P(w t |w ＜t ; θ) represents the probability of predicting the current word given the previous context; θ represents the model parameters; Extract deep text features: Extract deep text features through pre-training fine-tuning model GPT2 to capture information about disease descriptions and treatment plans, ensure that deep text features can reflect the complexity of medical content, and standardize deep text features to ensure consistent feature values ​​for easy model learning and analysis.

4. The prototype-based federated medical multimodal hash retrieval method according to claim 1, characterized in that: The details of medical multimodal fusion are as follows: Feature alignment: The feature aligner composed of multiple linear layers and activation functions obtains the aligned deep image features and aligned deep text features. The formula is as follows: Where X and Y represent deep image features and deep text features respectively; T x and T y denote the image feature aligner and text feature aligner respectively; θ dx and θ dy represents the training parameters; and They represent aligned deep image features and aligned deep text features respectively; Feature splicing and fusion: The aligned deep image features and aligned deep text features are processed through the feature splicing and fusion function to generate a feature fusion matrix. The formula is as follows: Among them, Integrate represents the feature concatenation and fusion function, and H represents the feature fusion matrix; and They represent aligned deep image features and aligned deep text features respectively; θ3 represents the training parameters; Represents a splicing operation; Hash mapping: The feature fusion matrix is ​​passed through the hash mapping function to generate a loose hash code, and then through the sign function to generate a hash code. The formula is as follows: b i =HashEncoder(H i ;θ4); B i =sign(b i ); Among them, HashEncoder represents the hash mapping function; b i Represents a loose hash code; B i represents the hash code; H represents the feature fusion matrix; θ4 represents the training parameter.

5. The prototype-based federated medical multimodal hash retrieval method according to claim 1 or 4, characterized in that: The prototype parameter reinforcement learning is as follows: Personalized prototype learning of the client: The feature fusion matrix is ​​averaged according to the label category to obtain the personalized prototype. The formula is as follows: in, represents the personalized prototype of the jth class in the i-th client; H represents the feature fusion matrix; m i represents the number of samples in the i-th client; l ij Indicates a label; represents the indicator function, which determines whether the label belongs to the jth category; Server prototype aggregation: The server receives personalized prototypes from all clients and obtains a centralized prototype through prototype aggregation. The formula is as follows: in, represents the jth concentrated prototype in the server; represents the personalized prototype of the jth class in the i-th client; represents the client containing the j-th type of sample; l ij Indicates a label; represents the indicator function, which determines whether the label belongs to the jth category; Prototype enhancement strategy: The server distributes the centralized prototypes to the client through the prototype enhancement strategy, and a systematic comparison is made between the centralized prototype and the client's personalized prototype. Through this comparison, the server identifies any missing category prototypes and performs prototype enhancement. The formula is as follows: in, represents the jth concentrated prototype in the server; represents the personalized prototype of the jth class in the i-th client; represents the client containing the j-th type of sample; The hash search query is as follows: Calculate the average precision using the following formula: Where N represents the total number of samples associated with the retrieval sample; P(m) represents the accuracy of the first M related instances, when the mth sample is related to the retrieval sample, δ(m) = 1, otherwise δ(m) = 0; Calculate the mean average precision, the formula is as follows: Where Q represents the number of retrieval sets; AP(i) represents the i-th AP value; The MAP value is obtained by calculating the mean average precision under different recall rates and averaging the mean average precision.

6. The prototype-based federated medical multimodal hash retrieval method according to claim 5, characterized in that: The training model is as follows: Construct loss function: The goal of the prototype-based federated medical multimodal hash retrieval model is to use the feature fusion matrix with personalized prototypes and centralized prototypes to train the federated learning process, and to train the efficient multimodal hash process through loose hash codes and hash codes, and finally achieve a more efficient federated multimodal hash retrieval, using the feature fusion matrix H, personalized prototype P, centralized prototype Loose hash code b i and hash code B i Construct loss function to train prototype-based federated medical multimodal hash retrieval model; Optimization model: The Adam algorithm is used as the optimization function of the prototype-based federated medical multimodal hash retrieval model; the learning rate parameter is set to 0.001, and other hyperparameters use the default values ​​in PyTorch; Among them, the loss function is constructed as follows: Construct the global contrast loss, the formula is as follows: in, represents the centralized prototype; H represents the fusion matrix; τ represents the temperature coefficient; Construct the local contrast loss, the formula is as follows: in, represents the multimodal data in the i-th client; m i represents the number of samples in the i-th client; P represents the personalized prototype; τ represents the temperature coefficient; Construct the hash loss, the formula is as follows: Among them, α and β represent optimization parameters; b i Represents a loose hash code; B i represents a hash code; l represents a label.

7. A prototype-based federated medical multimodal hash retrieval system, characterized in that: The system includes: The medical multimodal dataset construction unit is used to obtain image modality and text modality datasets, standardize the image modality dataset, standardize the medical terminology of the text modality dataset, and add the patient's clinical background information to enrich the context preprocessing operation, and then use the pre-trained fine-tuning model CLIP and the pre-trained fine-tuning model GPT2 to extract deep features from the image modality and text modality datasets respectively to obtain deep image features and deep text features, and evenly divide the dataset according to category and sample number to construct the final dataset including training set, test set and retrieval set; A prototype-based federated medical multimodal hash retrieval model construction unit, which is used to construct a prototype-based federated medical multimodal hash retrieval model using neural networks and deep learning technologies; A model training unit, used to train a prototype-based federated medical multimodal hash retrieval model using the final dataset; Among them, the prototype-based federated medical multimodal hash retrieval model construction unit includes: The medical multimodal fusion module is used to align the deep image features and the deep text features, then concatenate and fuse the deep image features and the deep text features to generate a feature fusion matrix, and output the hash code through the hash mapping part; The prototype parameter reinforcement learning module is used to generate a personalized prototype by calculating the feature fusion matrix in the client through the prototype. The server collects the personalized prototypes of each client to generate a centralized prototype, identifies and fills in the missing category prototypes in the client-specific prototype, and ensures that the personalized prototype generated by the client sent contains all categories. The hash retrieval query module is used to perform similarity retrieval based on the hash code through the Hamming distance, and the average precision is calculated through multiple iterations. The average precision is used to evaluate the retrieval performance.

8. The prototype-based federated medical multimodal hash retrieval system according to claim 7, characterized in that: The medical multimodal fusion module passes the deep image features and deep text features through the feature aligner to obtain the aligned deep image features and aligned deep text features, then passes the aligned deep image features and aligned deep text features through the feature splicing fusion function to generate a feature fusion matrix, then passes the feature fusion matrix through the hash mapping function to generate a loose hash code, and finally outputs the hash code through the sign function; The prototype parameter reinforcement learning module generates a personalized prototype by learning the feature fusion matrix in the client through the personalized prototype of the client. The server collects the personalized prototypes of each client and generates a centralized prototype through server prototype aggregation. The prototype reinforcement strategy is then used to identify and fill in the missing category prototypes in the client-specific prototype to ensure that the personalized prototype generated by the client sent contains all categories and the client obtains an updated personalized prototype. The hash retrieval query module calculates the Hamming distance through the hash code and performs similarity retrieval. It calculates the average precision through multiple iterations, and then calculates the average precision mean. The final MAP value is used to evaluate the retrieval performance.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the prototype-based federated medical multimodal hash retrieval method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the prototype-based federated medical multimodal hash retrieval method as described in any one of claims 1 to 6.

Images