Medical data processing method and apparatus

Abstract

This application provides a medical data processing method and apparatus. After obtaining a patient's medical images and medical records, the method utilizes a fusion network model to fuse the image features of the medical images and the case features of the medical records, obtaining fused features. Since the fusion network model is trained with the goal of minimizing the loss function of the disease classifier and maximizing the loss function of the privacy predictor, the fused features generated by the trained fusion network model are not only conducive to accurately determining disease diagnosis results but also effectively reduce the amount of privacy data contained in the fused features. Therefore, in the process of determining disease diagnosis results through a disease diagnosis platform based on the fused features, the possibility of inferring patient privacy data based on the fused features can be reduced, thus naturally reducing the risk of patient privacy data leakage.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a medical data processing method and apparatus. Background Technology

[0002] With the continuous development of medical technology, artificial intelligence technologies such as machine learning algorithms or neural network models can be used to more accurately diagnose diseases.

[0003] Typically, after obtaining a patient's medical images and medical records, medical research institutions can use medical diagnostic platforms to determine the patient's diagnosis. However, medical records often contain sensitive data such as the patient's age, occupation, and gender. Therefore, transmitting medical images and medical records to medical diagnostic platforms can easily lead to the leakage of patients' private data. Summary of the Invention

[0004] In view of this, this application provides a medical data processing method and apparatus to reduce the risk of patient privacy data leakage while using a medical diagnostic platform for disease diagnosis.

[0005] To achieve the above objectives, this application provides a medical data processing method, comprising:

[0006] Obtain the user's medical images and medical records;

[0007] Determine the target image features of the medical images and the target case features of the medical cases;

[0008] Based on the target image features and the target case features, a fusion network model is used to determine the target fusion features between the medical image and the medical case;

[0009] Based on the aforementioned target fusion features, a medical diagnostic platform is used to determine the disease diagnosis result;

[0010] The fusion network model is trained by using multiple sample pairs labeled with actual disease results and actual privacy results, with the goal of minimizing the first loss function value of the disease classifier and maximizing the second loss function value of the privacy predictor.

[0011] The first loss function value characterizes the difference between the disease classification result determined by the disease classifier based on the fusion features of the sample pair and the actual disease result labeled on the sample pair;

[0012] The second loss function value characterizes the difference between the privacy prediction result determined by the privacy predictor based on the fusion features of the sample pair and the actual privacy result labeled on the sample pair;

[0013] The sample pairs include medical image samples and medical case samples;

[0014] The fusion features of the sample pair are the fusion features determined by the fusion network model based on the image features of the medical image samples in the sample pair and the case features of the medical case samples in the sample pair.

[0015] In one possible implementation, during the training of the fusion network model using multiple sample pairs, the disease classifier is trained with the goal of minimizing the first loss function value, and the privacy predictor is trained with the goal of minimizing the second loss function value.

[0016] In another possible implementation, determining the target image features of the medical image and the target case features of the medical case includes:

[0017] Extract the basic image features of the medical images and the basic case features of the medical cases;

[0018] Based on the basic image features and basic case features, the cross-modal image features of the medical image are determined using the first cross-modal attention model, and the cross-modal image features are determined as the target image features of the medical image.

[0019] Based on the basic case features and basic image features, the cross-modal case features of the medical case are determined using the second cross-modal attention model, and the cross-modal case features are determined as the target case features of the medical case.

[0020] In another possible implementation, the extraction of basic image features of the medical image and basic case features of the medical case includes:

[0021] The initial image features of the medical images and the initial case features of the medical cases are extracted using a feature extraction model.

[0022] Based on the initial image features, the basic image features of the medical image are determined using a first self-attention model;

[0023] Based on the initial case characteristics, the basic case characteristics of the medical cases are determined using a second self-attention model.

[0024] In another possible implementation, determining the basic image features of the medical image based on the initial image features using a first self-attention model includes:

[0025] Convert the initial image features into features of the target dimension;

[0026] Based on the transformed initial image features, the basic image features of the medical image are determined using a first self-attention model.

[0027] The determination of the basic case characteristics of the medical cases based on the initial case characteristics using a second self-attention model includes:

[0028] The initial case features are converted into features of the target dimension;

[0029] Based on the transformed initial case features, the basic case features of the medical cases are determined using a second self-attention model.

[0030] In another possible implementation, the fusion network model is trained as follows:

[0031] We obtained multiple sample pairs labeled with actual disease outcomes and actual privacy outcomes;

[0032] For each sample pair, the first image feature of the medical image sample and the first case feature of the medical case sample in the sample pair are extracted using a feature extraction model.

[0033] The first image features and the first case features are respectively converted into features of the target dimension to obtain the converted first image features and the converted first case features;

[0034] Based on the transformed first image features, the second image features of the medical image sample are determined using a first self-attention model.

[0035] Based on the transformed first case features, the second case features of the medical case sample are determined using a second self-attention model;

[0036] Based on the second image features and the second case features, the cross-modal image features of the medical image sample are determined using the first cross-modal attention model;

[0037] Based on the second case features and the second image features, the cross-modal case features of the medical case sample are determined using the second cross-modal attention model;

[0038] Based on the cross-modal image features of the medical image samples and the cross-modal case features of the medical case samples, a fusion network model is used to determine the fusion features of the medical image samples and the medical case samples, thereby obtaining the fusion features of the sample pair;

[0039] For each sample pair, the fusion features of the sample pair are input into the disease classifier and the privacy predictor, respectively, to obtain the disease classification result of the sample pair determined by the disease classifier and the privacy prediction result of the sample pair determined by the privacy predictor.

[0040] Based on the disease classification results and actual disease results of each sample, the first loss function value of the disease classifier is determined;

[0041] Based on the privacy prediction results and actual privacy results of each sample, the second loss function value of the privacy predictor is determined;

[0042] With minimizing the first loss function value as the first training objective, the parameters of the feature extraction model, the first self-attention model, the second self-attention model, the first cross-modal attention model, the second cross-modal attention model, and the disease classifier are adjusted.

[0043] The parameters of the privacy predictor are adjusted with minimizing the second loss function value as the second training objective.

[0044] With minimizing the first loss function value and maximizing the second loss function value as the third training objective, the parameters of the fusion network model are adjusted, and the operation of extracting the first image features of the medical image samples and the first case features of the medical case samples in the sample pair using the feature extraction model is returned to be performed until the first training objective, the second training objective, and the third training objective are satisfied.

[0045] In another aspect, this application also provides a medical data processing device, comprising:

[0046] The data acquisition unit is used to acquire the user's medical images and medical records;

[0047] A feature determination unit is used to determine the target image features of the medical image and the target case features of the medical case.

[0048] The feature fusion unit is used to determine the target fusion features between the medical image and the medical case based on the target image features and the target case features using a fusion network model;

[0049] A disease diagnosis unit is used to determine a disease diagnosis result based on the target fusion features using a medical diagnosis platform;

[0050] The fusion network model used in the feature fusion unit is trained by using multiple sample pairs labeled with actual disease results and actual privacy results, with the goal of minimizing the first loss function value of the disease classifier and maximizing the second loss function value of the privacy predictor.

[0051] The first loss function value characterizes the difference between the disease classification result determined by the disease classifier based on the fusion features of the sample pair and the actual disease result labeled on the sample pair;

[0052] The second loss function value characterizes the difference between the privacy prediction result determined by the privacy predictor based on the fusion features of the sample pair and the actual privacy result labeled on the sample pair;

[0053] The sample pairs include medical image samples and medical case samples;

[0054] The fusion features of the sample pair are the fusion features determined by the fusion network model based on the image features of the medical image samples in the sample pair and the case features of the medical case samples in the sample pair.

[0055] In another possible implementation, during the process of training the fusion network model used in the feature fusion unit using multiple sample pairs, the disease classifier is trained with the goal of minimizing the first loss function value, and the privacy predictor is trained with the goal of minimizing the second loss function value.

[0056] In yet another possible implementation, the feature determination unit includes:

[0057] A feature extraction unit is used to extract basic image features of the medical images and basic case features of the medical cases.

[0058] The first cross-modal extraction unit is used to determine the cross-modal image features of the medical image based on the basic image features and basic case features using the first cross-modal attention model, and to determine the cross-modal image features as the target image features of the medical image.

[0059] The second cross-modal extraction unit is used to determine the cross-modal case features of the medical case based on the basic case features and basic image features using the second cross-modal attention model, and to determine the cross-modal case features as the target case features of the medical case.

[0060] In yet another possible implementation, the feature extraction unit includes:

[0061] The initial extraction unit is used to extract the initial image features of the medical image and the initial case features of the medical case using a feature extraction model.

[0062] The first attention processing unit is used to determine the basic image features of the medical image based on the initial image features using a first self-attention model.

[0063] The second attention processing unit is used to determine the basic case characteristics of the medical case based on the initial case characteristics using a second self-attention model.

[0064] As described above, after obtaining the patient's medical images and medical records, this application utilizes a fusion network model to fuse the image features of the medical images and the case features of the medical records to obtain fused features. Since the fusion network model is trained with the goal of minimizing the loss function value of the disease classifier and maximizing the loss function value of the privacy predictor, the fused features generated by the trained fusion network model can both accurately determine the disease diagnosis result and effectively reduce the privacy data information contained in the fused features. Therefore, in the process of determining the disease diagnosis result through the disease diagnosis platform based on the fused features, the possibility of inferring the patient's privacy data based on the fused features can be reduced, thus naturally reducing the risk of patient privacy data leakage. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0066] Figure 1 This invention illustrates a flowchart of a medical data processing method provided in an embodiment of this application.

[0067] Figure 2 This illustration shows yet another flowchart of the medical data processing method provided in an embodiment of this application;

[0068] Figure 3 This illustration shows a schematic diagram of an implementation process for extracting cross-modal features between medical images and medical records in an embodiment of this application;

[0069] Figure 4 This paper illustrates a flowchart of a model training method provided in an embodiment of this application.

[0070] Figure 5 A schematic diagram illustrating the principle of the model training method in an application scenario according to an embodiment of this application is shown;

[0071] Figure 6 A schematic diagram of the composition architecture of a medical data processing device provided in an embodiment of this application is shown. Detailed Implementation

[0072] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0073] like Figure 1 As shown, it illustrates a flowchart of a medical data processing method provided in an embodiment of this application. The method of this embodiment can be applied to a computer device, which can be any electronic device with image processing capabilities.

[0074] For example, the computer equipment can be a computer or server in a medical research institution, without restriction. Medical research institutions can be hospitals or disease research institutions that can obtain the user's medical images and medical records and need to determine the user's disease diagnosis.

[0075] The method in this embodiment may include the following steps:

[0076] S101 obtains the user's medical images and medical records.

[0077] Medical images, also known as medical pictures, are images of the internal tissues of the human body or a part of the human body obtained using medical imaging technology. For example, medical images can include one or more of the following: computed tomography (CT) scans, magnetic resonance imaging (MRI) images, and X-ray images.

[0078] Depending on the disease being diagnosed, the number of medical images obtained, the type of medical images, and the body parts included in the medical images can vary, and there are no restrictions on these.

[0079] For example, if it is necessary to analyze whether a user has Alzheimer's disease, then CT scans, MRI scans, and PET scans of the patient's brain are obtained as medical images.

[0080] Medical records can include various medical records of the user, such as: the user's electronic clinical records, scanned images of paper records, and test information obtained through medical devices, etc., without any restrictions.

[0081] In this application, medical records can also be in the form of images, or of course, in other forms, without limitation.

[0082] S102, determine the target image features of the medical image and the target case features of the medical case.

[0083] In this context, target image features are features extracted from medical images that reflect disease information. Similarly, target case features are features extracted from medical records that reflect disease information.

[0084] For ease of distinction, features extracted from medical images are called target image features, while features extracted from medical cases are called target case features.

[0085] It is understandable that there are many ways to extract features from medical images and medical records, and this application does not impose any restrictions on this.

[0086] For example, image feature extraction methods can be used to extract target image features from medical images and target case features from images of medical cases.

[0087] Of course, there are other feature extraction methods, which will be explained later, and will not be elaborated here.

[0088] S103, Based on the target image features and the target case features, a fusion network model is used to determine the target fusion features between the medical image and the medical case.

[0089] Among them, the target fusion feature is the feature after fusing the target image feature and the target case feature. For ease of distinction, this application refers to the feature after fusing the target image feature and the target case feature as the target fusion feature.

[0090] For example, the target image features and the target case features are input into a fusion network model, and the fusion network model performs feature fusion on the target image features and the target case features to obtain the target fused features.

[0091] The structure of this fusion network model can have various possibilities. For example, it can consist of multiple fully connected layers. Of course, it can also be other types of neural networks, or a combination of different neural networks; there are no restrictions on this.

[0092] In this application, the fusion network model is trained by using multiple sample pairs labeled with actual disease results and actual privacy results, with the goal of minimizing the first loss function value of the disease classifier and maximizing the second loss function value of the privacy predictor.

[0093] The first loss function value represents the difference between the disease classification result determined by the disease classifier based on the fusion features of the sample pairs and the actual disease result labeled on the sample pairs. The second loss function value represents the difference between the privacy prediction result determined by the privacy predictor based on the fusion features of the sample pairs and the actual privacy result labeled on the sample pairs.

[0094] The sample pairs include medical image samples and medical case samples. Medical image samples refer to medical images used as training samples, while medical case samples refer to medical cases used as training samples. Accordingly, the fusion features of the sample pairs are the fusion features determined by the fusion network model based on the image features of the medical image samples and the case features of the medical case samples in the sample pair.

[0095] The process of determining the fusion features between the sample pair of TCM image samples and medical case samples by the fusion network model is similar to the process of determining the target fusion features between medical images and medical cases by the fusion network model mentioned above, and will not be repeated here.

[0096] The disease classifier is used for diagnosing diseases. This classifier determines a user's disease diagnosis based on the fusion features between image features of medical images and case features of medical records. The diagnosis can indicate whether the user has a disease, whether they have a specific type of disease, or one or more categories of diseases, without limitation.

[0097] In practical applications, the specific classification task of the disease classifier will vary depending on the disease diagnosis task, and therefore, the diagnostic information represented by the disease diagnosis results will also vary. Accordingly, the actual disease results labeled on the sample pairs in this application are also related to the disease diagnosis task.

[0098] For example, taking the diagnosis of Alzheimer's disease as an example, the disease diagnosis result labeled on a sample pair can be a label representing whether a user has Alzheimer's disease. For example, the disease diagnosis result labeled on a sample pair can be: has Alzheimer's disease, or does not have Alzheimer's disease.

[0099] Correspondingly, a disease classifier can be constructed to diagnose the presence of Alzheimer's disease. By inputting the fused features determined based on the user's medical images and medical records into this disease classifier, the probability of the user having Alzheimer's disease can be output. Based on the probability of the user having Alzheimer's disease, a disease diagnosis result can be determined. For example, if the probability of the user having Alzheimer's disease exceeds 60%, the user is confirmed to have Alzheimer's disease.

[0100] This privacy predictor is used to predict a user's privacy information based on the fusion features between medical images and medical records; that is, to predict the user's privacy information contained in the fusion features. This user privacy information can involve multiple privacy items such as the user's occupation, age, and gender. Correspondingly, the privacy predictor can predict the user's value or category in at least one of these privacy items.

[0101] In practical applications, the types and specific content of the privacy prediction information included by the privacy predictor will vary depending on the specific task requiring the prediction of privacy information. Similarly, the actual privacy results of the labeled sample pairs are also related to the types of privacy information that the privacy predictor can predict.

[0102] In one possible implementation, the privacy predictor may include at least one privacy classifier, each of which is used to predict the value of a privacy item.

[0103] For example, a privacy classifier could be a classifier used to predict a user's gender. In this case, the privacy classifier determines the user's gender based on fused features. Therefore, the actual privacy result of the sample pair annotation could include the actual user's gender corresponding to the sample pair. Similarly, a privacy predictor could be a classifier used to predict a user's occupation. In this case, the privacy classifier can determine the user's occupation based on fused features. The actual privacy result of the sample pair annotation could at least include the actual user's occupation corresponding to the sample pair.

[0104] In practical applications, one or more different types of privacy classifiers can be set in the privacy predictor, depending on the privacy requirements of the project. There are no restrictions on this.

[0105] In this application, the disease classifier and privacy predictor can be a pre-trained neural network model or other machine learning model.

[0106] In one alternative approach, to ensure that the fused features generated by the fusion network model more accurately reflect the user's disease characteristics and further reduce information that reflects user privacy, the disease classifier and privacy predictor can be trained synchronously with the fusion network model in this application. Specifically, during the training of the fusion network model using multiple sample pairs, the disease classifier is trained with the goal of minimizing the first loss function value, and the privacy predictor is trained with the goal of minimizing the second loss function value.

[0107] For example, after obtaining multiple sample pairs labeled with actual disease results and actual privacy results, for each sample pair, the image features of the medical image sample and the case features of the medical case sample are determined. Then, the image features and case features of the sample pair are input into the fusion network model to be trained, obtaining the fusion features of the sample pair output by the fusion network model. For each sample pair, the fusion features of the sample pair are input into the disease classifier and privacy predictor to be trained, respectively, obtaining the disease classification result predicted by the disease classifier and the privacy prediction result determined by the privacy predictor.

[0108] Based on this, the first loss function value of the disease classifier is determined according to the disease classification results and actual disease results of each sample; the second loss function value of the privacy predictor is determined according to the privacy prediction results and actual privacy results of each sample. Accordingly, the parameters of the disease classifier are adjusted with minimizing the first loss function value as the first training objective; the parameters of the privacy predictor are adjusted with minimizing the second loss function value as the second training objective; and the parameters of the fusion network model are adjusted with minimizing the first loss function value and maximizing the second loss function value as the third training objective.

[0109] After adjusting the parameters of the privacy predictor, disease classifier, and fusion network model, you can continue to train these three models using sample pairs until all three models meet their respective training objectives.

[0110] Understandably, in this application, the fusion network model is trained with the training objective of minimizing the loss function value of the disease classifier and maximizing the loss function value of the privacy predictor. This allows the fusion network model to incorporate privacy protection processing during feature fusion. In the process of feature fusion, it considers both how to make the fused features better reflect the disease features and how to minimize or remove unnecessary user privacy information contained in the fused features.

[0111] S104. Based on the target fusion features, the disease diagnosis result is determined using a medical diagnostic platform.

[0112] Specifically, this medical diagnostic platform is a platform that uses the fusion features of user medical images and medical records provided by computer equipment to determine disease diagnosis results, and this medical diagnostic platform is a platform other than computer equipment. For example, the medical diagnostic platform can be a cloud service system capable of disease diagnosis, etc., without any restrictions.

[0113] Accordingly, after sending the identified target fusion features to the medical diagnostic platform, the disease diagnosis results returned by the medical diagnostic platform can be obtained. This application does not limit the specific process by which the medical diagnostic platform determines the disease diagnosis results.

[0114] As can be seen from step S103, the fusion network model trained in this application can construct target fusion features that can reflect disease characteristics and contain little or no user privacy. Therefore, in the process of sending the target fusion features to a medical diagnostic platform outside of computer devices, even if the target fusion features are maliciously stolen, the possibility of inferring or reconstructing user privacy information from the target fusion features can be reduced.

[0115] As described above, after obtaining the patient's medical images and medical records, this application utilizes a fusion network model to fuse the image features of the medical images and the case features of the medical records to obtain fused features. Since the fusion network model is trained with the goal of minimizing the loss function value of the disease classifier and maximizing the loss function value of the privacy predictor, the fused features generated by the trained fusion network model can both accurately determine the disease diagnosis result and effectively reduce the privacy data information contained in the fused features. Therefore, in the process of determining the disease diagnosis result through the disease diagnosis platform based on the fused features, the possibility of inferring the patient's privacy data based on the fused features can be reduced, thus naturally reducing the risk of patient privacy data leakage.

[0116] In this application, there are several possibilities for the specific implementation of the target image features of medical images and the target case features of medical cases.

[0117] In one possible implementation, in order to improve the efficiency of feature fusion in the fusion network model, this application can use a multimodal attention mechanism to perform preliminary cross-modal fusion of the two features after initially extracting the features of medical images and medical cases.

[0118] Specifically, the basic image features of the medical image and the basic case features of the medical case can be extracted first. In this step, conventional feature extraction methods can be used, or features can be extracted by combining attention mechanisms; there are no restrictions on which method is used.

[0119] Based on this, this application can determine the cross-modal image features of the medical image using a first cross-modal attention model based on basic image features and basic case features, and then identify these cross-modal image features as the target image features of the medical image. Correspondingly, based on basic case features and basic image features, a second cross-modal attention model is used to determine the cross-modal case features of the medical case, and then identify these cross-modal case features as the target case features of the medical image.

[0120] The cross-modal attention model can extract high-dimensional information from basic image features or basic case features to narrow the semantic gap between the two types of features, thereby achieving preliminary feature fusion. Specifically, the first cross-modal attention model is mainly used to integrate some features from the basic case features while retaining key features from the basic image features. The second cross-modal attention model is used to easily integrate features from the basic image cases while retaining key features from the basic case features.

[0121] The first cross-modal attention model and the second cross-modal attention model can be pre-trained, and there are no restrictions on the process of training these two models.

[0122] Furthermore, this application can also combine self-attention mechanism to extract features during the extraction of basic image features and basic case features, so as to highlight key features in medical images and medical cases, thereby further improving the efficiency of feature fusion.

[0123] The following explanation uses one implementation method as an example.

[0124] like Figure 2 As shown, it illustrates another flowchart of the medical data processing method provided in this application embodiment. The method of this embodiment may include:

[0125] S201, obtains the user's medical images and medical records.

[0126] Medical images and medical records can be found in the relevant descriptions in the preceding examples.

[0127] S202, use a feature extraction model to extract the initial image features of medical images and the initial case features of medical cases respectively.

[0128] The feature extraction model can take many forms, without any restrictions. To achieve more efficient feature extraction, the feature extraction model can be composed of a two-dimensional convolutional network and a max pooling layer.

[0129] For ease of distinction, the features extracted by the feature extraction model from medical images are called initial image features, while the features extracted by the feature extraction model from medical cases are called initial case features.

[0130] S203 converts the initial image features and initial case features into features of the target dimension, respectively, to obtain the converted initial image features and converted initial case features.

[0131] The number of dimensions in this target dimension can be set as needed.

[0132] The purpose of converting both initial image features and initial case features into features of the target dimension is to convert these two types of features into the same dimension, that is, to project these two types of features into the same feature subspace.

[0133] There are various possible methods for converting initial image features or initial case features into features of the target dimension, and this application does not impose any restrictions on this. For example, the initial image features or initial case features can be projected onto a defined feature subspace of the target dimension based on at least one fully connected network layer.

[0134] Understandably, the purpose of converting the initial image features and initial case features into features of the same dimension is to better measure the similarity between the two types of features in the future, so as to better perform preliminary feature fusion based on cross-modal attention mechanisms.

[0135] Of course, step S203 is optional. If the accuracy requirement for feature processing based on cross-modal attention mechanism is not high, this step can be omitted.

[0136] S204. Based on the transformed initial image features, the basic image features of the medical image are determined using the first self-attention model.

[0137] S205, Based on the transformed initial case characteristics, the basic case characteristics of the medical case are determined using a second self-attention model.

[0138] By using a self-attention model to extract features from initial image or case data, we can deeply mine the features in medical images or initial cases, thereby extracting more critical features for disease diagnosis and reducing attention to other non-critical features.

[0139] In this application, the self-attention model can have various structure, and there is no limitation on it.

[0140] In one possible implementation, the first self-attention model may include a self-attention network and a fully connected network layer and a normalization layer connected after the self-attention network. Based on this, the transformed initial case features are sequentially passed through the self-attention network, the fully connected network layer, and the normalization layer to obtain the basic image features. This further ensures smooth backpropagation of gradients during feature extraction and reduces feature loss caused by converting the initial image features to the target dimension (i.e., projecting the initial image features onto the same feature subspace), further ensuring the semantic integrity of the basic image features.

[0141] Correspondingly, the second self-attention model can also include a self-attention network, a fully connected network layer, and a normalization layer connected in sequence to reduce feature loss caused by converting the initial case features into features of the target dimension, and further ensure the semantic integrity of the basic case features.

[0142] In this application, the first self-attention model and the second self-attention model can be the same model, or they can be self-attention models trained separately for medical images and medical cases, without any limitation.

[0143] S206. Based on the basic image features and basic case features, the cross-modal image features of the medical image are determined using the first cross-modal attention model, and the cross-modal image features are determined as the target image features of the medical image.

[0144] S207. Based on the basic case features and basic image features, the cross-modal case features of the medical case are determined using the second cross-modal attention model, and the cross-modal case features are determined as the target case features of the medical case.

[0145] The first and second cross-modal attention models initially fuse basic case features and basic image features, so that the fused target image features contain information from the basic case features, and the fused target case features also contain information from the basic image features. This facilitates more efficient feature fusion in subsequent fusion network layers.

[0146] The first and second cross-modal attention models can have the same model structure, but their internal parameters may differ. For example, both models can include an attention-based cross-modal attention network, a fully connected layer, and a normalization layer. Correspondingly, processing the basic case features and basic image features sequentially through the cross-attention network, fully connected layer, and normalization layer before outputting ensures smooth backpropagation of gradients. This results in both the final target case features and target image features carrying information about another morphological feature, narrowing the distribution gap between the target case features and target image features within the same dimensional feature subspace.

[0147] The cross-modal attention mechanism within either of these two cross-modal attention models can be expressed as Equation 1 below:

[0148]

[0149] Here, Q and K are calculated from the two features to be fused in the current input, respectively. K is calculated based on the feature to be fused among the two features. For example, for the first cross-modal attention model, K is calculated based on the basic case features; for the second cross-modal attention model, K is calculated based on the basic image features. d is the dimension of the feature to be fused.

[0150] In this embodiment, the combination of self-attention mechanism and cross-modal attention mechanism for feature extraction not only maintains the semantic integrity of the two forms of feature representation, but also ensures that each morphological feature carries information of the other morphological feature, narrowing the distribution gap between image features and case features in the feature subspace, providing a guarantee for effective fusion, and also helping to improve feature fusion efficiency.

[0151] S208. Based on the target image features and target case features, the trained fusion network model is used to determine the target fusion features between medical images and medical cases.

[0152] For example, the target image features and target case features are stitched together, and the stitched features are input into the fusion network model to obtain the target fusion features output by the fusion network model.

[0153] S209, Based on the target fusion features, the disease diagnosis result is determined using a medical diagnostic platform.

[0154] The above steps S208 and S209 can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0155] To facilitate understanding of the process of extracting target image features and target case features in this embodiment, an application scenario is used as an example for illustration below:

[0156] Let's continue with the example of diagnosing Alzheimer's disease. In this scenario, the acquired medical images may include brain scans of the user, such as three central slices from an MRI brain scan, representing the frontal, lateral, and top views. Accordingly, to facilitate understanding of the process of extracting target image features and target case features in this embodiment, an application scenario will be used as an example below:

[0157] Let's take the Alzheimer's disease diagnosis scenario as an example. In this scenario, the obtained medical images can include brain scans of the user, such as three central slices from MRI brain scans, representing frontal, lateral, and top views. Correspondingly, a feature extraction network consisting of two 2D convolutional layers and a max-pooling layer is used to extract image features from these three images. These features are then compressed to one dimension and projected into the feature subspace using a fully connected network to obtain the brain scan image features.

[0158] Similarly, a feature extraction network is used to extract features from users' electronic clinical medical records to obtain the features of electronic clinical medical reports.

[0159] Based on the above, please refer to Figure 3 .like Figure 3 As shown in the first dashed box: brain scan image features are sequentially passed through an auto-attention layer and a fully connected layer to obtain the basic brain imaging features. Simultaneously, electronic clinical medical report features are sequentially passed through another auto-attention layer and a fully connected layer to obtain the basic case features.

[0160] Then, as Figure 3 As shown in the second dashed box, the basic image features obtained from brain scan image features after passing through the self-attention layer and fully connected layer, and the basic case features obtained from electronic clinical medical feature reports after processing through the self-attention mechanism, are sequentially input into... Figure 3 The second dashed box contains a cross-modal attention layer and a fully connected layer on the left, which integrate some information of the basic case features into the basic image features to obtain cross-modal features of brain scan images.

[0161] Similarly, the basic case features obtained from the electronic clinical medical report through self-attention processing and the basic image features obtained from the brain scan image through self-attention processing will also be sequentially input into... Figure 3 The cross-modal attention layer and fully connected layer on the right side of the second dashed box allow information from basic influence features to be incorporated into the basic case features, ultimately resulting in cross-modal features of the electronic clinical medical report.

[0162] exist Figure 3 Based on this, cross-modal features from brain scan images and electronic clinical medical reports are input into the fusion network model of this application to obtain fused features. These fused features are then processed by a medical diagnostic platform to determine whether a user has Alzheimer's disease.

[0163] It is understood that there may be multiple possible training methods for the various models mentioned above in this application, and this application does not impose any restrictions on them.

[0164] In one possible implementation, the various models involved in the above embodiments can be trained simultaneously. The model training process in this application is described below using one implementation as an example.

[0165] like Figure 4 As shown, it illustrates a flowchart of model training in this application. The method of this embodiment may include:

[0166] S401, obtained multiple sample pairs labeled with actual disease results and actual privacy results.

[0167] Each sample pair includes a medical image sample and a medical case sample from the same user.

[0168] To facilitate understanding of the training process, this embodiment still uses the example of diagnosing Alzheimer's disease, and specifically illustrates the task scenario where the user's occupation information cannot be inferred from their private information based on fusion features:

[0169] In this task scenario, for each sample pair, the medical image sample can be a brain scan image used as a training sample, such as an MRI image of the user's brain, while the medical case sample can be the user's electronic clinical medical report. Correspondingly, the actual disease result labeled on the sample pair can be the tag information indicating whether the user to whom the sample pair belongs actually has Alzheimer's disease, and the actual privacy result labeled on the sample pair can be the actual occupation of the user to whom the sample pair belongs.

[0170] S402, For each sample pair, the first image feature of the medical image sample and the first case feature of the medical case sample in the sample pair are extracted using the feature extraction model.

[0171] The possible structural forms of the feature extraction model can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0172] The process of extracting image features from medical image samples using a feature extraction model is similar to the process of extracting initial image features from medical images in the alternative embodiment above. Correspondingly, the process of extracting features from the first case is similar to the process of extracting initial case features mentioned above. For details, please refer to the relevant introduction above, which will not be repeated here.

[0173] S403, convert the first image features and the first case features into features of the target dimension respectively, to obtain the converted first image features and the converted first case features.

[0174] This step is the same as the previous process of converting the initial image features and initial case features into features of the target dimension, as detailed in the previous introduction.

[0175] S404, based on the transformed first image features, the second image features of the medical image sample are determined using the first self-attention model.

[0176] S405, based on the transformed first case features, uses a second self-attention model to determine the second case features of medical case samples.

[0177] S406, based on the second image features and the second case features, the cross-modal image features of the medical image sample are determined using the first cross-modal attention model.

[0178] S407, based on the second case features and the second image features, uses the second cross-modal attention model to determine the cross-modal case features of medical case samples.

[0179] It is understood that the above steps S404 to S407 can be referred to in the previous process of obtaining target image features based on the converted initial image features and obtaining target case features based on the converted initial case features, and will not be repeated here.

[0180] S408. Based on the cross-modal image features of medical image samples and the cross-modal case features of medical case samples, the fusion network model is used to determine the fusion features of medical image samples and medical case samples, thus obtaining the fusion features of the sample pairs.

[0181] Understandably, each sample pair needs to go through the above steps S402 to S408 to finally obtain the fusion features of each sample pair.

[0182] S409, for each sample pair, the fusion features of the sample pair are input into the disease classifier and the privacy predictor respectively to obtain the disease classification result of the sample pair determined by the disease classifier and the privacy prediction result of the sample pair determined by the privacy predictor.

[0183] For example, let's take the task scenario mentioned in step S401 as an example:

[0184] This disease classifier can be used to diagnose whether someone has Alzheimer's disease. Accordingly, the disease classification result determined by this classifier can indicate whether a user has Alzheimer's disease. This privacy predictor can be an occupation classifier that predicts a user's occupation. Accordingly, the privacy prediction result predicted by the privacy predictor can be the occupation to which a sample pair belongs.

[0185] S410, based on the disease classification results and actual disease results of each sample, determine the first loss function value of the disease classifier.

[0186] The first loss function value can reflect the difference between the disease classification result predicted by the disease classifier for the sample pair and the actual disease result of the sample pair. The smaller the difference, the more accurate the disease classification result predicted by the disease classifier.

[0187] The value of the first loss function can be calculated using the cross-entropy loss function, or it can be calculated using other loss functions without restriction.

[0188] S411, based on the privacy prediction results and actual privacy results of each sample, determine the second loss function value of the privacy predictor.

[0189] The second loss function value can reflect the difference between the privacy prediction result of the sample pair predicted by the privacy predictor and the actual privacy result of the sample pair. The smaller the difference, the more accurate the privacy information predicted by the privacy predictor.

[0190] The value of the second loss function can also be calculated based on the cross-entropy loss function, or it can be calculated based on other loss functions; there are no restrictions on this.

[0191] S412, with minimizing the first loss function value as the first training objective, adjust the parameters of the feature extraction model, the first self-attention model, the second self-attention model, the first cross-modal attention model, the second cross-modal attention model, and the disease classifier.

[0192] Understandably, the smaller the value of the first loss function, the closer the disease classification result predicted by the disease classifier is to the actual disease result of the sample pair, and the higher the disease prediction accuracy. Based on this, in order to accurately determine whether the fusion features output by the fusion network model can accurately represent disease information, it is necessary to improve the prediction accuracy of the disease classifier. Therefore, it is necessary to adjust the parameters of the individual attention models, cross-modal attention models, and disease classifier with the training objective of continuously reducing the first loss function value.

[0193] Among them, the parameters in the model mentioned above can be updated using gradient backpropagation based on the first training objective, or they can be updated in other ways without restriction.

[0194] S413, with minimizing the second loss function value as the second training objective, adjust the parameters of the privacy predictor.

[0195] Understandably, the smaller the value of the second loss function, the closer the privacy prediction result of the sample pair predicted by the privacy predictor is to the actual privacy result labeled on the sample pair, which means that the privacy predictor has a higher accuracy in predicting privacy information. However, in order to accurately determine whether the fused features generated by the fusion network model have removed privacy information to the greatest extent possible, making it impossible to recover the user's privacy information based on the fused features, the privacy predictor needs to have high prediction accuracy. Therefore, it is necessary to continuously optimize the privacy predictor to improve its accuracy in predicting privacy.

[0196] There are several ways to adjust the parameters of the privacy predictor based on the second training objective, and there are no restrictions on this. For example, gradient backpropagation can be used to update the parameters of the privacy predictor.

[0197] S414, with the third training objective being to minimize the first loss function value and maximize the second loss function value, adjust the parameters of the fusion network model.

[0198] Understandably, the fusion network model needs to ensure that the fused features accurately reflect the user's disease characteristics. Therefore, the value of the first loss function needs to be continuously reduced during training. At the same time, the fusion network model needs to ensure that the output fused features can cover fewer privacy-related features, so that the user's privacy information cannot be inferred based on the fused features. Therefore, the value of the first loss function needs to be continuously increased so that the fused features can contain less privacy information.

[0199] For example, the third training objective θ for training the fusion network model can be expressed as the following formula:

[0200] θ = argmax θ (Loss1-ρ·Loss2) (Formula 2);

[0201] Where Loss1 is the first loss function value, Loss2 is the second loss function value, 0 < ρ < 1, and the value of ρ can be set according to actual needs.

[0202] S415, if the first training objective, the second training objective, and the third training objective are not met, return to the operation of step S402 until the first training objective, the second training objective, and the third training objective are met.

[0203] Understandably, after each adjustment of the parameters of each model, steps S402 to S411 are re-executed to continuously optimize each model, ultimately ensuring that each training objective is met and model training is completed.

[0204] For example, taking the scenario of diagnosing Alzheimer's disease and predicting a user's occupation using a privacy predictor, as mentioned in step S401, the simple principle of the model training process in this embodiment can be explained as follows: Figure 5 As shown.

[0205] exist Figure 5 The encoder in relation to the feature extraction model. Figure 5 In this context, utility requirement can refer to the target task classifier's requirements. In this example, the target task classifier is to predict disease categories, i.e., it serves as a disease classifier. Privacy requirement refers to the specific requirements for predicting privacy. In this example, predicting occupation can be considered a privacy prediction requirement.

[0206] Depend on Figure 5 It can be intuitively seen that the brain scan image and electronic clinical medical report in each sample pair are sequentially processed by the encoder to extract features, and then sequentially passed through the self-attention layer, the cross-modal attention layer, and finally into the fusion network layer. After the fusion network layer outputs the fused features of the sample pair, these fused features are then input into the privacy predictor (…). Figure 5Privacy variable classifiers and disease classifiers Figure 5 Based on the target task classifier, and the predicted disease output of the target task classifier and the actual disease results of the sample pairs labeled with the actual disease, the first loss function value Loss1 can be calculated. Correspondingly, based on the user occupation information of the sample pairs predicted by the privacy variable classifier and the actual occupation information of the sample pairs labeled with the actual occupation, the second loss function value Loss2 can be calculated. Accordingly, the optimization objective θ = argmax of the fusion network layer can also be determined. θ (Loss1-ρ·Loss2).

[0207] Based on the above, and by combining various optimization objectives, continuous training can be achieved. Figure 5 Each model is used to ultimately obtain the trained models.

[0208] It is understood that the model training in this embodiment is illustrated by simultaneously training the fusion network model, feature extraction model, self-attention model, and cross-modal attention model. In practical applications, the self-attention model and cross-modal attention model can also be pre-trained. Based on this, in this embodiment, the parameters of the feature extraction model, the individual attention models, and the cross-modal attention model can be kept constant, while only the parameters of the disease classifier, the fusion network model, and the privacy predictor are adjusted. For example, in step S412, only the parameters of the disease classifier are adjusted, without adjusting the parameters of the other models.

[0209] Corresponding to the medical data processing method of this application, this application also provides a medical data processing device.

[0210] like Figure 6 The diagram illustrates a structural composition of a medical data processing device provided in an embodiment of this application. The device in this embodiment may include:

[0211] The data acquisition unit 601 is used to acquire the user's medical images and medical records;

[0212] Feature determination unit 602 is used to determine the target image features of the medical image and the target case features of the medical case.

[0213] The feature fusion unit 603 is used to determine the target fusion features between the medical image and the medical case based on the target image features and the target case features using a fusion network model;

[0214] The disease diagnosis unit 604 is used to determine the disease diagnosis result based on the target fusion features using a medical diagnosis platform;

[0215] The fusion network model used in the feature fusion unit is trained by using multiple sample pairs labeled with actual disease results and actual privacy results, with the goal of minimizing the first loss function value of the disease classifier and maximizing the second loss function value of the privacy predictor.

[0216] The first loss function value characterizes the difference between the disease classification result determined by the disease classifier based on the fusion features of the sample pair and the actual disease result labeled on the sample pair;

[0217] The second loss function value characterizes the difference between the privacy prediction result determined by the privacy predictor based on the fusion features of the sample pair and the actual privacy result labeled on the sample pair;

[0218] The sample pairs include medical image samples and medical case samples;

[0219] The fusion features of the sample pair are the fusion features determined by the fusion network model based on the image features of the medical image samples in the sample pair and the case features of the medical case samples in the sample pair.

[0220] In one possible implementation, during the process of training the fusion network model used in the feature fusion unit using multiple sample pairs, the disease classifier is trained with the goal of minimizing the first loss function value, and the privacy predictor is trained with the goal of minimizing the second loss function value.

[0221] In yet another possible implementation, the feature determining unit includes:

[0222] A feature extraction unit is used to extract basic image features of the medical images and basic case features of the medical cases.

[0223] The first cross-modal extraction unit is used to determine the cross-modal image features of the medical image based on the basic image features and basic case features using the first cross-modal attention model, and to determine the cross-modal image features as the target image features of the medical image.

[0224] The second cross-modal extraction unit is used to determine the cross-modal case features of the medical case based on the basic case features and basic image features using the second cross-modal attention model, and to determine the cross-modal case features as the target case features of the medical case.

[0225] In yet another possible implementation, the feature extraction unit includes:

[0226] The initial extraction unit is used to extract the initial image features of the medical image and the initial case features of the medical case using a feature extraction model.

[0227] The first attention processing unit is used to determine the basic image features of the medical image based on the initial image features using a first self-attention model.

[0228] The second attention processing unit is used to determine the basic case characteristics of the medical case based on the initial case characteristics using a second self-attention model.

[0229] In yet another possible implementation, the first attention processing unit includes:

[0230] The first dimension transformation subunit is used to transform the initial image features into features of the target dimension.

[0231] The first attention processing subunit is used to determine the basic image features of the medical image based on the transformed initial image features using a first self-attention model.

[0232] The second attention processing unit includes:

[0233] The second dimension transformation subunit is used to transform the initial case features into features of the target dimension.

[0234] The second attention processing unit is used to determine the basic case features of the medical case based on the transformed initial case features using a second self-attention model.

[0235] In another possible implementation, the device further includes a model training unit for training the fusion network model in the following manner:

[0236] We obtained multiple sample pairs labeled with actual disease outcomes and actual privacy outcomes;

[0237] For each sample pair, the first image feature of the medical image sample and the first case feature of the medical case sample in the sample pair are extracted using a feature extraction model.

[0238] The first image features and the first case features are respectively converted into features of the target dimension to obtain the converted first image features and the converted first case features;

[0239] Based on the transformed first image features, the second image features of the medical image sample are determined using a first self-attention model.

[0240] Based on the transformed first case features, the second case features of the medical case sample are determined using a second self-attention model;

[0241] Based on the second image features and the second case features, the cross-modal image features of the medical image sample are determined using the first cross-modal attention model;

[0242] Based on the second case features and the second image features, the cross-modal case features of the medical case sample are determined using the second cross-modal attention model;

[0243] Based on the cross-modal image features of the medical image samples and the cross-modal case features of the medical case samples, a fusion network model is used to determine the fusion features of the medical image samples and the medical case samples, thereby obtaining the fusion features of the sample pair;

[0244] For each sample pair, the fusion features of the sample pair are input into the disease classifier and the privacy predictor, respectively, to obtain the disease classification result of the sample pair determined by the disease classifier and the privacy prediction result of the sample pair determined by the privacy predictor.

[0245] Based on the disease classification results and actual disease results of each sample, the first loss function value of the disease classifier is determined;

[0246] Based on the privacy prediction results and actual privacy results of each sample, the second loss function value of the privacy predictor is determined;

[0247] With minimizing the first loss function value as the first training objective, the parameters of the feature extraction model, the first self-attention model, the second self-attention model, the first cross-modal attention model, the second cross-modal attention model, and the disease classifier are adjusted.

[0248] The parameters of the privacy predictor are adjusted with minimizing the second loss function value as the second training objective.

[0249] With minimizing the first loss function value and maximizing the second loss function value as the third training objective, the parameters of the fusion network model are adjusted, and the operation of extracting the first image features of the medical image samples and the first case features of the medical case samples in the sample pair using the feature extraction model is returned to be performed until the first training objective, the second training objective, and the third training objective are satisfied.

[0250] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Furthermore, the features described in the various embodiments of this specification can be substituted or combined with each other, enabling those skilled in the art to implement or use this application. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0251] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0252] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0253] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A medical data processing method, characterized by, include: Obtain the user's medical images and medical records; Determine the target image features of the medical images and the target case features of the medical cases; Based on the target image features and the target case features, a fusion network model is used to determine the target fusion features between the medical image and the medical case; Based on the aforementioned target fusion features, a medical diagnostic platform is used to determine the disease diagnosis result; The fusion network model is trained in the following manner: Multiple sample pairs were obtained, labeled with actual disease results and actual privacy results. The sample pairs included medical image samples and medical case samples. For each sample pair, the first image feature of the medical image sample and the first case feature of the medical case sample in the sample pair are extracted using a feature extraction model. The first image features and the first case features are respectively converted into features of the target dimension to obtain the converted first image features and the converted first case features; Based on the transformed first image features, the second image features of the medical image sample are determined using a first self-attention model. Based on the transformed first case features, the second case features of the medical case sample are determined using a second self-attention model; Based on the second image features and the second case features, the cross-modal image features of the medical image sample are determined using the first cross-modal attention model; Based on the second case features and the second image features, the cross-modal case features of the medical case sample are determined using the second cross-modal attention model; Based on the cross-modal image features of the medical image samples and the cross-modal case features of the medical case samples, a fusion network model is used to determine the fusion features of the medical image samples and the medical case samples, thereby obtaining the fusion features of the sample pair; For each sample pair, the fusion features of the sample pair are input into the disease classifier and the privacy predictor, respectively, to obtain the disease classification result of the sample pair determined by the disease classifier and the privacy prediction result of the sample pair determined by the privacy predictor. Based on the disease classification results and actual disease results of each sample, the first loss function value of the disease classifier is determined; Based on the privacy prediction results and actual privacy results of each sample, the second loss function value of the privacy predictor is determined; With minimizing the first loss function value as the first training objective, the parameters of the feature extraction model, the first self-attention model, the second self-attention model, the first cross-modal attention model, the second cross-modal attention model, and the disease classifier are adjusted. The parameters of the privacy predictor are adjusted with minimizing the second loss function value as the second training objective. With minimizing the first loss function value and maximizing the second loss function value as the third training objective, the parameters of the fusion network model are adjusted, and the operation of extracting the first image features of the Chinese medical image samples and the first case features of the Chinese medical case samples of the sample pair using the feature extraction model is returned to be performed until the first training objective, the second training objective and the third training objective are satisfied. Wherein, the first loss function value characterizes the difference between the disease classification result determined by the disease classifier based on the fusion features of the sample pair and the actual disease result labeled on the sample pair; The second loss function value characterizes the gap between the privacy prediction result determined by the privacy predictor based on the fusion features of the sample pair and the actual privacy result labeled on the sample pair.

2. The method of claim 1, wherein, The determination of the target image features of the medical image and the target case features of the medical case includes: Extract the basic image features of the medical images and the basic case features of the medical cases; Based on the basic image features and basic case features, the cross-modal image features of the medical image are determined using the first cross-modal attention model, and the cross-modal image features are determined as the target image features of the medical image. Based on the basic case features and basic image features, the cross-modal case features of the medical case are determined using the second cross-modal attention model, and the cross-modal case features are determined as the target case features of the medical case.

3. The method of claim 2, wherein, The extraction of basic image features from the medical images and basic case features from the medical cases includes: The initial image features of the medical images and the initial case features of the medical cases are extracted using a feature extraction model. Based on the initial image features, the basic image features of the medical image are determined using a first self-attention model; Based on the initial case characteristics, the basic case characteristics of the medical cases are determined using a second self-attention model.

4. The method of claim 3, wherein, The step of determining the basic image features of the medical image based on the initial image features using a first self-attention model includes: Convert the initial image features into features of the target dimension; Based on the transformed initial image features, the basic image features of the medical image are determined using a first self-attention model. The determination of the basic case characteristics of the medical cases based on the initial case characteristics using a second self-attention model includes: The initial case features are converted into features of the target dimension; Based on the transformed initial case features, the basic case features of the medical cases are determined using a second self-attention model.

5. A medical data processing apparatus characterized by comprising: include: The data acquisition unit is used to acquire the user's medical images and medical records; A feature determination unit is used to determine the target image features of the medical image and the target case features of the medical case. The feature fusion unit is used to determine the target fusion features between the medical image and the medical case based on the target image features and the target case features using a fusion network model; A disease diagnosis unit is used to determine a disease diagnosis result based on the target fusion features using a medical diagnosis platform; The fusion network model is trained in the following manner: Multiple sample pairs were obtained, labeled with actual disease results and actual privacy results. The sample pairs included medical image samples and medical case samples. For each sample pair, the first image feature of the medical image sample and the first case feature of the medical case sample in the sample pair are extracted using a feature extraction model. The first image features and the first case features are respectively converted into features of the target dimension to obtain the converted first image features and the converted first case features; Based on the transformed first image features, the second image features of the medical image sample are determined using a first self-attention model. Based on the transformed first case features, the second case features of the medical case sample are determined using a second self-attention model; Based on the second image features and the second case features, the cross-modal image features of the medical image sample are determined using the first cross-modal attention model; Based on the second case features and the second image features, the cross-modal case features of the medical case sample are determined using the second cross-modal attention model; Based on the cross-modal image features of the medical image samples and the cross-modal case features of the medical case samples, a fusion network model is used to determine the fusion features of the medical image samples and the medical case samples, thereby obtaining the fusion features of the sample pair; For each sample pair, the fusion features of the sample pair are input into the disease classifier and the privacy predictor, respectively, to obtain the disease classification result of the sample pair determined by the disease classifier and the privacy prediction result of the sample pair determined by the privacy predictor. Based on the disease classification results and actual disease results of each sample, the first loss function value of the disease classifier is determined; Based on the privacy prediction results and actual privacy results of each sample, the second loss function value of the privacy predictor is determined; With minimizing the first loss function value as the first training objective, the parameters of the feature extraction model, the first self-attention model, the second self-attention model, the first cross-modal attention model, the second cross-modal attention model, and the disease classifier are adjusted. The parameters of the privacy predictor are adjusted with minimizing the second loss function value as the second training objective. With minimizing the first loss function value and maximizing the second loss function value as the third training objective, the parameters of the fusion network model are adjusted, and the operation of extracting the first image features of the Chinese medical image samples and the first case features of the Chinese medical case samples of the sample pair using the feature extraction model is returned to be performed until the first training objective, the second training objective and the third training objective are satisfied. Wherein, the first loss function value characterizes the difference between the disease classification result determined by the disease classifier based on the fusion features of the sample pair and the actual disease result labeled on the sample pair; The second loss function value characterizes the gap between the privacy prediction result determined by the privacy predictor based on the fusion features of the sample pair and the actual privacy result labeled on the sample pair.

6. The apparatus of claim 5, wherein, The feature determination unit includes: A feature extraction unit is used to extract basic image features of the medical images and basic case features of the medical cases. The first cross-modal extraction unit is used to determine the cross-modal image features of the medical image based on the basic image features and basic case features using the first cross-modal attention model, and to determine the cross-modal image features as the target image features of the medical image. The second cross-modal extraction unit is used to determine the cross-modal case features of the medical case based on the basic case features and basic image features using the second cross-modal attention model, and to determine the cross-modal case features as the target case features of the medical case.

7. The apparatus of claim 6, wherein, The feature extraction unit includes: The initial extraction unit is used to extract the initial image features of the medical image and the initial case features of the medical case using a feature extraction model. The first attention processing unit is used to determine the basic image features of the medical image based on the initial image features using a first self-attention model. The second attention processing unit is used to determine the basic case features of the medical case based on the initial case features using a second self-attention model.

Images