Medical data fusion method and system based on multimodal feature filling

Abstract

The present invention belongs to the field of feature fusion technology, and discloses a medical data fusion method and system based on multimodal feature filling. It includes converting text features and CT image features in multimodal medical data into fused structured features, and forming a first data set with structured electronic medical record features; filling missing features of structured electronic medical records in the first data set to obtain a second data set; screening patients with complete multimodal medical data from the second data set as the first complete data set, and patients with missing text features and / or CT image features as the first missing data set; using the average text features and average CT image features in the first complete data set to calculate the text features and CT image features in the first missing data set, respectively, to achieve feature filling; using contrastive learning and subspace clustering to align the filled multimodal medical data at the modal level and feature level, respectively, and obtain multimodal representation after feature fusion and feature selection.

Description

Technical Field

[0001] The present invention relates to the technical field of feature fusion, and in particular to a medical data fusion method and system based on multimodal feature filling. Background Art

[0002] With the continuous advancement of medical measurement technology, we can now collect patient data from a variety of sensors and information sources, such as demographic data, treatment data, imaging data, etc. Multimodal data fusion has been widely used in various fields because it can make full use of the complementary information between different modalities to reduce the impact of noise in single modality data. In the field of medical information, multimodal data fusion plays a huge role in improving disease diagnosis, treatment and monitoring, as well as promoting early detection of diseases.

[0003] The current multimodal medical data fusion methods have the following shortcomings. First: Existing multimodal medical data fusion methods often require that each modality of the patient is complete and there must be no missing features; however, in actual medical scenarios, it is very common for a certain feature of the patient to be missing; in the case of missing data, it is difficult to implement traditional multimodal fusion methods to perform medical data fusion. Second: Current multimodal medical data fusion methods often use contrastive learning, adversarial learning and other methods to perform modal alignment, but these methods only perform modal alignment from the modal level, without considering alignment at a finer-grained feature level. Third: Current multimodal medical data fusion is often performed in a simple way (such as direct splicing, linear weighting, vector inner product, etc.), and fails to fully explore the correlation between modalities, which may lead to the generation of redundant features and the loss of important features.

[0004] In view of this, the present invention is proposed. Summary of the invention

[0005] The present invention aims to solve at least one of the above technical problems and provides a medical data fusion method and system based on multimodal feature filling.

[0006] To achieve the above objectives, the present invention provides the following technical solutions.

[0007] In a first aspect, the present invention provides a medical data fusion method based on multimodal feature filling, comprising:

[0008] Acquire multimodal medical data of a plurality of patients, wherein the multimodal medical data includes structured electronic medical record features, text features, and CT image features of each patient;

[0009] Converting text features and CT image features in the multimodal medical data into fused structured features, and forming a first data set with the structured electronic medical record features;

[0010] The MissForest algorithm is used to fill in the missing features of the structured electronic medical records in the first dataset to obtain the second dataset;

[0011] Screening out patients with complete multimodal medical data from the second data set as a first complete data set, and screening out patients with missing text features and / or CT image features as a first missing data set;

[0012] Based on the idea of ​​KNN algorithm, the average text features and average CT image features in the first complete data set are used to calculate the text features and CT image features in the first missing data set respectively, and the filled multimodal medical data is obtained;

[0013] Using contrastive learning and subspace clustering, the padded multimodal medical data are modally aligned at the modality level and feature level, and the aligned representation vectors are fused to obtain the first fused representation vector.

[0014] Feature selection is performed on the first fused representation vector to obtain a multimodal representation.

[0015] Preferably, the method for converting text features and CT image features in multimodal medical data into fused structured features is:

[0016] Screening out patients with complete multimodal medical data from the multimodal medical data as a second complete data set, and screening out patients with missing structured electronic medical record features as a second missing data set;

[0017] Input the text features and CT image features in the second missing data set into the pre-trained autoencoder to obtain a text representation vector and a CT image representation vector, and merge them into a second fusion representation vector;

[0018] The second fused representation vector is used to replace the original text features and CT image features in the second missing data set to obtain fused structured features.

[0019] Preferably, the method of using the average text features and the average CT image in the first complete data set to respectively calculate the text features and the CT image features in the first missing data set is:

[0020] Inputting the structured electronic medical record features in the first complete data set and the first missing data set into the pre-trained autoencoder to obtain corresponding structured electronic medical record representation vectors;

[0021] For each patient in the first missing data set, find the K patients in the first complete data set that are most similar to the structured electronic medical record representation vector of the patient, where K is a hyperparameter;

[0022] Inputting the text features and CT image features of the K patients into a pre-trained autoencoder to obtain a text representation vector and a CT image representation vector, and calculating the average text representation vector and the average CT image representation vector of the K patients;

[0023] The missing text representation vector and CT image representation vector in the first missing data set are respectively initialized using the average text representation vector and the average CT image representation vector, and the missing text representation vector and CT image representation vector in the first missing data set are input into the pre-trained autoencoder for iteration until the mean square error between the representation vector this time and the previous representation vector is less than a threshold, and finally the converged representation vector is input into the decoder in the pre-trained autoencoder to obtain the text features and CT image features in the first missing data set.

[0024] Preferably, the method of using contrastive learning to align the padded multimodal medical data at the modality level is:

[0025] Calculate the cosine similarity between the three modal features of each patient in the filled multimodal medical data, and the cosine similarity between the three modal features of each patient and the three modal features of other patients;

[0026] The contrastive learning loss function is calculated based on all the cosine similarities obtained, and the representation variables of the padded multimodal medical data are aligned through back propagation.

[0027] Preferably, the method of using subspace clustering to align the padded multimodal medical data at the feature level is:

[0028] Input the padded multimodal medical data into the pre-trained autoencoder to obtain the representation vectors of the three modal features;

[0029] The representation vectors of the three modal features are evenly divided into N subspace representation vectors respectively;

[0030] Perform k-means clustering on the 3N subspace representation vectors to obtain the clustering results of 3N subspaces;

[0031] Calculate the cosine similarity between the clustering result of each subspace of each modal feature and the clustering result of each subspace of other modal features;

[0032] According to the cosine similarity of the clustering results, the subspace closest to the clustering result of this subspace is selected from the other two modal features, and the representation vector of this subspace is replaced with the average value of the representation vectors of these three subspaces. After all 3N subspaces are replaced, modal alignment is achieved.

[0033] Preferably, the method for fusing the aligned representation vectors to obtain the first fused representation vector is:

[0034] The modality-level aligned feature vectors and the feature-level aligned feature vectors are concatenated column by column to obtain a concatenated vector representation;

[0035] Generate modality-specific representation vectors of the three modalities and modality-shared representation vectors of the three modalities respectively according to the concatenated vector representations;

[0036] The modality-specific representation vector and the modality-shared representation vector are pulled into one-dimensional vectors and then concatenated to obtain a first fused representation vector.

[0037] Preferably, the modality-specific representation vector and the modality-shared representation vector calculate the following loss function:

[0038]

[0039] Among them, H sp is the modality-specific representation vector, H sh is the modality sharing representation vector, To calculate the two-norm operation.

[0040] Preferably, the method of performing feature selection on the first fusion representation vector to obtain a multimodal representation is:

[0041] The clustering result of each patient obtained by the subspace clustering is used as a label;

[0042] Based on the Person correlation coefficient, the first important feature set in the first fusion representation vector is screened out;

[0043] Taking the first fused representation vector as input, and screening out a second important feature set by a chi-square test according to the label;

[0044] The first fused representation vector is used as an input to the random forest model, and the third most important feature set whose feature importance ranks in the top 50% is selected according to the label;

[0045] The first important feature set, the second important feature set, and the third important feature set are unioned to obtain a multimodal representation.

[0046] In a second aspect, the present invention provides a data fusion system based on multimodal feature filling, comprising:

[0047] An acquisition module, used to acquire multimodal medical data of multiple patients, wherein the multimodal medical data includes structured electronic medical record features, text features, and CT image features of each patient;

[0048] A conversion module, used for converting text features and CT image features in the multimodal medical data into fused structured features, and forming a first data set with the structured electronic medical record features;

[0049] A first filling module is used to fill in the missing features of the structured electronic medical records in the first data set using the MissForest algorithm to obtain a second data set;

[0050] A screening module, used to screen out patients with complete multimodal medical data from the second data set as a first complete data set, and screen out patients with missing text features and / or CT image features as a first missing data set;

[0051] The second filling module is used to calculate the text features and CT image features in the first missing data set respectively based on the idea of ​​the KNN algorithm by using the average text features and the average CT image features in the first complete data set to obtain the filled multimodal medical data;

[0052] An alignment and fusion module, used to align the padded multimodal medical data at the modality level and feature level respectively by using contrastive learning and subspace clustering, and to fuse the aligned representation vectors to obtain a first fused representation vector;

[0053] The feature selection module is used to perform feature selection on the first fusion representation vector to obtain a multimodal representation.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention can fill in the missing features in multimodal medical data, improve the accuracy of multimodal medical data fusion, and has a wider range of application scenarios.

[0056] The present invention performs modal alignment at both the modal level and the feature level, that is, mining the correlation between modalities at two levels. Compared with the traditional method of performing modal alignment only at the modal level, the present invention can obtain more accurate data representation.

[0057] The present invention provides a hybrid unsupervised and supervised feature selection algorithm to screen features, thereby reducing the feature dimension of multimodal representation and reducing the computational pressure of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a medical data fusion method based on multimodal feature filling provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of the structure of an autoencoder that can be used in an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of a medical data fusion system based on multimodal feature filling provided by an embodiment of the present invention;

[0061] Figure 4 FIG. 1 is a schematic block diagram of an exemplary electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The first embodiment of the present invention provides a medical data fusion method based on multimodal feature filling. Figure 1 A flowchart of a medical data fusion method based on multimodal feature filling provided in an embodiment of the present application.

[0064] like Figure 1 As shown, the medical data fusion method based on multimodal feature filling specifically includes the following steps:

[0065] S101, acquiring multimodal medical data of a plurality of patients, wherein the multimodal medical data includes structured electronic medical record features, text features, and CT image features of each patient.

[0066] Specifically, the patient's structured electronic medical record features generally include the patient's demographics, pathology, diagnosis and treatment information. The text features generally include information such as medical advice and CT text reports. The CT image features generally include information such as the grayscale and texture of the patient's diseased tissue. In actual medical scenarios, due to the large number of patients, certain features of the patients are usually missing. In order to achieve accurate fusion of multimodal medical data, the implementation method of the present invention first fills in the missing features.

[0067] S102, converting text features and CT image features in the multimodal medical data into fused structured features, and forming a first data set with the structured electronic medical record features.

[0068] The purpose of this step is to convert the text features and CT image features in the multimodal medical data into format features that can be processed by the MissForest algorithm in step S103. The specific method of this step includes:

[0069] S102-1, screening out patients with complete multimodal medical data from the multimodal medical data as a second complete data set, and screening out patients with missing structured electronic medical record features as a second missing data set.

[0070] S102-2, input the text features and CT image features in the second missing data set into the pre-trained autoencoder to obtain a text representation vector and a CT image representation vector, and merge them into a second fused representation vector.

[0071] The pre-trained autoencoder is obtained by training the autoencoder using the second complete data set. The loss function in the training process is in, and They are structured electronic medical record features, text features, and CT image features. and They are the structured electronic medical record features, reconstructed text features, and reconstructed CT image features reconstructed by the autoencoder during training.

[0072] The structure of the autoencoder used in the embodiments of the present invention can be designed by those skilled in the art according to actual functional requirements. The structures that can be referred to in the embodiments of the present invention are as follows: Figure 2 shown.

[0073] For each patient in the second missing data set, its text features and CT image features are input into the pre-trained autoencoder to obtain text representation vectors with lengths of 512. and CT image representation vector The text representation vector and the CT image representation vector of the patient are combined to obtain a second fused representation vector.

[0074] It is understandable that, since the patient's text features and CT image features may be missing, the merged second fusion representation vector may have the following four situations:

[0075] Case 1: Both the patient's text features and CT image features are present. In this case, the second fusion representation vector x tfCT for:

[0076]

[0077] Case 2: The patient's text features are not missing, but the CT image features are missing. In this case, the second fusion representation vector x tfCT for:

[0078]

[0079] Case 3: The patient’s text features are not missing, but the CT image features are missing. In this case, the second fusion representation vector x tfCT for:

[0080]

[0081] Case 4: Both the patient's text features and CT image features are missing. In this case, the second fusion representation vector x tfCT for:

[0082] x tfCT =0 512 ,

[0083] Among them, 0 512 Represents a vector of length 512 that is all zeros.

[0084] S102-3, using the second fused representation vector to replace the original text features and CT image features in the second missing data set to obtain fused structured features. At this time, the fused structured features and the structured electronic medical record features constitute the first data set, that is, the features in the first data set have all become structured data.

[0085] S103, using the MissForest algorithm to fill in the missing features of the structured electronic medical records in the first data set to obtain a second data set. At this point, the structured electronic medical record features of the patients in the second data set have been filled in.

[0086] S104, screening out patients with complete multimodal medical data from the second data set as a first complete data set, and screening out patients with missing text features and / or CT image features as a first missing data set.

[0087] S105, based on the idea of ​​KNN algorithm, the text features and CT image features in the first missing data set are respectively calculated using the average text features and the average CT image features in the first complete data set to obtain the filled multimodal medical data.

[0088] The purpose of this step is to fill in the missing text features and CT image features of the patient. Specifically, this step includes:

[0089] S105-1, inputting the structured electronic medical record features in the first complete data set and the first missing data set into a pre-trained autoencoder to obtain corresponding structured electronic medical record representation vectors.

[0090] It can be understood that the structured electronic medical record features in the first complete data set and the structured electronic medical record features in the first missing data set are respectively input into the pre-trained autoencoder to generate two structured electronic medical record representation vectors with a length of 512 each.

[0091] S105-2, for each patient in the first missing data set, find K patients in the first complete data set whose structured electronic medical record representation vectors are most similar to the patient, where K is a hyperparameter.

[0092] Using cosine similarity as a similarity measure, the K patients are found for each patient in the first missing data set, where K can be set by a person skilled in the art based on experience. It is understandable that the K patients are patients with complete multimodal medical data. The calculation formula for the cosine similarity of vector a and vector b is:

[0093] S105-3, input the text features and CT image features of the K patients into the pre-trained autoencoder to obtain a text representation vector and a CT image representation vector, and calculate the average text representation vector and the average CT image representation vector of the K patients.

[0094] The text features and CT image features of the K patients with complete multimodal medical data that are most similar to the patient are input into the pre-trained autoencoder to generate text representation (x text512 ) and CT imaging characterization (x CT512 ). The average text representation of K patients (x text_avg ) and average CT imaging characteristics (x CT_avg ), the calculation formula is as follows:

[0095]

[0096]

[0097] S105-4, use the average text representation vector and the average CT image representation vector to respectively initialize the missing text representation vector and CT image representation vector in the first missing data set, and input the missing text representation vector and CT image representation vector in the first missing data set into the pre-trained autoencoder for iteration until the mean square error between this representation vector and the previous representation vector is less than a threshold value, and finally input the converged representation vector into the decoder in the pre-trained autoencoder to obtain the text features and CT image features in the first missing data set.

[0098] It is understandable that if the text features or CT image features of some patients in the first missing data set are not missing, there is no need to fill in the missing features, and only the cases where the text features and / or CT image features of the patients are missing are replaced and filled in.

[0099] The specific implementation steps are as follows: (1) If the patient x in the first missing data set k If the text feature of patient x is missing, then initialize k The text representation vector For x text_avg , otherwise jump to (6);

[0100] (2) Input into the decoder of the pre-trained autoencoder to obtain the decoded text features

[0101]

[0102] (3) The decoded text features Input into the pre-trained autoencoder to get a new text representation vector again

[0103] (4) Calculation and Mean square error MSE text , the calculation formula is:

[0104]

[0105] (5) If MSE text Less than MSE thr , then jump to (6), otherwise Assign value to patient x k The text representation vector Jump to (2); where MSE thr is a hyperparameter and can be set by those skilled in the art based on experience;

[0106] (6) If the first missing data set contains patient x k If the CT image features of the patient are not missing, jump to (11), otherwise initialize the patient x k CT characterization vector For x CT_avg ;

[0107] (7) Input into the decoder of the pre-trained autoencoder to obtain the decoded CT image features

[0108] (8) Decoded CT image features Input to the pre-trained autoencoder to get a new CT representation vector again

[0109] (9) Calculation and Mean square error MSE CT , the calculation formula is:

[0110] (10) If MSE CT Less than MSE thr , jump to (6), otherwise Assign to and jump to (6);

[0111] (11) Complete the filling of all missing data of the patient and obtain the filled multimodal medical data.

[0112] S106, using contrastive learning and subspace clustering to perform modality alignment on the padded multimodal medical data at the modality level and feature level respectively, and fusing the aligned representation vectors to obtain a first fused representation vector.

[0113] The embodiment of the present invention can obtain more accurate data representation by aligning at the modality level and the feature level. This step specifically includes the following aspects:

[0114] S106-1, specifically, the method of using contrastive learning to perform modal alignment on the padded multimodal medical data at the modal level is as follows: calculating the cosine similarities between the three modal features of each patient in the padded multimodal medical data, and the cosine similarities between the three modal features of each patient and the three modal features of other patients; calculating the contrastive learning loss function based on all the cosine similarities obtained, and aligning the representation variables of the padded multimodal medical data through back propagation.

[0115] Understandably, the following six cosine similarities need to be calculated:

[0116] (1) The cosine similarity between each patient’s structured electronic medical record features and text features cmtext , the calculation formula is as follows:

[0117] in, They are the structured electronic medical record features and text features in the filled multimodal medical data, and N is the number of patients.

[0118] (2) The cosine similarity between each patient’s structured electronic medical record features and CT image features cmCT , the calculation formula is as follows:

[0119] in, CT image features in multimodal medical data after filling.

[0120] (3) The cosine similarity between each patient’s text features and CT image features textCT , the calculation formula is as follows:

[0121]

[0122] (4) The cosine similarity between each patient’s structured electronic medical record features and the text features of other patients cmOtext , the calculation formula is as follows:

[0123]

[0124] (5) The cosine similarity between each patient’s structured electronic medical record features and the CT image features of other patients cmOCT , the calculation formula is as follows:

[0125]

[0126] (6) The cosine similarity between each patient’s text features and the CT image features of other patients textOCT , the calculation formula is as follows:

[0127]

[0128] After obtaining the above cosine similarity, the contrastive learning loss function is calculated, and the representation variables of the filled structured electronic medical record features, text features, and CT image features are aligned through back propagation. The contrastive learning loss is:

[0129] L c =max{0,M-|L p |+|L n |},

[0130] Among them, L p =sim cmtext +sim cmCT +sim textCT , L n =sim cmOtext +sim cmOCT +sim textOCT , M is a hyperparameter and is set by technicians in this field based on practical experience.

[0131] Through the above steps, the modal alignment of the modal level is achieved, and the representation vector of the modal level alignment is obtained.

[0132] S106-2, a method for using subspace clustering to perform modal alignment on the padded multimodal medical data at the modal feature level is as follows: input the padded multimodal medical data into a pre-trained autoencoder to obtain representation vectors of three modal features; respectively divide the representation vectors of the three modal features evenly into N subspace representation vectors; perform k-means clustering on the 3N subspace representation vectors to obtain clustering results of 3N subspaces; calculate the cosine similarity between the clustering result of each subspace of each modal feature and the clustering result of each subspace of other modal features; select the subspace closest to the clustering result of the subspace from the other two modal features according to the cosine similarity of the clustering results, and replace the representation vector of the subspace with the average value of the representation vectors of the three subspaces; after the replacement of the 3N subspaces, the modal alignment of the feature level is achieved to obtain the representation vectors aligned at the feature level.

[0133] Specifically, the padded multimodal medical data is input into the pre-trained autoencoder to obtain structured electronic medical record representation vectors, text representation vectors, and CT image representation vectors of length 512; the structured electronic medical record representation vectors, text representation vectors, and CT image representation vectors are divided into 4 (N is 4 here only for the sake of convenience, and N can be set according to actual needs) subspace representations of length 128, respectively, and are represented by symbols x mcs ,x texts and x CTs Indicates; symbol x mcs [1] represents the first subspace representation of structured electronic medical records, x mcs [2] represents the second subspace representation of structured electronic medical records, and so on. texts and x CTs The same goes for this.

[0134] x mcs ,x texts and x CTs These 12 subspaces are clustered by k-means, and the clustering results of 12 subspaces are obtained, which are symbolized by y ks Indicates; where the number of clusters k in k-means clustering nums It is a hyperparameter and can be set by technicians in this field according to actual needs.

[0135] For each subspace of each modality, the cosine similarity between the clustering result of its own subspace and the clustering result of the subspace of other modalities is calculated; for example, the structured electronic medical record representation vector x is calculated mcs The first subspace of the text representation vector x texts The calculation formula for the first subspace of is:

[0136] sim mc0text0=cos(x mcs [1],x texts [1])

[0137] After calculating the clustering similarity of each subspace of each modality with the subspaces of other modalities, the subspace closest to the clustering result of the subspace is selected from the other two modalities, and the subspace is replaced with the average value of the three subspaces. mcs The first subspace of texts The second subspace of and x CTs The clustering results of the first subspace of are most similar, then x mcs The first subspace of will be updated to the average of these three subspaces, calculated as follows:

[0138]

[0139] After all subspaces are updated, the representation vector after feature level alignment can be obtained.

[0140] S106-3, fusing the above-mentioned modality-level aligned representation vector with the feature-level aligned representation vector to obtain a first fused representation vector.

[0141] The fusion method used in the embodiment of the present invention is a feature fusion method commonly used in the field. Specifically: the above-mentioned modal level aligned representation vector and the feature level aligned representation vector are spliced ​​column by column to obtain a spliced ​​vector; the spliced ​​vector is input into three independent fully connected networks to generate modal specific representation vectors of the three modalities respectively, and the spliced ​​vector is input into a shared fully connected network to generate a modal shared representation vector of the three modalities; the modal specific representation vector and the modal shared representation vector are pulled into one-dimensional vectors and then spliced ​​to obtain a first fused representation vector.

[0142] In order to reduce the correlation features between the modality-specific representation vector and the modality-shared representation vector, the following loss function is used to make the modality-specific representation vector and the modality-shared representation vector uncorrelated. The modality-specific representation vector and the modality-shared representation vector calculate the following loss function:

[0143]

[0144] Among them, H sp is the modality-specific representation vector, H sh is the modality sharing representation vector, To calculate the two-norm operation.

[0145] S107, performing feature selection on the first fused representation vector to obtain a multimodal representation.

[0146] The feature selection method used in the embodiment of the present invention is a commonly used method in the art, specifically:

[0147] The clustering result of each patient obtained by the subspace clustering is used as a label. Step S106-2 includes 3N subspaces. For the clustering results of these 3N subspaces, a voting method is used to obtain the clustering result of each patient. Based on the Person correlation coefficient, the first important feature set in the first fusion representation vector is screened out, which is recorded as H per ; Take the first fusion representation vector as input, and select the second important feature set based on the label in the chi-square test, denoted as H kf ; The first fused representation vector is used as the input of the random forest model, and according to the label, the third most important feature set with the feature importance ranking in the top 50% is selected, which is recorded as H rf ; Take the union of the first important feature set, the second important feature set, and the third important feature set to obtain the final multimodal representation x md , the calculation formula is as follows:

[0148] x md =Union(H per ,H kf ,H rf ), where Union(·) represents the operation of finding a union.

[0149] A second aspect of the present invention provides a data fusion system based on multimodal feature filling. Figure 3 A schematic diagram of the structure of a data fusion system based on multimodal feature filling provided in an embodiment of the present application.

[0150] The data fusion system 300 based on multimodal feature filling includes an acquisition module 301, a conversion module 302, a first filling module 303, a screening module 304, a second filling module 305, an alignment fusion module 306, and a feature selection module 307, which are specifically as follows:

[0151] An acquisition module 301 is used to acquire multimodal medical data of multiple patients, wherein the multimodal medical data includes structured electronic medical record features, text features, and CT image features of each patient;

[0152] A conversion module 302, used to convert text features and CT image features in the multimodal medical data into fused structured features, and form a first data set with the structured electronic medical record features;

[0153] A first filling module 303 is used to fill in missing features of structured electronic medical records in the first data set using the MissForest algorithm to obtain a second data set;

[0154] A screening module 304 is used to screen out patients with complete multimodal medical data from the second data set as a first complete data set, and screen out patients with missing text features and / or CT image features as a first missing data set;

[0155] A second filling module 305 is used to calculate the text features and CT image features in the first missing data set respectively based on the idea of ​​the KNN algorithm using the average text features and the average CT image features in the first complete data set to obtain the filled multimodal medical data;

[0156] An alignment and fusion module 306 is used to perform modality alignment on the padded multimodal medical data at the modality level and the feature level respectively by using contrastive learning and subspace clustering, and fuse the aligned representation vectors to obtain a first fused representation vector;

[0157] The feature selection module 307 is used to perform feature selection on the first fusion representation vector to obtain a multimodal representation.

[0158] Based on the above embodiments, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the medical data fusion method based on multimodal feature filling in the embodiment of the first aspect are implemented.

[0159] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0160] like Figure 4 As shown, the electronic device 400 may include a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0161] Multiple components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0162] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as omics data processing methods or model training methods for omics data processing. For example, in some embodiments, the omics data processing method or the model training method for omics data processing may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the omics data processing method or the model training method for omics data processing described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the omics data processing method or the model training method for omics data processing in any other suitable manner (eg, by means of firmware).

[0163] Based on the above embodiments, the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the cancer dynamic survival analysis method based on hierarchical attention disclosed in the embodiments of the present invention.

[0164] Based on the above embodiments, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the cancer dynamic survival analysis method based on hierarchical attention disclosed in the embodiments of the present invention.

[0165] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0166] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0171] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A medical data fusion method based on multimodal feature filling, characterized in that: Include: Acquire multimodal medical data of a plurality of patients, wherein the multimodal medical data includes structured electronic medical record features, text features, and CT image features of each patient; Converting text features and CT image features in the multimodal medical data into fused structured features, and forming a first data set with the structured electronic medical record features; The MissForest algorithm is used to fill in the missing features of the structured electronic medical records in the first dataset to obtain the second dataset; Screening out patients with complete multimodal medical data from the second data set as a first complete data set, and screening out patients with missing text features and / or CT image features as a first missing data set; Inputting the structured electronic medical record features in the first complete data set and the first missing data set into the pre-trained autoencoder to obtain corresponding structured electronic medical record representation vectors; For each patient in the first missing data set, find the K patients in the first complete data set that are most similar to the structured electronic medical record representation vector of the patient, where K is a hyperparameter; Inputting the text features and CT image features of the K patients into a pre-trained autoencoder to obtain a text representation vector and a CT image representation vector, and calculating the average text representation vector and the average CT image representation vector of the K patients; The missing text representation vector and CT image representation vector in the first missing data set are respectively initialized using the average text representation vector and the average CT image representation vector, and the missing text representation vector and CT image representation vector in the first missing data set are input into the pre-trained autoencoder for iteration until the mean square error between the representation vector this time and the representation vector last time is less than a threshold value, and finally the converged representation vector is input into the decoder in the pre-trained autoencoder to obtain the text features and CT image features in the first missing data set, and obtain the filled multimodal medical data; Contrastive learning and subspace clustering are used to align the padded multimodal medical data at the modality level and the finer-grained feature level, and the aligned representation vectors are fused to obtain the first fused representation vector. Feature selection is performed on the first fused representation vector to obtain a multimodal representation.

2. The medical data fusion method based on multimodal feature filling as claimed in claim 1, characterized in that: The method for converting text features and CT image features in multimodal medical data into fused structured features is: Screening out patients with complete multimodal medical data from the multimodal medical data as a second complete data set, and screening out patients with missing structured electronic medical record features as a second missing data set; Input the text features and CT image features in the second missing data set into the pre-trained autoencoder to obtain a text representation vector and a CT image representation vector, and merge them into a second fusion representation vector; The second fused representation vector is used to replace the original text features and CT image features in the second missing data set to obtain fused structured features.

3. The medical data fusion method based on multimodal feature filling as claimed in claim 1, characterized in that: The method of using contrastive learning to align the padded multimodal medical data at the modality level is as follows: calculating the cosine similarity between the three modal features of each patient in the padded multimodal medical data, and the cosine similarity between the three modal features of each patient and the three modal features of other patients; The contrastive learning loss function is calculated based on all the cosine similarities obtained, and the representation variables of the padded multimodal medical data are aligned through back propagation.

4. The medical data fusion method based on multimodal feature filling as claimed in claim 1, characterized in that: The method of using subspace clustering to align the padded multimodal medical data at the feature level is: Input the padded multimodal medical data into the pre-trained autoencoder to obtain the representation vectors of the three modal features; The representation vectors of the three modal features are evenly divided into N subspace representation vectors respectively; Perform k-means clustering on the 3N subspace representation vectors to obtain the clustering results of 3N subspaces; Calculate the cosine similarity between the clustering result of each subspace of each modal feature and the clustering result of each subspace of other modal features; According to the cosine similarity of the clustering results, the subspace closest to the clustering result of this subspace is selected from the other two modal features, and the representation vector of this subspace is replaced with the average value of the representation vectors of these three subspaces. After all 3N subspaces are replaced, modal alignment is achieved.

5. The medical data fusion method based on multimodal feature filling as claimed in claim 1, characterized in that: The method of fusing the aligned representation vectors to obtain the first fused representation vector is: The modality-level aligned feature vectors and the feature-level aligned feature vectors are concatenated column by column to obtain a concatenated vector representation; Generate modality-specific representation vectors of the three modalities and modality-shared representation vectors of the three modalities respectively according to the concatenated vector representations; The modality-specific representation vector and the modality-shared representation vector are pulled into one-dimensional vectors and then concatenated to obtain a first fused representation vector.

6. The medical data fusion method based on multimodal feature filling as claimed in claim 5, characterized in that: The modality-specific representation vector and the modality-shared representation vector calculate the following loss function: , in, is the modality-specific representation vector, is the modality sharing representation vector, To calculate the two-norm operation.

7. The medical data fusion method based on multimodal feature filling as claimed in claim 1, characterized in that: The method for performing feature selection on the first fusion representation vector to obtain a multimodal representation is: The clustering result of each patient obtained by the subspace clustering is used as a label; Based on the Person correlation coefficient, the first important feature set in the first fusion representation vector is screened out; Taking the first fused representation vector as input, and screening out a second important feature set by a chi-square test according to the label; The first fused representation vector is used as an input to the random forest model, and the third most important feature set whose feature importance ranks in the top 50% is selected according to the label; The first important feature set, the second important feature set, and the third important feature set are unioned to obtain a multimodal representation.

8. A data fusion system based on multimodal feature filling, characterized in that: Include: An acquisition module, used to acquire multimodal medical data of multiple patients, wherein the multimodal medical data includes structured electronic medical record features, text features, and CT image features of each patient; A conversion module, used for converting text features and CT image features in the multimodal medical data into fused structured features, and forming a first data set with the structured electronic medical record features; A first filling module is used to fill in the missing features of the structured electronic medical records in the first data set using the MissForest algorithm to obtain a second data set; A screening module, used to screen out patients with complete multimodal medical data from the second data set as a first complete data set, and screen out patients with missing text features and / or CT image features as a first missing data set; The second filling module is used to input the structured electronic medical record features in the first complete data set and the first missing data set into the pre-trained autoencoder to obtain the corresponding structured electronic medical record representation vector; for each patient in the first missing data set, find the K patients whose structured electronic medical record representation vectors are most similar to the patient in the first complete data set, where K is a hyperparameter; input the text features and CT image features of the K patients into the pre-trained autoencoder to obtain the text representation vector and the CT image representation vector, and calculate the average text representation vector and the average CT image representation vector of the K patients; use the average text representation vector and the average CT image representation vector to initialize the missing text representation vector and the CT image representation vector in the first missing data set respectively, and input the missing text representation vector and the CT image representation vector in the first missing data set into the pre-trained autoencoder for iteration until the mean square error between the representation vector this time and the representation vector last time is less than a threshold value, and finally input the converged representation vector into the decoder in the pre-trained autoencoder to obtain the text features and CT image features in the first missing data set, and obtain the filled multimodal medical data; An alignment and fusion module, used to align the padded multimodal medical data at the modality level and at a finer-grained feature level using contrastive learning and subspace clustering, and to fuse the aligned representation vectors to obtain a first fused representation vector; The feature selection module is used to perform feature selection on the first fusion representation vector to obtain a multimodal representation.

Images