Multi-modal medical data progressive fusion method and system

By combining multi-layer feature fusion and cross-layer attention mechanisms with dynamic graph learning and graph autoencoders, the problems of information loss and insufficient adaptability in multimodal medical data fusion are solved, achieving more accurate information transmission and feature interaction.

CN119004362AActive Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202411032143.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

In existing multimodal medical data fusion methods, complementary information between modalities may be discarded, static graph representations cannot be adaptively filtered, and the alignment process relies on high-quality data augmentation which is unstable.

Method used

Employing a multi-layer feature fusion and cross-layer attention mechanism, combined with dynamic graph learning and a masked graph autoencoder, image, electronic medical record and text data are aligned and fused layer by layer. Through adaptive learning and feature reconstruction, and pruning of similarity matrices, dynamic graph construction and information transmission are achieved.

Benefits of technology

It improves the accuracy and adaptability of information transmission, captures cross-modal feature interactions, enhances the applicability of downstream tasks, and reduces information loss and noise interference.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004362A_ABST
    Figure CN119004362A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of feature fusion, and discloses a multi-modal medical data progressive fusion method and system. According to the method, hierarchical fusion feature vector interaction is realized based on a cross-hierarchical attention mechanism through a progressive fusion method, the problem that shallow information is easy to lose due to increase of the number of layers is relieved by using a residual network, and hierarchical information contained in progressive fusion features is maximized in a progressive interaction mode. According to the dynamic graph learning method provided by the invention, non-linear interaction relations of features are captured through graph adaptive learning, heterogeneity of similar matrixes is reduced, and accuracy of information transmission is improved; in the adaptive learning process, threshold prediction is carried out for the features of each patient to prune the similar matrix, so that the pruned similar matrix is more accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of feature fusion, and in particular relates to a method and system for progressive fusion of multimodal medical data. Background Art

[0002] With the improvement of science and technology, we can obtain a large amount of medical data in different modalities, such as electronic medical record data, imaging data, and text data, so as to make a more comprehensive assessment of the patient's health status from different angles, helping doctors and researchers to more accurately understand the disease and predict its progression. Multimodal medical data fusion plays a huge role in improving the accuracy of disease diagnosis and promoting early detection of diseases by finding complementary information between modalities and studying complex cross-modal interactions.

[0003] The multimodal medical data fusion strategy can be divided into early fusion, mid-term fusion and late fusion according to the fusion stage. In the early fusion stage, data from different modalities are fused into a richer input feature after operations such as feature screening and feature mapping, and then input into the model; in the mid-term fusion stage, data from different modalities are fused after passing through the feature extraction modules of each modality; in the late fusion stage, the prediction vectors of each modality for downstream tasks are fused.

[0004] At present, the multimodal medical data fusion method has the following shortcomings. First, as the multimodal fusion stage progresses, the features of each modality are continuously purified, filtering out redundant information and noise interference. However, the information contained is more biased towards a single modality, and complementary information between modalities may be discarded. Second, the graph representation structure constructed by non-graph data is often a static graph. When calculating patient similarity, all features are given the same importance and are not matched with downstream tasks. In addition, when removing low-correlation edges, it is impossible to perform adaptive screening for the patient itself. Third, in modal alignment, contrastive learning plays an important role, but the reliance on high-quality data enhancement has always been its pain point, and graph enhancement is mostly based on heuristic methods, and the effect is unstable in different task scenarios. Summary of the invention

[0005] The present invention aims to solve at least one of the above technical problems and provides a method and system for progressive fusion of multimodal medical data.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for progressive fusion of multimodal medical data, comprising:

[0008] Extracting N layers of features from the patient's multimodal medical data, where N is an integer ≥ 2, the multimodal medical data includes image data, structured electronic medical record data, and text data, and the extracted N layers of features include N layers of image features, N layers of structured electronic medical record features, and N layers of text features;

[0009] All modal features in each layer are fused to obtain N levels of fused features;

[0010] Align the hierarchical fusion features of each layer separately;

[0011] Based on the cross-layer attention mechanism, the aligned hierarchical fusion features of the first layer and the second layer are fused to obtain the first progressive fusion feature, and then the first progressive fusion feature and the aligned hierarchical fusion features of the third layer are fused based on the cross-layer attention mechanism to obtain the second progressive fusion feature, and so on. The fusion steps based on the cross-layer attention mechanism are repeated until the fusion of the N layers of aligned hierarchical fusion features is completed.

[0012] Preferably, the method of fusing all modal features in each layer of features to obtain N levels of fused features is:

[0013] For each layer of features, all modal features in the layer are mapped and concatenated into concatenated features;

[0014] Filter out important features in the splicing features that are highly relevant to downstream tasks;

[0015] Calculate the patient-patient similarity matrix based on the important features obtained by screening;

[0016] After pruning the similarity matrix, information is transferred with the important features to obtain the hierarchical fusion features of each layer.

[0017] Preferably, through the loss function After achieving the mapping, the feature vectors of different modalities for the same patient are similar:

[0018]

[0019] Among them, cos is the cosine similarity calculation, and the calculation formula is and Image features Structured electronic medical record characteristics and text features The mapped feature vector, j = {1, 2, ..., N}, represents the jth layer.

[0020] Preferably, the pruning method is:

[0021] The similarity threshold of each patient was calculated based on the important features obtained by screening;

[0022] The similarity matrix is ​​pruned according to the patient's similarity threshold, edges less than the similarity threshold are removed, and edges greater than or equal to the similarity threshold are retained.

[0023] Preferably, through the loss function Measure the similarity between the important features reconstructed at each layer and the important features screened out,

[0024] in, It is an important feature of reconstruction. is the important feature selected, j = {1,2,...,N}, represents the jth layer, cos is the cosine similarity calculation, and the calculation formula is

[0025] Among them, the important features of the reconstruction are obtained through the masked graph decoder, the specific method is as follows:

[0026] The pruned similarity matrix is ​​path-masked, the hierarchical fusion features are unsupervisedly masked, and then input into the graph convolution decoder for decoding to obtain the important features of each layer reconstruction.

[0027] Preferably, the method of aligning the hierarchical fusion features of each layer is shown in the following formula:

[0028] Among them, FC gcn It is a fully connected network. It is a hierarchical fusion feature After mapping, BatchNormal is an operation to normalize the feature vector by column, j = {1, 2, ..., N}, representing the jth layer.

[0029] Preferably, the formula based on the cross-layer attention mechanism fusion is:

[0030]

[0031] in, It is the progressive fusion feature after fusing the hierarchical fusion features of the first j layers. Softmax is a normalized exponential function, and its formula is is the aligned hierarchical fusion feature The dimension of means The transposed matrix of It is a fully connected network, j = {1, 2, ..., N}, representing the jth layer.

[0032] Preferably, the image data includes X-rays, CT scan images, and MRI images; the structured electronic medical record data includes demographic characteristics and clinical examination information; and the text data includes medical history and treatment records.

[0033] In a second aspect, the present invention provides a system for progressive fusion of multimodal medical data, comprising:

[0034] An extraction module, used to extract N layers of features of the patient's multimodal medical data, where N is an integer ≥ 2, the multimodal medical data includes image data, structured electronic medical record data and text data, and the extracted N layers of features include N layers of image features, N layers of structured electronic medical record features and N layers of text features;

[0035] The first fusion module is used to fuse all modal features in each layer of features to obtain N levels of fused features;

[0036] Alignment module, used to align the hierarchical fusion features of each layer separately;

[0037] The second fusion module is used to fuse the aligned hierarchical fusion features of the first layer and the second layer based on the cross-layer attention mechanism to obtain the first progressive fusion feature, and then fuse the first progressive fusion feature with the aligned hierarchical fusion features of the third layer based on the cross-layer attention mechanism to obtain the second progressive fusion feature, and so on, repeating this fusion step based on the cross-layer attention mechanism until the fusion of N layers of aligned hierarchical fusion features is completed.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention realizes the interaction of hierarchical fusion feature vectors based on the cross-level attention mechanism through the progressive fusion method, and utilizes the residual network to alleviate the problem of easy loss of shallow information caused by the increase in the number of layers, so as to maximize the hierarchical information contained in the progressive fusion features in a progressive interactive manner.

[0040] The present invention provides a dynamic graph learning method, which captures the nonlinear interaction relationship of features through graph adaptive learning, reduces the heterogeneity of the similarity matrix, and improves the accuracy of information transmission; in the adaptive learning process, a threshold prediction is performed for the features of each patient to prune the similarity matrix, so that the pruned similarity matrix is ​​more accurate.

[0041] The present invention uses a masked graph autoencoder for modality alignment, which is more universal for different downstream tasks and can effectively capture the interaction relationship between cross-modal features. In this process, important features are reconstructed by decoding, and a feature reconstruction loss function is set, so that while encoding and compressing features, important information can be learned and complex interactions between features can be explored. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a process for progressive fusion of multimodal medical data provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the structure of a masked graph autoencoder provided by one embodiment of the present invention;

[0044] Figure 3 A framework diagram of progressive fusion of multimodal medical data provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the cross-level attention mechanism structure provided for one embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a system for progressive fusion of multimodal medical data provided by an embodiment of the present invention;

[0047] Figure 6 FIG. 1 is a schematic block diagram of an exemplary electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The first embodiment of the present invention provides a method for progressive fusion of multimodal medical data. Figure 1 A flowchart of a method for progressive fusion of multimodal medical data provided by an embodiment of the present invention.

[0050] like Figure 1 As shown, the method for progressive fusion of multimodal medical data specifically includes the following steps:

[0051] S101, extracting N layers of features of the patient's multimodal medical data, where N is an integer ≥ 2, the multimodal medical data includes image data, structured electronic medical record data, and text data, and the extracted N layers of features include N layers of image features, N layers of structured electronic medical record features, and N layers of text features.

[0052] Understandably, N is a hyperparameter, and users can determine it based on their own requirements for the accuracy and time cost of downstream prediction tasks. For example, it can be 2, 3, 4, 7, 9, 10, 15, etc.

[0053] Specifically, the multimodal medical data of patients includes imaging data (x image), structured electronic medical record data (x mc ) and text data (x text ). Among them, the imaging data includes data forms such as X-rays, CT scan images, and MRI images; the structured electronic medical record data includes information such as demographic characteristics and clinical tests; and the text data includes information such as medical history and treatment records. The symbol M represents the number of patients with complete multimodal medical data collected, and the entire multimodal medical data set is represented by P as:

[0054]

[0055] For the method of extracting N-layer features of multimodal medical data of patients, each modality of medical data is input into its own feature extraction module to obtain its own feature vector. Specifically, the image data, structured electronic medical record data and text data in the multimodal medical data of the patient are respectively input into the image feature extraction module IFM, the structured electronic medical record feature extraction module MFM and the text feature extraction module TFM to extract the feature vectors of the three modal data of each patient. in, and are the image features, electronic medical record features, and text features of the i-th patient respectively.

[0056] Since the image feature extraction module IFM is composed of N layers of image feature extraction functions IFE, the image data of the i-th patient can be obtained The multi-layer image features, namely in, is the feature of the image data of the i-th patient after being extracted by the first j layers of image extraction functions. For a better description below, let Similarly, multiple layers of structured electronic medical record features and text features can be obtained, and finally N layers of image features, N layers of structured electronic medical record features, and N layers of text features can be obtained.

[0057] S102, all modal features in each layer of features are fused to obtain N levels of fused features.

[0058] It can be understood that the method of fusion of each layer of features is the same, and finally the hierarchical fusion features of N layers of features are obtained. Figure 2 , the specific method includes the following steps:

[0059] S102-1, for each layer of features, all modal features in the layer of features are mapped and concatenated into concatenated features.

[0060] For example, for the jth layer feature of a patient Input into the j-th layer image feature vector mapping function respectively The j-th layer electronic medical record feature vector mapping function The j-th layer text feature vector mapping function And concatenate them, the specific formula is as follows:

[0061]

[0062] in, and is a fully connected layer network, and They are and The mapped feature vector, Concate is the operation of concatenating the feature vectors. is to make the j layer and The concatenated feature vector, j = {1, 2, ..., N}, represents the Nth layer.

[0063] In some preferred embodiments, in order to reduce the heterogeneity between modalities, it is required that the feature vectors of different modalities of the same patient should be similar after mapping, which is achieved using the following loss function:

[0064]

[0065]

[0066] Among them, cos is the cosine similarity calculation, and the calculation formula is is the j-th layer modality mapping loss function.

[0067] S102-2, filter out important features in the splicing features that are highly correlated with downstream tasks.

[0068] Exemplarily, the splicing features obtained in S102-1, i.e., the jth layer of the patient Input to the Gate feature screening module to screen out important features that are highly relevant to downstream tasks. The specific formula is as follows:

[0069]

[0070] Among them, FC Gate is a fully connected layer, Sigmoid and Relu are activation functions, AttentionSorce j is the importance of each feature of each patient in the jth layer, mul is the operation of multiplying the corresponding elements of two vectors, are the important features screened out, j = {1, 2, ..., N}.

[0071] In some preferred embodiments, in order to make AttentionSorce j Being able to identify the importance of features, this is achieved using the following loss function:

[0072]

[0073] Among them, mean is the average value, It is the loss function of the j-layer Gate filtering feature.

[0074] S102-3, calculating the similarity matrix between patients based on the important features obtained by screening.

[0075] For example, Input to the graph adaptive learning module to calculate the similarity between patients. The specific formula is as follows:

[0076]

[0077] Among them, W is the learnable parameter matrix, Matrix j is the similarity matrix between patients at the jth layer, yes The transpose operation of , j = {1,2,...,N}.

[0078] S102-4, pruning the similarity matrix and transferring information with the important features to obtain hierarchical fusion features of each layer.

[0079] In some preferred embodiments, the method for pruning the similarity matrix is: calculating the similarity threshold of each patient based on the screened important features; pruning the similarity matrix based on the patient's similarity threshold, removing edges less than the similarity threshold, and retaining edges greater than or equal to the similarity threshold.

[0080] The calculation formula for the similarity threshold is: Among them FC threshold It is a fully connected network, Threshold j is the similarity threshold for each patient in layer j, j = {1, 2, ..., N}.

[0081] The formula for pruning is:

[0082]

[0083] Among them, Matrix j (m,n) represents the similarity between the mth patient and the nth patient in the jth layer, Threshold j (m) represents the threshold of the mth patient in the jth layer, yes The transposed matrix of represents the similarity matrix between patients after pruning, j = {1,2,...,N}.

[0084] For information transfer, the pruned similarity matrix and the important features are input into the graph convolutional network (GCN) module for information transfer. The specific formula is:

[0085]

[0086] in, express The feature vector after the graph convolution module is the hierarchical fusion feature under the jth layer, called the hierarchical fusion feature, GCN encoder represents the graph convolutional encoder module, j = {1, 2, ..., N}.

[0087] It can be seen that S102-2 to S102-4 is the process of dynamic graph learning, which realizes the dynamic graph construction of non-graph data. In general, the extracted features often contain redundant information that is irrelevant to the downstream tasks. Dynamic graph learning is used to screen the features, eliminate noise information, and retain features that are highly relevant to the downstream tasks. The prior art usually measures linear similarity in the composition method, ignoring the complex relationship between features. In the implementation mode of the present invention, dynamic graph learning uses a graph adaptive learning module to capture the nonlinear interaction relationship of features, reduce the heterogeneity of the similarity matrix, and improve the correctness of information transmission. In addition, in the prior art, similarity matrix pruning is generally performed through the KNN algorithm to retain k edges with high similarity, but this approach is a global screening, ignoring the local situation of each patient. In the implementation mode of the present invention, dynamic graph learning performs threshold prediction on the features of each patient, so that the pruned similarity matrix is ​​more accurate.

[0088] In some more preferred embodiments, the loss function Weigh the similarity between the important features reconstructed at each layer and the important features screened out to enhance the accuracy of the above adaptive composition:

[0089] in, It is an important feature of reconstruction. is the important feature selected, j = {1,2,...,N}, represents the jth layer, cos is the cosine similarity calculation, and the calculation formula is

[0090] Among them, the reconstructed important features are obtained through a masked graph decoder. The specific method is: the pruned similarity matrix is ​​path-masked, the hierarchical fusion features are unsupervised feature-masked, and then input into the graph convolution decoder for decoding to obtain the reconstructed important features.

[0091] For example, the pruned similarity matrix Input to the path mask module for path masking. In the path mask module, Sampling is performed to obtain its subgraph. Whether each edge is kept or masked follows the Bernoulli distribution. The specific formula is as follows:

[0092]

[0093] in, After the path mask Bernouli is the Bernoulli distribution, Rate edge is the ratio of the path mask, ~ refers to It obeys Bernouli distribution, j={1,2,...,N}.

[0094] The patients are randomly masked, that is, the hierarchical fusion features are unsupervised feature masked. The order of the patients is randomly shuffled, and the masked patients and the unchanged patients are selected in proportion. The specific formula is as follows:

[0095] NumMask=int(M*Rate Patient ),

[0096] MaskPatient=Shuffle(P)[:NumMask],

[0097]

[0098] Where M is the number of patients, NumMask is the number of patients that need to be masked, * is the multiplication operation of two numbers, and Rate Patient is the proportion of patients masked, int is a rounding operation, Shuffle is an operation to shuffle the list for the purpose of random sampling, [:NumMask] is an operation to take the first NumMask item in the list, yes Feature vector after feature masking j = {1, 2, ..., N}.

[0099] The unsupervised feature masked hierarchical fusion features and the similarity matrix after path masking Input into the graph convolution decoder to decode the information. The specific formula is as follows:

[0100]

[0101] Among them, GCN decoder is the graph convolutional decoder module, It is the feature vector after passing through the graph decoder, that is, the important feature of reconstruction, j = {1, 2, ..., N}.

[0102] By decoding and reconstructing important features, and setting the feature reconstruction loss function, the encoding and compression features can be used to learn important information and explore the complex interaction between features. The above masked encoder perturbs the features and similarity matrix so that neighbor nodes play a more important role in feature reconstruction, thereby improving the accuracy of the similarity matrix and maximizing the extracted information.

[0103] S103, aligning the hierarchical fusion features of each layer respectively.

[0104] Since the hierarchical fusion feature vectors at different levels are in different feature spaces and have heterogeneity, an alignment operation is required. The specific method is shown in the following formula:

[0105]

[0106] Among them, FC gcn It is a fully connected network. It is a hierarchical fusion feature After mapping, BatchNormal is an operation to normalize the feature vector by column.

[0107] S104, after fusing the aligned hierarchical fusion features of the first layer and the second layer based on the cross-layer attention mechanism to obtain the first progressive fusion feature, the first progressive fusion feature and the aligned hierarchical fusion features of the third layer are fused based on the cross-layer attention mechanism to obtain the second progressive fusion feature, and so on, repeating this fusion step based on the cross-layer attention mechanism until the fusion of the N layers of aligned hierarchical fusion features is completed.

[0108] Specifically, refer to Figure 3-4 , Figure 3 This is a schematic diagram of the framework for progressive fusion of multimodal medical data. Figure 4This is a schematic diagram of the cross-layer attention mechanism structure. This step is to explore the relationship between cross-layer fusion features through the self-attention mechanism to achieve the interaction of layer fusion feature vectors. As the number of layers increases, the influence of early layer fusion features on progressive fusion features gradually weakens. The residual method is used to alleviate the problem of easy loss of shallow information when the number of layers increases. Since there are N layers of aligned layer fusion features, the cross-layer attention mechanism fusion is performed N-1 times. In this way, the layer fusion features of different levels interact continuously, and as the fusion stage advances, progressive fusion is formed to obtain the final progressive fusion feature vector. The layer information contained in the progressive fusion feature is maximized through progressive interaction.

[0109] The formula for the fusion process is:

[0110]

[0111] in, It is the feature vector after the hierarchical fusion features of the first j layers are fused, which is called the progressive fusion feature. Softmax is a normalized exponential function, and its formula is yes The dimension of means The transposed matrix of It is a fully connected network.

[0112] The second embodiment of the present invention provides a system for progressive fusion of multimodal medical data.

[0113] Figure 5 A schematic diagram of the structure of a system for progressive fusion of multimodal medical data provided in an embodiment of the present application.

[0114] like Figure 5 As shown, the system 500 for progressive fusion of multimodal medical data includes an extraction module 501, a first fusion module 502, an alignment module 503, and a second fusion module 504. The functions of each module are described in detail as follows:

[0115] An extraction module 501 is used to extract N layers of features from the patient's multimodal medical data, where N is an integer ≥ 2, the multimodal medical data includes image data, structured electronic medical record data, and text data, and the extracted N layers of features include N layers of image features, N layers of structured electronic medical record features, and N layers of text features;

[0116] The first fusion module 502 is used to fuse all modal features in each layer of features to obtain N layers of fused features;

[0117] An alignment module 503, used to align the hierarchical fusion features of each layer respectively;

[0118] The second fusion module 504 is used to fuse the first layer aligned hierarchical fusion features and the second layer aligned hierarchical fusion features based on the cross-layer attention mechanism to obtain the first progressive fusion features, and then fuse the first progressive fusion features and the third layer aligned hierarchical fusion features based on the cross-layer attention mechanism to obtain the second progressive fusion features, and repeat this fusion step based on the cross-layer attention mechanism until the fusion of N layers of aligned hierarchical fusion features is completed.

[0119] Based on the above embodiments, the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the method for progressive fusion of multimodal medical data in the embodiment of the first aspect are implemented.

[0120] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0121] like Figure 6 As shown, the electronic device 600 may include a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0122] Multiple components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as omics data processing methods or model training methods for omics data processing. For example, in some embodiments, the omics data processing method or the model training method for omics data processing may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the omics data processing method or the model training method for omics data processing described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the omics data processing method or the model training method for omics data processing in any other suitable manner (e.g., by means of firmware).

[0124] Based on the above embodiments, the present invention further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the method for progressive fusion of multimodal medical data disclosed in the embodiments of the present invention.

[0125] Based on the above embodiments, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the method for progressive fusion of multimodal medical data disclosed in the embodiments of the present invention.

[0126] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0127] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0132] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. Methods for progressive fusion of multimodal medical data, It is characterized in that Include: Extracting N layers of features from the patient's multimodal medical data, where N is an integer ≥ 2, the multimodal medical data includes image data, structured electronic medical record data, and text data, and the extracted N layers of features include N layers of image features, N layers of structured electronic medical record features, and N layers of text features; All modal features in each layer of features are fused to obtain N levels of fused features; Align the hierarchical fusion features of each layer separately; Based on the cross-layer attention mechanism, the aligned hierarchical fusion features of the first layer and the second layer are fused to obtain the first progressive fusion feature, and then the first progressive fusion feature and the aligned hierarchical fusion features of the third layer are fused based on the cross-layer attention mechanism to obtain the second progressive fusion feature, and so on. The fusion steps based on the cross-layer attention mechanism are repeated until the fusion of the N layers of aligned hierarchical fusion features is completed.

2. The method for progressive fusion of multimodal medical data as claimed in claim 1, It is characterized in that The method of fusing all modal features in each layer of features to obtain N-level fusion features is: For each layer of features, all modal features in the layer are mapped and concatenated into concatenated features; Filter out important features in the splicing features that are highly correlated with downstream tasks; Calculate the patient-patient similarity matrix based on the important features obtained by screening; After pruning the similarity matrix, information is transferred with the important features to obtain the hierarchical fusion features of each layer.

3. The method for progressive fusion of multimodal medical data as claimed in claim 2, It is characterized in that Through the loss function After achieving the mapping, the feature vectors of different modalities for the same patient are similar: Among them, cos is the cosine similarity calculation, and the calculation formula is and Image features Structured electronic medical record characteristics and text features The mapped feature vector, j = {1, 2, ..., N}, represents the jth layer.

4. The method for progressive fusion of multimodal medical data as claimed in claim 2, It is characterized in that The pruning method is: The similarity threshold of each patient was calculated based on the important features obtained by screening; The similarity matrix is ​​pruned according to the patient's similarity threshold, edges less than the similarity threshold are removed, and edges greater than or equal to the similarity threshold are retained.

5. The method for progressive fusion of multimodal medical data as claimed in claim 2, It is characterized in that Through the loss function Measure the similarity between the important features reconstructed at each layer and the important features screened out, in, An important feature of reconstruction is is the important feature selected, j = {1,2,...,N}, represents the jth layer, cos is the cosine similarity calculation, and the calculation formula is Among them, the important features of the reconstruction are obtained through the masked graph decoder, the specific method is as follows: The pruned similarity matrix is ​​path-masked, the hierarchical fusion features are unsupervisedly masked, and then input into the graph convolution decoder for decoding to obtain the important features of each layer reconstruction.

6. The method for progressive fusion of multimodal medical data as claimed in claim 1, It is characterized in that The method of aligning the hierarchical fusion features of each layer is shown in the following formula: Among them, FC gcn It is a fully connected network. It is a hierarchical fusion feature After mapping, BatchNormal is the operation of normalizing the feature vector by column, j = {1, 2, ..., N}, representing the jth layer.

7. The method for progressive fusion of multimodal medical data as claimed in claim 1, It is characterized in that The formula based on the cross-layer attention mechanism fusion is: in, It is the progressive fusion feature after fusing the hierarchical fusion features of the first j layers. Softmax is a normalized exponential function, and its formula is is the aligned hierarchical fusion feature The dimension of means The transposed matrix of It is a fully connected network, j = {1, 2, ..., N}, representing the jth layer.

8. The method for progressive fusion of multimodal medical data as claimed in claim 1, It is characterized in that The image data includes X-rays, CT scan images, and MRI images; the structured electronic medical record data includes demographic characteristics and clinical examination information; and the text data includes medical history and treatment records.

9. A system for the progressive fusion of multimodal medical data. It is characterized in that Include: An extraction module, used to extract N layers of features of the patient's multimodal medical data, where N is an integer ≥ 2, the multimodal medical data includes image data, structured electronic medical record data and text data, and the extracted N layers of features include N layers of image features, N layers of structured electronic medical record features and N layers of text features; The first fusion module is used to fuse all modal features in each layer of features to obtain N levels of fused features; Alignment module, used to align the hierarchical fusion features of each layer separately; The second fusion module is used to fuse the aligned hierarchical fusion features of the first layer and the second layer based on the cross-layer attention mechanism to obtain the first progressive fusion feature, and then fuse the first progressive fusion feature with the aligned hierarchical fusion features of the third layer based on the cross-layer attention mechanism to obtain the second progressive fusion feature, and so on, repeating this fusion step based on the cross-layer attention mechanism until the fusion of N layers of aligned hierarchical fusion features is completed.

Citation Information

Patent Citations

  • Pulmonary nodule intelligent grading method and system based on multi-modal feature fusion

    CN116883768A

  • Image processing method and device, equipment and medium

    CN117711001A

  • Method and system for predicting death rate of patient based on multi-modal missing data

    CN117954090A

  • Medical data fusion method and system based on multi-modal feature filling

    CN118364423A

  • Social relation analysis method and system based on multi-modal data, and storage medium

    WO2024031933A1

Cited By

  • Multimodal medical data fusion method and system based on reinforcement learning and heterogeneous hypergraph

    CN120493149A

  • Biological characteristic analysis system based on multi-modal data and residual image attention network

    CN120655990A

  • Medical data fusion method and system based on large model and heterogeneous hypergraph learning

    CN121641480A