Medical multi-modal data fusion diagnosis device and method, medium and electronic equipment
Through cross-attention fusion technology combined with wide-deep network, the problem of multimodal medical data integration is solved, the diagnostic accuracy and efficiency are improved, the risk of misdiagnosis is reduced, and rapid analysis and efficient diagnosis are achieved.
Patent Information
- Application Number
- CN202410872863.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-15
AI Technical Summary
The existing technology is difficult to effectively integrate and mine information in multimodal medical data, resulting in insufficient diagnostic accuracy and personalized support, and traditional methods have degraded performance when processing complex or heterogeneous data sets, and require a large amount of parameter adjustment and optimization. Increases the complexity and cost of model deployment.
A wide-deep network based on cross-attention fusion is adopted, and patient statistical information and blood test data are processed through a wide network module. The deep network module extracts deep features of medical image data and uses cross-attention fusion technology to fuse feature vectors to obtain diagnostic results.
It improves the accuracy and efficiency of diagnosis, reduces the risk of misdiagnosis and missed diagnosis, can quickly process and analyze a large amount of multimodal medical data, improves the reliability and resource utilization efficiency of diagnosis, and provides timely treatment advice to patients.
Smart Images

Figure CN118969239A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of medical data processing, and relates to a medical multimodal data fusion diagnostic device, and in particular to a diagnostic device, method, medium and electronic device for performing medical multimodal data fusion using a wide and deep network based on cross-attention fusion. Background Art
[0002] In the field of modern medical diagnosis, the comprehensive application of multimodal data has become an important means to improve the accuracy and comprehensiveness of diagnosis. The data types of multimodal data include medical images, genomic data, electronic health records, etc. Each data source carries patient information at different levels and dimensions. Fusion of these data can provide doctors with a more comprehensive and comprehensive basis for diagnosis. However, due to the heterogeneity of these data sources and the correlation between data, how to effectively integrate and mine their information to provide more accurate and personalized diagnostic support has become one of the technical problems that relevant technicians need to solve urgently. Summary of the invention
[0003] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a medical multimodal data fusion diagnosis device, method, medium and electronic equipment to improve the accuracy and efficiency of diagnosis.
[0004] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a medical multimodal data fusion diagnosis device, which includes: a data acquisition module, used to acquire medical imaging data, patient statistical information and blood test data; a wide network module, used to process the patient statistical information and the blood test data to obtain a first feature vector; a deep network module, used to process the medical imaging data to obtain a second feature vector; a feature fusion module, used to fuse the first feature vector and the second feature vector using cross-attention fusion technology to obtain a fusion feature; and a diagnosis result acquisition module, used to obtain a diagnosis result based on the fusion feature.
[0005] In an implementation of the first aspect, the wide network module includes: an encoding unit, used to perform one-hot encoding and sparse matrix encoding on the patient statistical information and the blood test data; a splicing unit, used to splice the one-hot encoding results and sparse matrix encoding results of the patient statistical information and the blood test data to obtain the first feature vector.
[0006] In an implementation manner of the first aspect, the deep network module uses a ViT model to extract deep features of the medical image data to obtain the second feature vector.
[0007] In an implementation of the first aspect, the ViT model includes several Transformer blocks, and the Transformer block includes MLP, MSA, LayerNorm and residual connection.
[0008] In an implementation of the first aspect, the feature fusion module includes: a first processing unit, used to create a query matrix from the first feature vector; a second processing unit, used to create a key matrix and a value matrix from the second feature vector; and a feature fusion unit, used to obtain the fused feature according to the query matrix, the key matrix and the value matrix.
[0009] In an implementation of the first aspect, the feature fusion unit includes: an attention score calculation subunit, used to calculate the attention scores of the query matrix and the key matrix and normalize them; a feature fusion subunit, used to weightedly fuse the normalized attention scores with the value matrix to obtain the fused features.
[0010] In an implementation manner of the first aspect, the diagnosis result acquisition module inputs the fusion feature into a 1-layer MLP to obtain the diagnosis result.
[0011] In a second aspect, an embodiment of the present application provides a medical multimodal data fusion diagnosis method, the method comprising: acquiring medical imaging data, patient statistical information and blood test data; processing the patient statistical information and the blood test data to obtain a first feature vector; processing the medical imaging data to obtain a second feature vector; using cross-attention fusion technology to fuse the first feature vector and the second feature vector to obtain a fusion feature; and obtaining a diagnosis result based on the fusion feature.
[0012] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the medical multimodal data fusion diagnosis method described in the second aspect of the embodiment of the present application.
[0013] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory storing a computer program; and a processor communicatively connected to the memory, for executing the medical multimodal data fusion diagnosis method described in the second aspect of the embodiment of the present application when the computer program is called.
[0014] As described above, the medical multimodal data fusion diagnosis device provided in the embodiment of the present application comprehensively utilizes wide and deep networks (Wide and Deep Networks) and cross-attention fusion technology to effectively process and fuse medical data from different data sources to improve the accuracy and efficiency of diagnosis. The medical multimodal data fusion diagnosis technology device provided in the embodiment of the present application has important practical significance and broad application prospects for processing multimodal and high-dimensional medical data.
[0015] The medical multimodal data fusion diagnosis device provided in the embodiment of the present application can reduce misdiagnosis caused by missing or insufficient information. Through deep network processing and wide network information capture, the risk of misjudgment or missed diagnosis that may occur during the diagnosis process can be effectively reduced, thereby improving the reliability of diagnosis.
[0016] Compared with traditional methods, the wide and deep network based on cross-attention in the embodiments of the present application can process and analyze a large amount of multimodal medical data more quickly. This helps to speed up the diagnosis process, make medical resources more efficiently used, and provide timely treatment advice to patients. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 Shown is a schematic diagram of the structure of an electronic device in an embodiment of the present application.
[0018] Figure 2 Shown is a schematic diagram of the structure of a medical multimodal data fusion diagnosis device in an embodiment of the present application.
[0019] Figure 3 Shown is a schematic diagram of the structure of a wide network module in an embodiment of the present application.
[0020] Figure 4 Shown is the architecture diagram of the ViT model in the embodiment of the present application.
[0021] Figure 5 Shown is a structural schematic diagram of the feature fusion module in an embodiment of the present application.
[0022] Figure 6 Shown is a schematic diagram of the structure of the feature fusion unit in an embodiment of the present application.
[0023] Figure 7 Shown is a schematic diagram of fusion diagnosis of medical multimodal data in an embodiment of the present application.
[0024] Figure 8 Shown is a flow chart of the medical multimodal data fusion diagnosis method provided in an embodiment of the present application.
[0025] Component number description
[0026] 1 Electronic equipment
[0027] 11 Memory
[0028] 12 Display
[0029] 13 General Purpose Processor
[0030] 131 CPU
[0031] 132 Neural Network Processor
[0032] 2 Medical multimodal data fusion diagnostic device
[0033] 21 Data acquisition module
[0034] 22 Wide Network Module
[0035] 221 coding unit
[0036] 222 splicing unit
[0037] 23 Deep Network Module
[0038] 24 Feature Fusion Module
[0039] 241 First processing unit
[0040] 242 Second processing unit
[0041] 243 Feature Fusion Unit
[0042] 25 Diagnosis result acquisition module
[0043] Steps S81 to S85 DETAILED DESCRIPTION
[0044] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0045] In the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first XX and the second XX are only used to distinguish different XXs, and do not limit their order. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit them to be different.
[0046] It should be noted that, in the embodiments of the present application, the words "exemplarily" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.
[0047] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.
[0048] In the field of modern medical diagnosis, the comprehensive application of multimodal data has become an important means to improve the accuracy and comprehensiveness of diagnosis. The data types of multimodal data include medical images, genomic data, electronic health records, etc. Each data source carries patient information at different levels and dimensions. The integration of these data can provide doctors with a more comprehensive and comprehensive basis for diagnosis.
[0049] In some technical solutions, multimodal data is fused by early fusion, late fusion, etc. However, this method cannot fully utilize the deep correlation between data and leads to partial loss of diagnostic information, making it difficult to effectively integrate the fusion results and mine the information of multimodal data to provide more accurate and personalized diagnostic support.
[0050] In some technical solutions, multimodal data is fused based on direct connection or weighting methods. This approach cannot effectively solve the complementarity problem in the feature space, which will lead to a decrease in the performance of the model when processing more complex or heterogeneous data sets. In addition, these methods usually require a lot of parameter adjustment and optimization to achieve the ideal fusion effect, which increases the complexity and cost of model deployment.
[0051] At least to address the above problems, the present application provides a medical multimodal data fusion diagnosis device. The medical multimodal data fusion diagnosis device provided by the present application will be described in detail below by way of specific embodiments combined with the accompanying drawings.
[0052] Figure 1 The structure diagram of an electronic device in the embodiment of the present application is shown. The medical multimodal data fusion diagnosis device provided in the embodiment of the present application can be deployed in Figure 1 The electronic equipment shown. Figure 1 As shown, the electronic device 1 includes a memory 11 , a display 12 and at least one general purpose processor 13 .
[0053] The memory 11 may include a volatile memory, such as a random access memory (RAM) and a cache. The memory 11 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD). The memory 11 may be used to store program instructions for the processor to call and execute at least a portion of the functions of the medical multimodal data fusion diagnosis device provided in the embodiment of the present application.
[0054] The display 12 may specifically include a display screen (display panel). In some implementations, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. In addition, the display 12 may also be a touch panel (touch screen, touch screen), which may include a display screen and a touch-sensitive surface. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the general processor 13 to determine the type of touch event, and then the general processor 13 provides a corresponding visual output on the display device according to the type of touch event.
[0055] The general processor 13 can be any type of device capable of processing electronic instructions. In the embodiment of the present application, the electronic device 1 may include one or more general processors 13, such as one or two of a central processing unit (CPU) 131 and a neural network processor (NPU) 132. In addition, it may also include one or more of a graphics processor (GPU), a microprocessor, a microcontroller, a main processor, a controller, and an ASIC (Application Specific Integrated Circuit). The general processor 13 is configured to execute various types of digital storage instructions, such as software or firmware programs stored in the memory 11, which enables the electronic device 1 to provide a variety of services. For example, the processor 11 can execute a program or process data to perform at least a portion of the functions of the medical multimodal data fusion diagnosis device provided in the embodiment of the present application.
[0056] The main function of the central processing unit 131 is to parse computer instructions and process data in computer software, realize overall control of the electronic device 1, and control and allocate all hardware resources of the electronic device 1 (such as storage resources, communication resources, I / O interfaces, etc.).
[0057] Neural network processor is a general term for new processors based on neural network algorithms and acceleration. It is specially designed for artificial intelligence and is used to accelerate neural network calculations and solve the problem of low efficiency of traditional chips in neural network calculations.
[0058] It should be noted that the name of the neural network processor does not constitute a limitation of the present application. For example, in other application scenarios, the neural network processor can also be transformed and replaced by other processors with similar functions, such as a tensor processor (Tensor Processing Unit, TPU), a deep learning processor (Deep learning Processing Unit, DPU), and so on.
[0059] Figure 2 The structure diagram of the medical multimodal data fusion diagnosis device 2 provided in the embodiment of the present application is shown. Figure 2 As shown, the medical multimodal data fusion diagnosis device 2 includes a data acquisition module 21, a wide network module 22, a deep network module 23, a feature fusion module 24 and a diagnosis result acquisition module 25.
[0060] The data acquisition module 21 is used to acquire multimodal medical data. Multimodal medical data refers to information from different types of medical data sources, which may include a variety of different forms, such as images, texts, signals, etc. Each data modality provides information about different aspects of the patient's health status. In an embodiment of the present application, multimodal medical data includes medical imaging data, patient statistics, and blood test data. Among them, medical imaging data, for example, includes X-rays, computed tomography (CT) images, magnetic resonance imaging (MRI) images, ultrasonic images, etc. Patient statistics information, for example, includes information such as the patient's age, gender, historical disease records, etc. Blood test data are various biochemical and physiological indicators obtained by analyzing blood samples, such as blood routine test data, biochemical test data, immunological test data, coagulation function test data, etc.
[0061] The wide network module 22 is used to process the patient's statistical information and blood test data to obtain the first feature vector. The wide network mainly captures the relationship between the input features by directly connecting the input features and feature combinations to the output layer. In the embodiment of the present application, the first feature vector is extracted from the patient's statistical information and blood test data through the wide network, which can retain the linear relationship between the wide features and the original data and enhance the memory capacity of the model.
[0062] The deep network module 23 is used to process the medical image data to obtain the second feature vector. The deep network (DeepNetwork) gradually extracts and combines the features of the input data through multiple layers of nonlinear transformations, and can automatically learn complex feature representations. In the embodiment of the present application, the deep network can be used to extract the deep features of the medical image data. These features can reveal the nonlinear and complex patterns in the medical image data, which is particularly critical for understanding complex medical images.
[0063] The feature fusion module 24 is used to fuse the first feature vector and the second feature vector using a cross-attention fusion technique to obtain a fusion feature. The cross-attention fusion technique uses an attention mechanism to establish a connection between data of different modalities to better fuse information from multiple modalities, thereby achieving a good fusion effect.
[0064] The diagnosis result acquisition module 25 is used to acquire the diagnosis result according to the fusion feature.
[0065] According to the above content, the medical multimodal data fusion diagnosis device provided in the embodiment of the present application comprehensively utilizes wide and deep networks (Wide and Deep Networks) and cross-attention fusion technology, which not only optimizes the feature fusion process from different data sources, but also ensures the maximum preservation of the information of the fused features. The application of cross-attention fusion ensures that deep features are efficiently extracted and fused on the basis of maintaining data characteristics, thereby significantly improving the accuracy and efficiency of medical diagnosis.
[0066] Figure 3 The diagram is a schematic diagram of the structure of the wide network module 22 in the embodiment of the present application. Figure 3 As shown, the wide network module 22 includes an encoding unit 221 and a splicing unit 222 .
[0067] The encoding unit 221 is used to perform one-hot encoding and sparse matrix encoding on the patient statistical information and blood test data.
[0068] The concatenation unit 222 is used to concatenate the one-hot encoding results of the patient statistical information and the blood test data and the sparse matrix encoding results to obtain a first eigenvector.
[0069] Exemplarily, the encoding unit 221 may perform one-hot encoding and sparse matrix encoding on the patient statistical information and the blood test data respectively to obtain four encoding results. The concatenation unit 222 concatenates the four encoding results to obtain a first eigenvector.
[0070] Exemplarily, the encoding unit 221 may use sklearn.feature_extractor.DictVectorizer (sparse = True) to perform sparse matrix encoding on the patient statistical information and blood test data. Wherein, sklearn.feature_extraction.DictVectorizer is a tool in the scikit-learn library for encoding categorical features into sparse matrices, where the key is the feature name and the value is the feature value.
[0071] In some implementations, the deep network module 23 uses the ViT model to extract deep features of the medical imaging data to obtain a second feature vector, and these deep features can reveal nonlinear and complex patterns in the data.
[0072] Figure 4Shown is a schematic diagram of the architecture of the ViT model. The ViT model includes several Transformer Blocks. For example, in some embodiments, the ViT model includes 12 Transformer Blocks. Transformer block includes MLP (Multi-Layer Perceptron), MSA (Multi-Head Self-Attention), LayerNorm (layer normalization) and residual connection. MLP is, for example, a two-layer fully connected network using a GELU activation function. MLP, MSA, LayerNorm and residual connections can enhance the learning ability and stability of the neural network, thereby better extracting deep features of medical imaging data.
[0073] Figure 5 The structure diagram of the feature fusion module 24 in the embodiment of the present application is shown. Figure 5 As shown, the feature fusion module 24 includes a first processing unit 241 , a second processing unit 242 and a feature fusion unit 243 .
[0074] The first processing unit 241 is used to create a query matrix from the first eigenvector. Specifically, the query matrix is generated by the first eigenvector through linear transformation, and is used to find information in the key matrix related thereto. Each row of the query matrix corresponds to a query vector, which is used to calculate the correlation with each key vector in the key matrix.
[0075] The second processing unit 242 is used to create a key matrix and a value matrix from the second eigenvector. The key matrix and the value matrix are generated by linear transformation of the second eigenvector. Each row of the key matrix corresponds to a key vector, and each key vector is dot-producted with the query vector to evaluate the similarity or correlation between the query vector and the key vector. Each row of the value matrix corresponds to a value vector. When the correlation between the query vector and the key vector is determined, the value vectors in the value matrix are weighted and summed to generate the final attention output.
[0076] The feature fusion unit 243 is used to obtain fusion features according to the query matrix, the key matrix and the value matrix.
[0077] Figure 6 The structure diagram of the feature fusion unit 243 in the embodiment of the present application is shown. Figure 6 As shown, the feature fusion unit 243 includes an attention score calculation subunit 2431 and a feature fusion subunit 2432. The attention score calculation subunit 2431 is used to calculate the attention scores of the query matrix and the key matrix and normalize them. The feature fusion subunit 2432 is used to weightedly fuse the normalized attention scores with the value matrix to obtain a fused feature.
[0078] For example, the fused feature f fusion for:
[0079]
[0080] Among them, softmax is the activation function, Q is the query matrix, K is the key matrix, V is the value matrix, d q is the dimension of the query matrix.
[0081] In some implementations, the diagnosis result acquisition module 25 inputs the fused features into a 1-layer MLP to obtain a diagnosis result. The 1-layer MLP may include one or more hidden layers and an output layer. Each hidden layer may include a nonlinear activation function, such as ReLU (Rectified Linear Unit), to introduce nonlinearity. The output layer may use a softmax activation function to convert the output of the MLP into a probability distribution representing different diagnosis results. By training the MLP, the diagnosis result acquisition module 25 can map the fused features to the correct diagnosis result, thereby achieving accurate diagnosis and prediction.
[0082] Figure 7 The following is a schematic diagram of a medical multimodal data fusion diagnostic device in an example. Figure 7 As shown, multimodal medical data includes medical imaging data, patient statistics and blood test data. In this example, patient statistics and blood test data are one-hot encoded and sparse matrix encoded using sklearn.feature_extractor.DictVectorizer(sparse=True), and then the encoding results are concatenated to obtain the first feature vector. The medical imaging data is input into the ViT model, and the ViT model extracts the second feature vector of the medical imaging data. The first feature vector and the second feature vector are fused using the cross attention fusion technique to obtain the fused feature. The fused feature is input into a 1-layer MLP to obtain the diagnosis result.
[0083] The embodiment of the present application also provides a medical multimodal data fusion diagnosis method. Figure 8 The flowchart of the medical multimodal data fusion diagnosis method is shown. Figure 8 As shown, the medical multimodal data fusion diagnosis method provided in the embodiment of the present application includes the following steps S81 to S85.
[0084] S81, obtaining medical imaging data, patient statistics and blood test data.
[0085] S82, processing the patient's statistical information and blood test data to obtain a first feature vector.
[0086] S83, processing the medical image data to obtain a second eigenvector.
[0087] S84, using cross attention fusion technology to fuse the first feature vector and the second feature vector to obtain a fusion feature.
[0088] S85, obtaining a diagnosis result according to the fusion feature.
[0089] It should be noted that the above steps S81 to S85 are Figure 2 The modules in the medical multimodal data fusion diagnosis device 2 are in one-to-one correspondence and will not be described in detail here.
[0090] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.
[0091] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0092] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0093] The present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by the processor, the medical multimodal data fusion diagnosis method provided by the present application is implemented. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.
[0094] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.
[0095] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.
[0096] The embodiment of the present application also provides an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program. The processor is communicatively connected to the memory, and executes the medical multimodal data fusion diagnosis method provided in the embodiment of the present application when the computer program is called.
[0097] In some implementations, the electronic device may further include a display. The display is communicatively connected to the memory and the processor, and is used to display a GUI related to the medical multimodal data fusion diagnosis method provided in the embodiment of the present application.
[0098] In summary, the medical multimodal data fusion diagnosis device provided in the embodiment of the present application comprehensively utilizes the wide and deep network and the cross-attention fusion technology to effectively process and fuse medical data from different data sources to improve the accuracy and efficiency of diagnosis. The medical multimodal data fusion diagnosis technology provided in the embodiment of the present application has important practical significance and broad application prospects for processing multimodal and high-dimensional medical data. In addition, the medical multimodal data fusion diagnosis device provided in the embodiment of the present application can reduce the misdiagnosis caused by missing or insufficient information. Through the processing of the deep network and the information capture of the wide network, the risk of misjudgment or missed diagnosis that may occur during the diagnosis process can be effectively reduced, and the reliability of the diagnosis is improved. Compared with the traditional method, the wide and deep network based on cross-attention in the embodiment of the present application can process and analyze a large amount of multimodal medical data more quickly. This helps to accelerate the diagnosis process, make medical resources more efficiently utilized, and provide treatment advice to patients in a timely manner. Therefore, the present application effectively overcomes the various defects in the prior art and has a high industrial value and a wide application prospect.
[0099] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.
[0100] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.
Claims
1. A medical multimodal data fusion diagnostic device, characterized in that: include: A data acquisition module for acquiring medical imaging data, patient statistics, and blood test data; A wide network module, used for processing the patient statistical information and the blood test data to obtain a first feature vector; A deep network module, used for processing the medical image data to obtain a second eigenvector; A feature fusion module, used for fusing the first feature vector and the second feature vector using a cross attention fusion technique to obtain a fusion feature; The diagnosis result acquisition module is used to acquire the diagnosis result according to the fusion feature.
2. The medical multimodal data fusion diagnostic device according to claim 1, characterized in that: The wide network module comprises: An encoding unit, used for performing one-hot encoding and sparse matrix encoding on the patient statistical information and the blood test data; A concatenation unit is used to concatenate the patient statistical information and the one-hot encoding result and the sparse matrix encoding result of the blood test data to obtain the first feature vector.
3. The medical multimodal data fusion diagnostic device according to claim 1, characterized in that: The deep network module uses the ViT model to extract deep features of the medical image data to obtain the second feature vector.
4. The medical multimodal data fusion diagnosis device according to claim 3, characterized in that: The ViT model includes several Transformer blocks, and the Transformer block includes MLP, MSA, LayerNorm and residual connection.
5. The medical multimodal data fusion diagnostic device according to claim 1, characterized in that: The feature fusion module includes: a first processing unit, configured to create a query matrix from the first feature vector; a second processing unit, configured to create a key matrix and a value matrix from the second eigenvector; A feature fusion unit is used to obtain the fused feature according to the query matrix, the key matrix and the value matrix.
6. The medical multimodal data fusion diagnostic device according to claim 5, characterized in that: The feature fusion unit comprises: An attention score calculation subunit, used to calculate the attention scores of the query matrix and the key matrix and normalize them; The feature fusion subunit is used to perform weighted fusion on the normalized attention score and the value matrix to obtain the fused feature.
7. The medical multimodal data fusion diagnostic device according to claim 1, characterized in that: The diagnosis result acquisition module inputs the fused features into a 1-layer MLP to obtain the diagnosis result.
8. A medical multimodal data fusion diagnosis method, characterized in that: include: Access medical imaging data, patient demographic information, and blood test data; Processing the patient statistical information and the blood test data to obtain a first feature vector; Processing the medical image data to obtain a second eigenvector; Using a cross attention fusion technique to fuse the first feature vector and the second feature vector to obtain a fusion feature; A diagnosis result is obtained according to the fusion feature.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the medical multimodal data fusion diagnosis method according to claim 8 is implemented.
10. An electronic device, characterized in that: The electronic device comprises: A memory storing a computer program; A processor is communicatively connected to the memory, and executes the medical multimodal data fusion diagnosis method according to claim 8 when calling the computer program.
