Multi-modal data fusion diagnosis auxiliary method and device, storage medium and equipment

Abstract

The invention provides a multi-modal data fusion diagnosis assistance method and device, a storage medium and equipment, and the method comprises the steps: obtaining multi-modal data sent by a data collection module, the multi-modal data comprising at least two of 2D fundus color photo data, 3D OCT body data, OCTA blood flow parameters and wide-area fundus spliced image parameters; inputting the multi-modal data into a trained layered feature extraction module to obtain a multi-modal data feature vector; inputting the multi-modal data feature vector into a trained feature compensation-fusion module to obtain a complete multi-modal data fusion feature; and inputting the complete multi-modal data fusion features into a trained dynamic expert hybrid system to obtain structured diagnosis and treatment data and a text diagnosis auxiliary report, and sending the structured diagnosis and treatment data and the text diagnosis auxiliary report to a data display module.

Description

Technical Field

[0001] This invention relates to the field of medical diagnostics, and more specifically to diagnostic aids, devices, storage media, and equipment that integrate multimodal data. Background Technology

[0002] In traditional large-scale medical model deployments, data typically originates from relatively stable internal hospital information systems, such as Electronic Medical Record Systems (EMRS) and Picture Archiving and Communication Systems (PACS). Data acquisition is generally centralized and standardized. However, with the increasing dynamism of the medical environment, mobile diagnostic vehicles are finding more and more applications, leading to a more complex and diverse range of data sources. These sources include not only basic patient information and disease descriptions, but also various physiological data collected by the mobile diagnostic vehicle and image data acquired by its imaging equipment. This data is inconsistent, transmission is unstable, and there are significant differences in image dimensions (including 2D and 3D images), resolution (from megapixels to hundreds of millions of pixels), and medical meaning (such as structural analysis and fundus blood flow function assessment). Therefore, efficiently acquiring and integrating this multi-source, heterogeneous data is one of the key challenges that mobile diagnostic vehicles need to address.

[0003] Traditional large-scale medical models are mostly deployed in hospitals with their powerful computer rooms and high-performance computing clusters, providing ample computing resources to support model operation and training. However, mobile medical vehicles have limited space and cannot accommodate large-scale computing devices, resulting in relatively scarce computing resources. This necessitates that large-scale multimodal base models for mobile medical vehicles, while ensuring performance, possess lightweight and efficient computing characteristics. They should be able to quickly process and analyze ophthalmic multimodal data, such as fundus images and video data, with limited computing resources, achieving accurate diagnostic assistance without diagnostic delays or errors caused by insufficient computing resources.

[0004] Existing large-scale ophthalmic AI models typically rely on massive parameter scales (exceeding one billion parameters), requiring high-performance computing equipment. While their real-time requirements are relatively relaxed, allowing for batch processing and optimization in the background, mobile diagnostic vehicles have limited computing resources. Directly deploying large models in these vehicles leads to slow response times and excessive energy consumption, failing to meet the real-time diagnostic needs of mobile diagnostic vehicles. Furthermore, considering the diversity and urgency of patients treated in mobile diagnostic vehicles, the deployed models need to quickly adapt to different clinical scenarios. For example, in the emergency treatment of patients with ocular trauma, they need to rapidly analyze relevant imaging and physiological data to provide accurate diagnostic and treatment recommendations. Summary of the Invention

[0005] Based on this, the present invention provides a diagnostic assistance method, device, storage medium and equipment that integrates multimodal data. It uses a specific model to extract features based on the characteristics of various modal data, and dynamically integrates the features of multiple modal data by combining a missing modality compensation mechanism. It also obtains auxiliary diagnostic results by integrating a modality-specific expert mechanism, enabling mobile medical vehicles to quickly meet various medical needs and provide accurate and efficient diagnostic and treatment suggestions.

[0006] In a first aspect, the present invention provides a method for assisting diagnosis by fusing multimodal data. This method is applied to a mobile diagnostic vehicle, which includes at least two data acquisition modules, a data processing module, and a data display module. The data processing module is communicatively connected to both the data acquisition modules and the data display module. The data processing module executes the method for assisting diagnosis by fusing multimodal data, comprising the following steps:

[0007] Acquire multimodal data sent by the data acquisition module, wherein the multimodal data includes at least two of the following: 2D fundus color image data, 3DOCT volume data, OCTA blood flow parameters, and wide-area fundus stitched image parameters;

[0008] The multimodal data is input into the trained hierarchical feature extraction module to obtain the multimodal data feature vector;

[0009] The multimodal data feature vectors are input into the trained feature compensation-fusion module to obtain complete multimodal data fusion features;

[0010] The complete multimodal data fusion features are input into the trained dynamic expert hybrid system to obtain structured diagnosis and treatment data and text-based diagnostic assistance reports, which are then sent to the data display module.

[0011] Furthermore, the step of inputting the multimodal data into the trained hierarchical feature extraction module to obtain the multimodal data feature vector specifically includes the following steps:

[0012] Extract data type labels from the multimodal data;

[0013] Based on the data type labels, the multimodal data of each type is pushed to the corresponding feature extraction module to obtain the feature vector of each type of multimodal data.

[0014] Furthermore, the step of pushing multimodal data of each type to the corresponding feature extraction module according to the data type label to obtain feature vectors of each type of multimodal data includes:

[0015] If the data type label includes a first label, the first modal data is extracted according to the data address indicated by the first label, and the first modal data is input into the ResNet model that combines spatial attention mechanism to obtain the first modal data feature vector;

[0016] If the data type label includes a second label, the second modal data is extracted according to the data address indicated by the second label, and the second modal data is input into the 3D Swin Transformer model to obtain the feature vector of the second modal data;

[0017] If the data type label includes a third label, the third modality data is extracted according to the data address indicated by the third label, and the third modality data is input into the 3D U-Net model to obtain the third modality data feature vector;

[0018] If the data type label includes a fourth label, the fourth modality data is extracted according to the data address indicated by the fourth label, and the fourth modality data is input into the block model to obtain the fourth modality data feature vector.

[0019] Furthermore, the step of inputting the multimodal data feature vector into the trained feature compensation-fusion module to obtain complete multimodal data fusion features specifically includes the following steps:

[0020] If the types of multimodal data feature vectors do not meet the conditions for complete fusion data, the multimodal data feature vectors are input into the conditional adversarial network to generate approximate feature vectors for the missing modalities.

[0021] The feature vectors of the multimodal data are concatenated by combining the feature confidence scores to obtain the complete multimodal data fusion features.

[0022] Furthermore, the step of inputting the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnostic data and text-based diagnostic assistance reports includes:

[0023] The complete multimodal data fusion features are input into the trained dynamic expert hybrid system, and the weights of each modality-specific expert model are determined by the dynamic routing algorithm of the gated network.

[0024] Based on the weights of each modality-specific expert model, relevant modality-specific expert models are activated. After weighted fusion of the preliminary diagnostic results output by several expert models, the results are passed to the fully connected network of the output layer to generate a text-based diagnostic assistance report.

[0025] Secondly, the present invention also provides a fusion multimodal data diagnostic auxiliary device, which is applied to a mobile diagnostic vehicle. The mobile diagnostic vehicle includes at least two data acquisition modules, a data processing module, and a data display module. The data processing module is communicatively connected to both the data acquisition modules and the data display module. The data processing module executes the fusion multimodal data diagnostic auxiliary device and includes the following modules:

[0026] The data acquisition module is used to acquire multimodal data sent by the data acquisition module. The multimodal data includes at least two of the following: 2D fundus color image data, 3D OCT volume data, OCTA blood flow parameters, and wide-area fundus stitched image parameters.

[0027] The feature extraction module is used to input the multimodal data into the trained hierarchical feature extraction module to obtain multimodal data feature vectors;

[0028] The feature completion module is used to input the multimodal data feature vector into the trained feature compensation-fusion module to obtain complete multimodal data fusion features;

[0029] The diagnostic assistance module is used to input the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnosis and treatment data and text diagnostic assistance reports, and send the structured diagnosis and treatment data and text diagnostic assistance reports to the data display module.

[0030] Furthermore, the feature completion module includes the following units:

[0031] The missing modality feature generation unit is used to generate approximate feature vectors of the missing modality by inputting the multimodal data feature vectors into the conditional adversarial network if the types of the multimodal data feature vectors do not meet the condition for complete fusion data.

[0032] The feature concatenation unit is used to concatenate the multimodal data feature vectors and the multimodal data feature vectors by combining feature confidence to obtain complete multimodal data fusion features.

[0033] Furthermore, the step of inputting the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnostic data and text-based diagnostic assistance reports includes:

[0034] The complete multimodal data fusion features are input into the trained dynamic expert hybrid system, and the weights of each modality-specific expert model are determined by the dynamic routing algorithm of the gated network.

[0035] Based on the weights of each modality-specific expert model, relevant modality-specific expert models are activated. After weighted fusion of the preliminary diagnostic results output by several expert models, the results are passed to the fully connected network of the output layer to generate a text-based diagnostic assistance report.

[0036] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the integrated multimodal data diagnostic assistance methods in the first aspect.

[0037] Fourthly, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform any one of the multimodal data fusion diagnostic assistance methods in the first aspect.

[0038] The beneficial effects of adopting the above technical solution are as follows: This embodiment effectively improves the model's ability to capture key case features and enhances the model's interpretability by introducing spatial and channel attention mechanisms and deformable convolution to process multimodal data; then, by using conditional adversarial networks to generate cross-modal features, it can provide reliable diagnostic support in certain medical image modalities, which not only improves data utilization but also dynamically adjusts decisions based on quantified uncertainty, ensuring the accuracy and reliability of diagnosis; finally, by adopting a modality-specific expert system combined with a gating network design, it achieves effective resource allocation and improved computational efficiency, reduces unnecessary computation, and can dynamically activate relevant experts according to the characteristics of input data, thereby improving the scope and capability of mobile diagnostic vehicles in assisting case diagnosis. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0040] Figure 1 This is a schematic diagram of a diagnostic assistance method that integrates multimodal data in one embodiment of this application;

[0041] Figure 2 This is a schematic diagram of a diagnostic assistance method that integrates multimodal data in one embodiment of this application;

[0042] Figure 3 This is a schematic diagram of a diagnostic auxiliary device that integrates multimodal data in one embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. To describe the present invention in more detail, the fusion multimodal data diagnostic auxiliary method, apparatus, storage medium, and device provided by the present invention will be specifically described below with reference to the accompanying drawings.

[0044] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an," "a," or "the" do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. The terms "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0045] Ophthalmic image analysis is an important branch of medical imaging. In recent years, with the rapid development of optical imaging technology and artificial intelligence algorithms, such as Optical Coherence Tomography (OCT), Optical Coherence Tomography Angiography (OCTA), and ultra-wide-angle imaging, ophthalmic image analysis has expanded from simple image segmentation to intelligent diagnosis of complex diseases. However, different modalities of data structures differ significantly. For example, 2D fundus color images and 3D OCT data differ significantly in spatial resolution and imaging principles. Effectively aligning these heterogeneous data and extracting key features helpful for diagnosis has become a major challenge. Traditional block modal alignment methods often use simple feature stitching or mapping to a common spatial representation, but this method ignores the unique information hierarchy between heterogeneous data modalities, leading to limited diagnostic performance. Furthermore, for complex lesions, such as diabetic retinopathy, it is necessary to combine information from multiple modalities for comprehensive judgment. This requires the model to not only process data from a single modality but also possess strong cross-modal fusion capabilities to accurately capture the lesion area and its influence range.

[0046] Due to limitations in equipment and patient conditions, certain modalities may be missing. In such cases, relying solely on existing technologies for diagnosis may lead to biased results. Currently, methods for compensating for missing modalities mainly include interpolation and prediction based on statistical models. However, these methods often fail to fully consider the intrinsic relationships between different modalities, resulting in low-quality generated data that is insufficient to meet the needs of precision medicine. In recent years, with the development of Generative Adversarial Networks (GANs), using conditional GANs to generate known modalities from known modalities has become a new direction. However, these methods still face many challenges when dealing with high-dimensional medical images, such as how to ensure the authenticity and consistency of the generated data, and how to assess the reliability of the generated results.

[0047] With the widespread application of deep learning models in medical image analysis, their computational complexity and resource consumption have become significant factors limiting their practical adoption. Especially in resource-constrained environments, such as mobile devices or remote medical institutions, running large deep learning models becomes impractical. Therefore, developing efficient and accurate lightweight inference engines is crucial. Existing lightweight model design strategies mainly include model pruning, quantization, and low-rank decomposition, but these methods often sacrifice some accuracy for computational efficiency gains. Furthermore, most lightweight models are primarily designed for single-task or modal optimization, lacking effective support for multimodal data and struggling to adapt to complex clinical application scenarios.

[0048] Currently, the mainstream approach is to deploy large-scale ophthalmology-related model systems in traditional medical settings. Among these, VisionFM is the most similar. It first utilizes a large and diverse amount of ophthalmological data for pre-training, covering various ophthalmic diseases, imaging modalities (including fundus photography (FP), optical coherence tomography (OCT), and fluorescein angiography (FFA),) different equipment, and demographic information. To further enhance its performance, in addition to real-world data, synthetic ophthalmic imaging data is used. This data undergoes a visual Turing test to ensure its quality is sufficient to support VisionFM's learning process, thereby achieving significant performance improvements in downstream ophthalmic AI tasks.

[0049] Based on this, the present invention provides a method, apparatus, storage medium, and device for diagnostic assistance using fused multimodal data. Applying the fused multimodal data diagnostic assistance method to a mobile diagnostic vehicle, when processing multimodal data, the introduction of spatial and channel attention mechanisms and deformable convolution effectively improves the model's ability to capture key pathological features and enhances the model's interpretability. Utilizing conditional adversarial networks to generate cross-modal features provides reliable diagnostic support when certain modal image data is available, not only improving data utilization but also ensuring diagnostic accuracy and reliability by dynamically adjusting decision thresholds through quantified uncertainty. The design of a modality-specific expert system combined with a gating network achieves effective resource allocation and improved computational efficiency, reducing unnecessary computation and dynamically activating relevant experts based on the characteristics of the input data, thereby improving the ability to diagnose complex pathologies. The method is illustrated using an application to a terminal device as an example, in conjunction with the attached... Figure 1 The diagram shown illustrates a diagnostic aid method that integrates multimodal data. Figure 2 The diagram shows a flowchart of a diagnostic assistance method that integrates multimodal data.

[0050] The fusion multimodal data diagnostic assistance method of this application is applied to a mobile medical vehicle, which includes at least two data acquisition modules, a data processing module, and a data display module.

[0051] The data acquisition module is used to acquire medical imaging data of various modalities, such as 2D fundus ultrasound images, 3D OCT volumetric data, OCTA blood flow parameters, and wide-area fundus stitched images. The data acquisition module includes, but is not limited to, 2D fundus ultrasound acquisition equipment, 3D OCT volumetric data acquisition equipment, OCTA blood flow parameter acquisition equipment, and wide-area fundus stitched image acquisition equipment. In this embodiment, the 2D fundus ultrasound acquisition equipment can be the Zeiss CLARUS 500 ultra-wide-angle fundus camera; the 3D OCT volumetric data acquisition equipment can be the Zeiss CIRRUS HD-OCT 6000 or the Micro-Light Medical Cornaris Mobile mobile OCT system; the OCTA blood flow parameter acquisition equipment can be a wide-angle sweep frequency OCTA device or a multi-wavelength color imaging system; and the wide-area fundus stitched image equipment can be the Mirante full-color ultra-wide-angle laser imaging system or the Sovereign Ophthalmic Wide-Area Imaging System SW-8000.

[0052] The data processing module is used to process the multimodal data sent by the data acquisition module. The data processing module includes, but is not limited to, a control unit, an arithmetic unit, and registers. The control unit includes an instruction register, a program counter, and an instruction decoder for instruction decoding and operation timing control. The arithmetic unit performs arithmetic and logical operations, and the status register records the operation result flags. The storage unit includes a cache and a register group to temporarily store instructions and data to improve access efficiency.

[0053] The data display module is specifically the display module of the mobile medical vehicle, used to display multimodal data, structured medical data, and text-based diagnostic auxiliary reports.

[0054] Based on the aforementioned mobile diagnostic vehicle, and taking the implementation of a multimodal data fusion diagnostic assistance method by the data processing module as an application background, this paper explains the multimodal data fusion diagnostic assistance method:

[0055] Step S100: Acquire multimodal data sent by the data acquisition module. The multimodal data includes at least two of the following: 2D fundus color image data, 3D OCT volume data, OCTA blood flow parameters, and wide-area fundus stitched image parameters.

[0056] Specifically, 2D fundus color imaging data consists of static retinal images acquired through a fundus camera, including field of view, resolution, refractive compensation range, and pupillary requirements. This modal data can be used to annotate the macular region, optic disc, and vascular morphology, reflecting conditions such as diabetic retinopathy and macular degeneration. 3D OCT volumetric data is a collection of three-dimensional spatial structural information acquired through Optical Coherence Tomography (OCT). It is a three-dimensional voxel matrix (e.g., 550×550×550 pixels) formed by superimposing multiple consecutive two-dimensional tomographic scans (B-scans) along the slow axis, covering the three-dimensional spatial information of the sample. 3D OCT volumetric data can be used in mobile ophthalmic diagnostic vehicles to reflect retinal layered imaging and, combined with OCTA blood flow data, to perform vascular and structural fusion analysis. OCTA blood flow parameters are acquired through Optical Coherence Tomography Angiography (OCTA) and include structural parameters (such as vessel density, vessel diameter, and area of ​​the avascular zone in the fovea) and functional parameters (such as blood flow velocity and blood flow index). These parameters are used for precise blood flow analysis at different retinal levels. Wide-field fundus mosaic images, by integrating multi-field images, provide complete coverage of the central and peripheral areas of the fundus, offering data support for the early diagnosis of diseases such as diabetic retinopathy and glaucoma.

[0057] In this embodiment, the optometry device on the mobile medical vehicle can acquire refractive data and visual acuity in text form, the intraocular pressure meter can acquire visual acuity and intraocular pressure data, the fundus camera can acquire 2D fundus color images, and OCT and OCTA can acquire 3D volume data. All data acquisition modules on the mobile medical vehicle can operate simultaneously.

[0058] Step S200: Input the multimodal data into the trained hierarchical feature extraction module to obtain the multimodal data feature vector.

[0059] Specifically, in the field of ophthalmic image analysis, hierarchical feature extraction methods can effectively capture key pathological features from multimodal data. This embodiment pushes multimodal data to the corresponding feature extraction module according to the specific data type labels carried by the multimodal data, including the following steps:

[0060] Step S211: Extract data type labels from the multimodal data.

[0061] Step S212: Based on the data type label, push the multimodal data of each type to the corresponding feature extraction module to obtain the feature vector of each type of multimodal data.

[0062] Specifically, step S212 above, which pushes multimodal data of each type to the corresponding feature extraction module according to the data type label to obtain feature vectors of each type of multimodal data, may include the following steps:

[0063] Step S221: If the data type label includes a first label, extract the first modal data according to the data address indicated by the first label, input the first modal data into the ResNet model that combines spatial attention mechanism, and obtain the first modal data feature vector.

[0064] Specifically, when the data type label includes the first label, it indicates that the multimodal data contains 2D fundus color images. The first modality data (i.e., 2D fundus color images) is extracted according to the data address indicated by the first label, and the first modality data is input into a deep residual network (ResNet) model that combines spatial attention mechanism (Convolutional Block Attention Module, or CBAM) to obtain the feature vector of the first modality data, i.e., the 2D fundus color image feature vector.

[0065] In this embodiment, the ResNet model is also improved by adding residual connections, adjusting the number of channels, or optimizing the initialization strategy. For example, when using the improved ResNet model architecture, spatial attention mechanism (CBAM) is combined. After CBAM processes the first modality data, residual connections are added. Attention-enhanced features and original features are dynamically fused through attention correction to further strengthen feature expression, thereby enhancing the sensitivity to lesion areas and the feature extraction capability, so as to improve the recognition of fundus lesions such as macular edema and retinal hemorrhage.

[0066] Furthermore, to enhance the interpretability of the model, the attention heatmap of CBAM can be overlaid onto the original image using Grad-CAM or feature gradient visualization to help clinicians verify the rationality of the model's attention regions.

[0067] In addition, when labeled data is insufficient, the ResNet model can also perform self-supervised pre-training, such as through contrastive learning or masked image modeling, to improve its generalization ability in scenarios with a small number of samples.

[0068] Step S222: If the data type label includes a second label, extract the second modal data according to the data address indicated by the second label, input the second modal data into the 3D Swin Transformer model, and obtain the feature vector of the second modal data.

[0069] Specifically, when the data type label includes a second label, it indicates that the multimodal data contains 3D OCT volume data. The second modal data (i.e., 3D OCT volume data) is extracted according to the data address indicated by the second label, and the second modal data is input into the 3D Swin Transformer model to obtain the second modal data feature vector, i.e., the 3D OCT volume data feature vector.

[0070] In this embodiment, the 3D Swin Transformer utilizes local window attention to calculate voxel-level global contextual information and progressively reduces spatial resolution through hierarchical downsampling to extract multi-scale features. To maintain computational efficiency, this embodiment also employs Deformable Conv3D to compress the Z-axis dimension, preserving key information while reducing computational complexity. For example, the original 512×128×64 voxel data is compressed to 512×128×16, ensuring that the compressed 2.5D feature map still reflects the depth information of the lesion region. The 2.5D feature map represents an intermediate state between two-dimensional (2D) and three-dimensional (3D) representations.

[0071] Furthermore, to improve the sensitivity of the 3D Swin Transformer model to subtle structural changes, the 3DSwin Transformer can be pre-trained on large-scale unlabeled 3D OCTR volume data.

[0072] Step S223: If the data type label includes a third label, extract the third modality data according to the data address indicated by the third label, input the third modality data into the 3D U-Net model, and obtain the third modality data feature vector.

[0073] Specifically, when the data type label includes a third label, it indicates that the multimodal data contains OCTA blood flow parameters. The third modality data (i.e., OCTA blood flow parameters) is extracted according to the data address indicated by the third label, and the third modality data is input into the 3D U-Net model to obtain the third modality data feature vector, i.e., the vascular network structure vector (including vascular structure and vascular topology features).

[0074] Specifically, for OCTA blood flow parameters, a 3D U-Net model is used to segment the vascular structure and extract vascular topological features, ultimately outputting a 256-dimensional vector representing the vascular network structure. This process involves not only an encoder-decoder structure for extracting multi-scale features but also morphological processing of the segmented vascular mask, such as connectivity analysis and branch detection. To better understand the hemodynamic mechanisms of lesions, the correlation between the vascular segmentation mask and blood flow parameters is visualized to enhance the model's interpretability. In this embodiment, the process of segmenting vascular structures and extracting vascular topological features using a 3D U-Net model is as follows: Utilizing the connected component analysis characteristics of the 3D U-Net model, small-area noise (such as missegmented pixels) is removed by marking connected components in the vascular mask, while preserving the true vascular structure (e.g., setting a minimum connected area threshold of 50 pixels and filtering regions smaller than the threshold to remove small-area noise while preserving the true vascular structure as much as possible). Secondly, using branch detection characteristics, the main trunk and branches of the blood vessels are extracted, and topological features such as the number of branch points and vessel length are calculated. The segmented vascular mask is then skeletonized to generate a tree-like vascular structure, and the branch point density is statistically analyzed to better assess the degree of vascular lesions. Finally, morphological filtering characteristics are used to smooth vascular boundaries, fill small holes, and improve the continuity and accuracy of the mask.

[0075] Step S224: If the data type label includes a fourth label, extract the fourth modal data according to the data address indicated by the fourth label, input the fourth modal data into the block model, and obtain the fourth modal data feature vector.

[0076] Specifically, when the data type label includes the fourth label, it indicates that the multimodal data contains wide-area fundus images. The fourth modality data (i.e., the wide-area fundus image) is extracted according to the data address indicated by the fourth label, and the fourth modality data is input into the block model. The block strategy is adopted to evenly divide the high-resolution wide-area fundus image into multiple sub-blocks. The fourth modality data feature vector, i.e., the global feature vector, is obtained by combining the ViT-Tiny encoded sub-blocks with shared weights and GraphSAGE aggregation.

[0077] The block model not only solves the computational efficiency problem of wide-area fundus images, but also captures the contextual relationships between sub-blocks, such as the trend of lesion spread or the global connectivity of vascular networks.

[0078] Furthermore, to improve the robustness of local features, ViT-Tiny was pre-trained on large-scale unlabeled wide-area fundus images.

[0079] Multi-task learning of global features and local lesion features from 2D fundus images can help jointly predict disease severity, such as glaucoma staging.

[0080] Step S300: Input the multimodal data feature vector into the trained feature compensation-fusion module to obtain complete multimodal data fusion features.

[0081] Specifically, step S300, which inputs the multimodal data feature vector into the trained feature compensation-fusion module to obtain the complete multimodal data fusion feature, includes the following specific steps:

[0082] Step S310: If the types of multimodal data feature vectors do not meet the conditions for complete fusion data, the multimodal data feature vectors are input into the conditional adversarial network to generate approximate feature vectors for the missing modalities.

[0083] Step S320: Combine the feature confidence level to concatenate the approximate feature vector and the multimodal data feature vector to obtain the complete multimodal data fusion feature.

[0084] Specifically, in this embodiment, based on the original hierarchical feature extraction architecture of the feature compensation-fusion module in step 300, a cross-modal feature generator and an uncertainty perception module are further integrated to form a complete true modality compensation mechanism, ensuring reliable diagnostic support even when data is incomplete. The cross-modal feature generator and the uncertainty perception module further extend and improve the hierarchical feature extraction architecture, jointly constituting a missing modality compensation mechanism to address diagnostic problems in scenarios with incomplete data. The cross-modal feature generator is located after the unified representation encoder and immediately before the multimodal fusion module. It takes into account a unified feature vector (including missing modality markers, such as "OCT missing") and consistent modality features (such as fundus color imaging features and OCTA features), outputs generated missing modality approximation features (such as simulated OCT features), and concatenates the missing modality approximation features with the actual known features to form a complete feature vector. The uncertainty perception module runs in parallel with the cross-modal feature generator and is used to generate feature fusion results. It takes as input the confidence assessment of the generated features (such as the variance calculated by Monte Carlo Dropout), outputs a dynamically adjusted decision threshold and uncertainty heatmap, and feeds it back to the multimodal fusion module to adjust the feature fusion weights (such as reducing the weight of high uncertainty features).

[0085] This embodiment learns the implicit mapping relationship between multimodal data through a Conditional GAN. When a certain modality of data is missing, it uses the features of other modalities to generate an approximate feature vector of the missing modality.

[0086] First, the multimodal data feature vectors obtained in step S200 (such as CBAM features from fundus color imaging, 2.5D features from OCT, and blood flow information from OCTA) are mapped to the same latent space. It should be noted that the same latent space here refers to the generated approximate feature vectors of missing modalities, which are used to concatenate features from other modalities into subsequent modules and belong to the same latent space. In this part, a unified representation encoder is used to map high-dimensional heterogeneous features of different morphologies to a shared semantic subspace (i.e., the aforementioned same latent space). A feature alignment strategy is adopted, introducing a modal attention gating module before the shared layer to dynamically adjust the contribution weights of different modal features to the unified space. Through cross-modal contrastive loss, different modal features of similar samples are forced to be close together in the same latent space.

[0087] The generator takes feature vectors from multiple modalities as input (missing modalities are labeled as missing, and other modalities are used as conditions) and outputs approximate feature vectors for the missing modalities. Adversarial training ensures consistency in statistical distribution between the generated and real features. When multiple modalities are simultaneously missing, the system can still generate features through joint inference using the conditional information from the remaining modalities. For example, when OCT data is missing, the system will generate approximate 2.5D OCT feature vectors based on lesion region features from fundus ultrasound and vascular topology information from OCTA to reflect changes in retinal thickness or structural abnormalities.

[0088] The discriminator optimizes the fidelity of generated features through adversarial training, and combines feature matching loss to ensure that the generated features are statistically consistent with the real features. The generated feature vectors are then concatenated with the real features and input into the subsequent multimodal fusion module, for example, through cross-attention mechanisms or graph neural networks for collaborative analysis. The multimodal fusion module is positioned after the cross-modal feature generator and the uncertainty perception module, and before the dynamic expert hybrid system. This configuration integrates multimodal information (including generating missing modal features), enhances the complementarity between features, and provides more comprehensive input data for subsequent diagnosis. For example, fusing vascular features with OCT structural features through cross-attention can jointly analyze multimodal associations of ischemic lesions, such as diabetic retinopathy; feature-level fusion of 2.5D features with local region features from fundus color imaging helps improve cross-modal lesion localization capabilities.

[0089] Furthermore, by quantifying the uncertainty caused by modality loss and dynamically adjusting the model's decision threshold, a balance between diagnostic accuracy and reliability is achieved. Specific implementation steps include:

[0090] First, when the generator outputs, Monte Carlo Dropout or a Bayesian neural network is used to estimate the confidence of the generated features (such as the variance of OCT features).

[0091] Then, in the classification or segmentation task after multimodal fusion, the confidence level of the final prediction is calculated through ensemble learning or uncertainty propagation.

[0092] Finally, based on uncertainty and confidence, the classification threshold is dynamically adjusted: when the uncertainty of the generated features is high (e.g., the variance exceeds the threshold of 0.3), the confidence threshold is increased to reduce the risk of misdiagnosis and trigger the "re-examination recommended" flag; if the uncertainty is moderate, the threshold is reduced to improve the recall rate of lesions (e.g., early screening of diabetic retinopathy).

[0093] In addition, this embodiment can also set up a visual uncertainty heatmap to help doctors focus on potential error areas (such as marking the OCT generation area with an orange warning), and further optimize the fusion results by dynamically adjusting the weight of multimodal features (such as reducing the weight of the generation features when OCT is missing, and increasing the weight of OCTA vascular features).

[0094] Step S400: Input the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnosis and treatment data and text-based diagnostic assistance reports, and send the structured diagnosis and treatment data and text-based diagnostic assistance reports to the data display module.

[0095] Specifically, based on the complete multimodal data fusion features obtained from the above steps, a dynamic expert hybrid system can be used to weight and fuse the preliminary diagnostic results corresponding to each feature to obtain highly reliable structured diagnostic data and text-based diagnostic assistance reports. In step S400, the core of the dynamic expert hybrid system is a neural network framework trained on historical data, which serves as an intermediary module for feature fusion and decision-making.

[0096] Step S410: Input the complete multimodal data fusion features into the trained dynamic expert hybrid system, and determine the weights of each modality-specific expert model through the dynamic routing algorithm of the gating network.

[0097] The system receives complete multimodal data fusion features (such as CBAM features from fundus color images and 2.5D features from OCT) from a unified representation encoder via a gated network. It then uses patterns learned from historical training data to determine the weights of each modality-specific expert model through a dynamic routing algorithm.

[0098] Specifically, the core of weight calculation for each modality-specific expert model is to generate a sparse weight distribution through adaptive mapping of input features. The weights of the relevant modality-specific expert models to be activated are represented as follows: , The weights of the relevant modality-specific expert models to be activated. The initial weights for each modality-specific expert model. , For sparse mask The initial weights of each modality-specific expert model are sampled using the Gumbel-softmax reparameterization technique to generate a sparse mask.

[0099] Step S420: Select and activate relevant modality-specific expert models according to the weights of each modality-specific expert model, and after weighted fusion of the preliminary diagnostic results output by several expert models, pass them to the fully connected network of the output layer to generate a text diagnostic auxiliary report.

[0100] Based on the weights of each modality-specific expert model calculated in step S410, the modality-specific expert model most relevant to the current input (such as a fundus lesion analysis expert or a retinal structure analysis expert) is selected and activated. The preliminary diagnostic results output by these expert models are weighted and fused, and then passed to the fully connected network of the output layer to generate structured diagnostic results (such as lesion classification and severity score). Finally, the text generation module converts the results into a diagnostic report.

[0101] Furthermore, this embodiment can also introduce a multi-head attention module, which enhances the fusion of cross-modal features by establishing correlations between features of different modalities, such as calculating the attention weights of fundus hemorrhage lesions and changes in macular thickness in OCT. This design not only optimizes resource allocation but also improves the ability to diagnose complex pathologies.

[0102] The correlation between different modal features is represented by attention weights for different modal features. For feature vectors from different modalities (such as fundus lesion features) OCT structural features The query vector, key vector, and value vector representations are generated respectively using a shared linear projection matrix. , To query the vector learning parameters, key vector , Learn parameters for the key vector and the value vector. , Parameters are learned for the value vectors. To reduce the number of parameters, the projection matrices of the key vectors and value vectors are shared across modalities (e.g., OCT and OCTTA share a set of key vectors and value vectors, and only the query matrix is ​​distinguished by modality).

[0103] In this step, each expert model (such as the fundus photography expert model and the OCT expert model) adopts a narrow and deep network structure (such as a simplified ResNet branch). Parameter sharing and low-rank decomposition reduce the number of parameters in a single expert model. The gated network receives concatenated vectors of multimodal features and computes sparse routing weights through lightweight fully connected layers, activating only the Top-K expert models. This sparse routing weight setting ensures that only a portion of the expert model parameters need to be loaded into memory for each inference, reducing the computational resource requirements for generating diagnostic results.

[0104] The diagnostic assistance method for fusing multimodal data disclosed in this embodiment includes three improvements. First, it employs a hierarchical cross-modal alignment technique. This scheme, by combining an improved ResNet architecture and CBAM modules, achieves more accurate feature extraction and lesion localization. Especially when processing multimodal data, the introduction of spatial and channel attention mechanisms and deformable convolution effectively improves the model's ability to capture key pathological features and enhances its interpretability. Second, it introduces a missing modality compensation mechanism, utilizing Conditional GANs for cross-modal feature generation, which can provide reliable diagnostic support even when certain image modalities are missing. This method not only improves data utilization but also dynamically adjusts the decision threshold by quantifying uncertainty, ensuring the accuracy and reliability of the diagnosis. Third, it features a lightweight inference engine architecture, employing a modality-specific expert model combined with a gating network design, achieving effective resource allocation and improved computational efficiency. This design not only reduces unnecessary computation but also dynamically activates relevant experts based on the characteristics of the input data, improving the ability to diagnose complex pathologies.

[0105] It should be understood that, although attached Figure 1 The steps in the flowchart are shown sequentially according to the arrow lock indicators, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders. Furthermore, [the flowchart is attached]. Figure 1 At least some of the steps in the process may include multiple sub-steps or sub-stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0106] The above-described embodiments of the present invention provide a detailed description of a diagnostic assistance method that integrates multimodal data. Since this method can be implemented using various types of devices, the present invention also discloses a diagnostic assistance device that integrates multimodal data. Figure 3 The schematic diagram of the fusion multimodal data diagnostic auxiliary device shown below is followed by a detailed description of specific embodiments.

[0107] The data acquisition module 501 is used to acquire multimodal data sent by the data acquisition module. The multimodal data includes at least two of the following: 2D fundus color image data, 3D OCT volume data, OCTA blood flow parameters, and wide-area fundus stitched image parameters.

[0108] The feature extraction module 502 is used to input the multimodal data into the trained hierarchical feature extraction module to obtain the multimodal data feature vector;

[0109] The feature completion module 503 is used to input the multimodal data feature vector into the trained feature compensation-fusion module to obtain complete multimodal data fusion features;

[0110] The diagnostic assistance module 504 is used to input the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnosis and treatment data and text diagnostic assistance reports, and send the structured diagnosis and treatment data and text diagnostic assistance reports to the data display module.

[0111] Furthermore, the feature completion module includes the following units:

[0112] The missing modality feature generation unit is used to generate approximate feature vectors of the missing modality by inputting the multimodal data feature vectors into the conditional adversarial network if the types of the multimodal data feature vectors do not meet the condition for complete fusion data.

[0113] The feature concatenation unit is used to concatenate the multimodal data feature vectors and the multimodal data feature vectors by combining feature confidence to obtain complete multimodal data fusion features.

[0114] Furthermore, the step of inputting the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnostic data and text-based diagnostic assistance reports includes:

[0115] The complete multimodal data fusion features are input into the trained dynamic expert hybrid system, and the weights of each modality-specific expert model are determined by the dynamic routing algorithm of the gated network.

[0116] Based on the weights of each modality-specific expert model, relevant modality-specific expert models are activated. After weighted fusion of the preliminary diagnostic results output by several expert models, the results are passed to the fully connected network of the output layer to generate a text-based diagnostic assistance report.

[0117] For details regarding the diagnostic auxiliary device that integrates multimodal data, please refer to the method specifications above; they will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the terminal device in hardware form or independently of it, or stored in the memory of the terminal device in software form, so that the processor can call and execute the corresponding operations of each module.

[0118] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for assisting diagnosis using fused multimodal data.

[0119] The computer-readable storage medium may be an electronic storage device such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), hard disk, or ROM. Optionally, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products, and the program code may be compressed in an appropriate form.

[0120] In one embodiment, the present invention provides a computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the above-described fusion multimodal data diagnostic assistance method.

[0121] The computer device includes a memory, a processor, and one or more computer programs, wherein the one or more computer programs may be stored in the memory and configured to be executed by one or more processors, and the one or more application programs are configured to perform the above-described fusion multimodal data diagnostic assistance method.

[0122] A processor may include one or more processing cores. The processor connects to various parts of the computer device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, and by calling data stored in memory. Optionally, the processor may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also be implemented separately as a communication chip, without being integrated into the processor.

[0123] The memory may include random access memory (RAM) or read-only memory (ROM). The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described above. The data storage area may also store data created by the terminal device during use.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A diagnostic assistance method integrating multimodal data, characterized in that, The fusion of multimodal data for diagnostic assistance is applied to a mobile diagnostic vehicle, which includes at least two data acquisition modules, a data processing module, and a data display module. The data processing module is communicatively connected to both the data acquisition module and the data display module. The data processing module executes the fusion of multimodal data for diagnostic assistance, including the following steps: Acquire multimodal data sent by the data acquisition module, wherein the multimodal data includes at least two of the following: 2D fundus color image data, 3D OCT volume data, OCTA blood flow parameters, and wide-area fundus stitched image parameters; The multimodal data is input into the trained hierarchical feature extraction module to obtain the multimodal data feature vector; The multimodal data feature vectors are input into the trained feature compensation-fusion module to obtain complete multimodal data fusion features; The complete multimodal data fusion features are input into the trained dynamic expert hybrid system to obtain structured diagnosis and treatment data and text-based diagnostic assistance reports, which are then sent to the data display module.

2. The diagnostic assistance method for fusing multimodal data as described in claim 1, characterized in that, The step of inputting the multimodal data into the trained hierarchical feature extraction module to obtain the multimodal data feature vector specifically includes the following steps: Extract data type labels from the multimodal data; Based on the data type labels, the multimodal data of each type is pushed to the corresponding feature extraction module to obtain the feature vectors of each type of multimodal data.

3. The diagnostic assistance method for fusing multimodal data as described in claim 2, characterized in that, The step of pushing multimodal data of each type to the corresponding feature extraction module according to the data type label to obtain feature vectors of each type of multimodal data includes: If the data type label includes a first label, the first modal data is extracted according to the data address indicated by the first label, and the first modal data is input into the ResNet model that combines spatial attention mechanism to obtain the first modal data feature vector; If the data type label includes a second label, the second modal data is extracted according to the data address indicated by the second label, and the second modal data is input into the 3D Swin Transformer model to obtain the feature vector of the second modal data; If the data type label includes a third label, the third modality data is extracted according to the data address indicated by the third label, and the third modality data is input into the 3D U-Net model to obtain the third modality data feature vector; If the data type label includes a fourth label, the fourth modality data is extracted according to the data address indicated by the fourth label, and the fourth modality data is input into the block model to obtain the fourth modality data feature vector.

4. The diagnostic assistance method for fusing multimodal data as described in claim 3, characterized in that, The step of inputting the multimodal data feature vector into the trained feature compensation-fusion module to obtain complete multimodal data fusion features specifically includes the following steps: If the types of multimodal data feature vectors do not meet the conditions for complete fusion data, the multimodal data feature vectors are input into the conditional adversarial network to generate approximate feature vectors for the missing modalities. The feature vectors of the multimodal data are concatenated by combining the feature confidence scores to obtain the complete multimodal data fusion features.

5. The diagnostic assistance method for fusing multimodal data as described in claim 4, characterized in that, The process of inputting the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnostic data and text-based diagnostic assistance reports includes: The complete multimodal data fusion features are input into the trained dynamic expert hybrid system, and the weights of each modality-specific expert model are determined by the dynamic routing algorithm of the gated network. Based on the weights of each modality-specific expert model, relevant modality-specific expert models are activated. After weighted fusion of the preliminary diagnostic results output by several expert models, the results are passed to the fully connected network of the output layer to generate a text-based diagnostic assistance report.

6. A diagnostic auxiliary device integrating multimodal data, characterized in that, The fusion multimodal data diagnostic auxiliary device is applied to a mobile diagnostic vehicle, which includes at least two data acquisition modules, a data processing module, and a data display module. The data processing module is communicatively connected to both the data acquisition module and the data display module. The data processing module executes the fusion multimodal data diagnostic auxiliary device and includes the following modules: The data acquisition module is used to acquire multimodal data sent by the data acquisition module. The multimodal data includes at least two of the following: 2D fundus color image data, 3D OCT volume data, OCTA blood flow parameters, and wide-area fundus stitched image parameters. The feature extraction module is used to input the multimodal data into the trained hierarchical feature extraction module to obtain multimodal data feature vectors; The feature completion module is used to input the multimodal data feature vector into the trained feature compensation-fusion module to obtain complete multimodal data fusion features; The diagnostic assistance module is used to input the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnosis and treatment data and text diagnostic assistance reports, and send the structured diagnosis and treatment data and text diagnostic assistance reports to the data display module.

7. The fusion multimodal data diagnostic auxiliary device as described in claim 6, characterized in that, The feature completion module includes the following units: The missing modality feature generation unit is used to generate an approximate feature vector of the missing modality based on the input of the multimodal data feature vector to the conditional adversarial network if the types of the multimodal data feature vector do not meet the condition of complete fused data. The feature concatenation unit is used to concatenate the multimodal data feature vectors and the multimodal data feature vectors by combining feature confidence to obtain complete multimodal data fusion features.

8. The fusion multimodal data diagnostic auxiliary device as described in claim 7, characterized in that, The process of inputting the complete multimodal data fusion features into the trained dynamic expert hybrid system to obtain structured diagnostic data and text-based diagnostic assistance reports includes: The complete multimodal data fusion features are input into the trained dynamic expert hybrid system, and the weights of each modality-specific expert model are determined by the dynamic routing algorithm of the gated network. Based on the weights of each modality-specific expert model, relevant modality-specific expert models are activated. After weighted fusion of the preliminary diagnostic results output by several expert models, the results are passed to the fully connected network of the output layer to generate a text-based diagnostic assistance report.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of any one of the multimodal data diagnostic assistance methods in claims 1-5.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it performs any one of the multimodal data diagnostic assistance methods according to claims 1-5.

Images