Multi-modal medical image intelligent analysis system and method based on large language model

Abstract

The invention provides a multi-modal medical image intelligent analysis system and method based on a large language model, and the system comprises a key frame extraction module which carries out the key frame extraction through a multi-frame quality evaluation algorithm; the coronary artery blood vessel extraction module is used for realizing accurate extraction of a coronary artery tree structure through multi-scale blood vessel enhancement; the pathological recognition module adopts an improved DeepLabV3 + algorithm to construct a cardiac blood vessel blockage region segmentation model; the multi-modal diagnosis module adopts LLaVA-Med multi-modal diagnosis, integrates vision and language understanding ability, and combines DeepSeek to generate a standardized diagnosis report; the model optimization module is used for implementing model parameter iterative updating based on a near-end strategy optimization algorithm; and the visual interaction module is used for visually displaying the lesion position and the spatial relationship thereof, so that a doctor can conveniently carry out preoperative path planning and intraoperative navigation judgment. On the basis of a large language model and a multi-mode technology, full-process support from image uploading to diagnosis decision making is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image intelligent analysis, and in particular to a multimodal medical image intelligent analysis system and method based on a large language model. Background Art

[0002] With the continued rise in the incidence of cardiovascular and cerebrovascular diseases, the importance of medical imaging in early screening and auxiliary diagnosis of diseases has become increasingly prominent. Especially in the diagnosis of cardiovascular diseases, angiography (such as DSA) is a clinical standard. Because of its intuitive presentation of vascular structure and degree of stenosis, it is widely used in clinical interventional treatment decisions. However, the image analysis workflow of traditional cardiac angiography is highly dependent on the subjective experience of doctors, the operation process is complicated, and there is a certain risk of misdiagnosis and missed diagnosis. Especially in primary medical institutions, due to insufficient professional capabilities and high equipment operation thresholds, the problem of "affordable but poorly used" medical imaging equipment is relatively common, which seriously restricts the release of the clinical value of high-end imaging equipment.

[0003] In recent years, artificial intelligence technology has rapidly developed in the field of medical imaging, with key technologies such as deep learning-based image segmentation, object detection, and structured report generation gradually maturing. Some commercial products have begun to be used in the field of image-assisted diagnosis in tertiary hospitals, but these systems generally suffer from three limitations: first, they rely on large-scale manually annotated data, resulting in limited generalization capabilities; second, they lack natural language interaction and clinical reasoning capabilities, making it difficult to meet the personalized diagnosis and treatment needs of physicians; and third, their high system complexity makes it difficult for primary care physicians to use them, limiting their adoption. Furthermore, most current systems use single-modal input, making it difficult to integrate imaging information with multi-source data such as clinical descriptions and interview information, resulting in limited diagnostic assistance capabilities. Summary of the Invention

[0004] In response to the defects in the existing technology, the purpose of the present invention is to provide a multimodal medical image intelligent analysis system and method based on a large language model. Based on the large language model and multimodal technology, it supports the precise segmentation and intelligent analysis of medical images, and provides full-process support from image uploading to diagnostic decision-making, thereby improving the utilization efficiency and diagnostic level of high-end medical equipment in primary medical institutions and providing technical support for promoting the development of digital medicine.

[0005] In order to solve the above problems, the technical solution of the present invention is:

[0006] A multimodal medical image intelligent analysis system based on a large language model, comprising:

[0007] Key frame extraction module, which extracts key frames through multi-frame quality assessment algorithm;

[0008] The coronary artery extraction module achieves accurate extraction of the coronary artery tree structure through multi-scale vessel enhancement;

[0009] The pathology recognition module uses the improved DeepLabV3+ algorithm to build a segmentation model for cardiovascular blockage areas;

[0010] The multimodal diagnosis module uses LLaVA-Med multimodal diagnosis, integrates visual and language understanding capabilities, and combines DeepSeek to generate standardized diagnostic reports;

[0011] Model optimization module, which implements iterative updates of model parameters based on the proximal strategy optimization algorithm;

[0012] The visual interaction module is used to intuitively display the location of the lesion and its spatial relationship, facilitating doctors' preoperative path planning and intraoperative navigation judgment.

[0013] Preferably, the coronary artery extraction module uses a Frangi filter based on the Hessian matrix to perform multi-scale vessel enhancement, and by analyzing the eigenvalue combination of the Hessian matrix, distinguishes between vascular and non-vascular areas, eliminates pseudo-vascular noise, and achieves accurate extraction of the coronary artery tree structure.

[0014] Preferably, the improved DeepLabV3+ algorithm segmentation model includes:

[0015] The improved atrous spatial pyramid pooling module captures multi-scale information of vascular structures by introducing multiple convolution kernels with different dilation rates and performing convolution operations at different scales.

[0016] Hierarchical feature fusion module, which dynamically adjusts the fusion weights of deep and shallow features through the SENet module to adaptively recalibrate the relationship between channels, strengthen the model's focus on important features, and suppress the interference of irrelevant features;

[0017] The edge-aware constraint module performs auxiliary training by calculating the difference between the predicted edge and the actual edge, thereby introducing edge-aware constraints into the loss function and enhancing the model's positioning ability at the boundary.

[0018] A multimodal medical image intelligent analysis method based on a large language model includes the following steps:

[0019] After the user uploads the digital subtraction angiography sequence, key frames are extracted using a multi-frame quality assessment algorithm;

[0020] After acquiring the key frames, multi-scale vascular enhancement is performed to achieve accurate extraction of the coronary artery tree structure;

[0021] The improved DeepLabV3+ network model is used to automatically identify blood vessel obstruction areas;

[0022] Generate structured diagnostic reports using a multimodal diagnostic model combined with DeepSeek;

[0023] The model parameters are iteratively optimized through the proximal strategy optimization algorithm.

[0024] Preferably, after the user uploads the digital subtraction angiography sequence, the step of extracting key frames through a multi-frame quality assessment algorithm specifically includes: after the user uploads the digital subtraction angiography sequence, the system first extracts key frames through a multi-frame quality assessment algorithm. The key frame extraction adopts a weighted scoring model based on a spatial gradient operator, combined with non-local mean filtering and restricted contrast histogram equalization technology, to screen out vascular contrast peak frames from the dynamic sequence and eliminate blurring interference caused by respiratory movement or heart beat.

[0025] Preferably, after acquiring the key frame, performing multi-scale vascular enhancement to achieve accurate extraction of the coronary artery tree structure specifically includes: after acquiring the key frame, using a Frangi filter based on the Hessian matrix to perform multi-scale vascular enhancement, by analyzing the eigenvalue combination of the Hessian matrix, distinguishing between vascular and non-vascular areas, eliminating pseudo-vascular noise, and achieving accurate extraction of the coronary artery tree structure, thereby achieving noise suppression and structure enhancement.

[0026] Prior to this, the step of automatically identifying vascular blockage areas using the improved DeepLabV3+ network model specifically includes: the vascular segmentation results enter the improved DeepLabV3+ pathology recognition network model, which captures multi-scale vascular features through a void space pyramid pooling module with an adaptive expansion rate, and introduces a channel attention mechanism to optimize the feature fusion strategy, combining dynamic weighted cross entropy loss and hard sample mining technology to improve the positioning accuracy of stenosis areas.

[0027] Preferably, the steps of generating a structured diagnostic report through a multimodal diagnostic model in combination with DeepSeek specifically include: the segmentation results are further input into a diagnostic report generation module, and the LLaVA-Med multimodal large language model performs intelligent diagnosis. The module integrates DICOM metadata projection angle parameters and lesion coordinate information through a learnable prompt word project, constructs an "image feature-text description" cross-modal alignment mechanism, generates a structured diagnostic report that complies with the ACC / AHA guidelines, and calls the DeepSeek API engine to realize real-time conversion of Chinese and English reports.

[0028] Preferably, the steps of iteratively optimizing model parameters through the proximal strategy optimization algorithm specifically include: users raise clinical questions in natural language through the interactive AI question-and-answer module, the system uses a semantic parsing engine to extract key terms, combines image features with the medical knowledge base to generate professional advice, and feeds back the question-and-answer results to the user interface, the manual correction module allows doctors to manually adjust the mask boundaries or report content generated by AI, stores the correction data in the correction data storage, and uses the proximal strategy optimization algorithm to implement iterative updates of model parameters to achieve continuous optimization of model performance and form a "manual feedback-model iteration" closed loop.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This invention is based on a large language model and multimodal technology, deeply integrating the DeepLabV3+ algorithm, LLaVA-Med and DeepSeek models, supporting precise segmentation and intelligent analysis of medical images, and providing full-process support from image uploading to diagnostic decision-making.

[0031] 2. The present invention introduces a manual correction feedback mechanism and a reinforcement learning framework, which can feed back the doctor's modified data on masks and reports to the model in real time for self-optimization, thereby achieving continuous iteration and improvement of system performance. Compared with traditional static model diagnosis systems, the present invention has obvious advantages in accuracy, interpretability and the ability to adapt to complex clinical scenarios, and can significantly improve diagnostic consistency and efficiency.

[0032] 3. The present invention integrates three-dimensional visualization and interactive functions, which can automatically convert two-dimensional angiographic images into high-fidelity cardiovascular models, and supports multi-view interactive observation and lesion highlighting. Compared with existing systems that only provide static image output, the present invention can more intuitively display the location of lesions and their spatial relationships, facilitating doctors' preoperative path planning and intraoperative navigation judgment, significantly enhancing their clinical practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0034] Figure 1 Schematic diagram of the architecture of the multimodal medical image intelligent analysis system based on a large language model of the present invention;

[0035] Figure 2 Extract schematic diagrams for keyframes;

[0036] Figure 3 Schematic diagram of blood vessel extraction;

[0037] Figure 4 Schematic diagram for pathological identification;

[0038] Figure 5 Schematic diagram of LLaVA-Med's interaction with users;

[0039] Figure 6 This is a flowchart of the multimodal medical image intelligent analysis method based on a large language model of the present invention;

[0040] Figure 7 This is a detailed flow chart of the multimodal medical image intelligent analysis method based on a large language model of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0042] Specifically, the present invention provides a hierarchical architecture diagram of a multimodal medical image intelligent analysis system based on a large language model, such as Figure 1 As shown, the system includes a key frame extraction module 1, a coronary artery extraction module 2, a pathology recognition module 3, a multimodal diagnosis module 4, a model optimization module 5 and a visualization interaction module 6.

[0043] 1. Keyframe extraction module

[0044] Keyframe extraction is performed using a multi-frame quality assessment algorithm. Because cardiac DSA sequences exhibit dynamic blood perfusion (average velocity 15-25 cm / s), vascular edge clarity exhibits periodic fluctuations due to the dual influence of the cardiac cycle (systole / diastole) and respiratory motion artifacts. By extracting peak vascular contrast frames and discarding invalid frames in the DSA sequence, errors in vascular boundary positioning caused by motion blur can be effectively avoided, providing a reliable imaging benchmark for subsequent stenosis location and stenosis measurement.

[0045] Specifically, let the original video sequence be: First, the video preprocessing step is performed.

[0046] The first step is to calculate the t Perform non-local means filtering:

[0047]

[0048] Where Ω is the search window, is the normalization constant, and h is the filtering parameter.

[0049] The second step is to use limited contrast histogram equalization (CLAHE):

[0050]

[0051] Among them, N tiles represents the number of sub-region divisions, and φ is the contrast limit threshold.

[0052] Next, we construct a vascular clarity quantification model and characterize the local vascular structure based on the Hessian matrix:

[0053]

[0054] in represents the second-order derivative at scale σ.

[0055] Frangi filter response function:

[0056]

[0057] in, is the anisotropy ratio, is the structural significance, β,c are the adjustment parameters. Multi-scale vascular significance:

[0058]

[0059] where ∑=[σ min ,σ max ] is the preset scale range.

[0060] The second parameter is the edge sharpness metric, which is calculated based on the modified gradient magnitude:

[0061]

[0062] in, is the Sobel operator.

[0063] The third parameter is the information entropy metric, which defines a dynamic ROI based on the vascular enhancement image:

[0064]

[0065] in is an adaptive mask based on vascular response, V th is the significance threshold.

[0066] By integrating the above three parameters, the feature fusion equation is established:

[0067]

[0068] Where μ(·) and σ(·) are the mean and standard deviation of each feature, and the weight coefficient is determined by the entropy weight method:

[0069]

[0070] Finally, build the regularized scoring function:

[0071]

[0072] Where λ is the time series smoothing factor and θ is the attenuation coefficient.

[0073] Score each frame of the image and extract the frame with the highest score at each angle as the key frame, such as Figure 2 shown.

[0074] 2. Coronary artery extraction module

[0075] Specifically, after acquiring the key frames, the original images often introduce pseudo-vascular structures due to factors such as tissue overlap, noise interference, and uneven contrast, which affects the accurate extraction and quantitative analysis of the coronary artery vascular tree. In order to suppress such interference and enhance vascular features, the present invention uses a Frangi filter based on the Hessian matrix to perform multi-scale vascular enhancement. The theoretical basis of this method is that the vascular structure appears as a linear or tubular geometric shape in a local area, and the eigenvalues ​​of its Hessian matrix can quantify the morphological characteristics. By analyzing the eigenvalue combination of the Hessian matrix, it is possible to distinguish between vascular and non-vascular areas, thereby achieving noise suppression and structure enhancement, such as Figure 3 shown.

[0076] The core idea of ​​the Frangi filter based on the Hessian matrix is ​​to quantify the local geometric structure through the second-order differential characteristics and suppress non-tubular noise. σ The Gaussian kernel G is defined as the second-order partial derivative convolution of image I at scale σ σ :

[0077]

[0078] By eigenvalue decomposition, we get λ1 and λ2 (satisfying |λ1|≤|λ2|). The vascular region is characterized by λ1≈0 and |λ1|<<|λ2|. The Frangi filter enhances the tubular structure by the following metric function:

[0079]

[0080] in, represents the aspect ratio constraint, which makes the blood vessels approach 0 and the noise approach 1; represents the structural strength, β controls the tubular sensitivity, and c suppresses background noise.

[0081] 3. Pathology Recognition Module

[0082] In order to accurately locate the blocked area, this paper constructs a cardiovascular blocked area segmentation model based on the DeepLabV3+ architecture, and performs multi-dimensional optimization based on the characteristics of medical imaging.

[0083] Based on the classic DeepLabV3+ encoder-decoder architecture, this paper makes the following improvements to the characteristics of angiographic images:

[0084] Improved Atrous Spatial Pyramid Pooling (ASPP) module

[0085] Based on the standard Atrous Spatial Pyramid Pooling (ASPP) module, a parallel convolution structure with adaptive dilation rate is designed to better capture the characteristics of blood vessels at different scales. The improved ASPP module enhances the network's ability to recognize subtle structures such as blood vessels by introducing multiple dilation rates and performing convolution operations at different scales. Specifically, given an input feature map x∈R H×W×C , the improved ASPP module can be expressed as:

[0086]

[0087] Where R = {6, 12, 18} represents the expansion rate set, Represents feature concatenation operation, DConv 3×3,r (x) represents a 3×3 dilated convolution operation with dilation rate r, and GlobalPool represents a global average pooling layer. By using convolution kernels with different dilation rates, the ASPP module processes input feature maps at multiple scales, capturing multi-scale information about vascular structures. The global average pooling layer further enhances the model's ability to extract global contextual information, thereby improving its expressiveness and robustness.

[0088] Hierarchical feature fusion module

[0089] In the decoding phase, to improve the ability to effectively fuse features at different levels, a channel attention mechanism is introduced. The SENet (Squeeze-and-Excitation Network) module dynamically adjusts the fusion weights of deep and shallow features to adaptively recalibrate the relationship between channels, strengthen the model's focus on important features, and suppress the interference of irrelevant features. The specific formula is as follows:

[0090] α c =σ(W2δ(W1GAP(F c )))

[0091] Where GAP represents global average pooling, δ is the ReLU activation function, and σ is the Sigmoid function. is the parameter of the fully connected layer. The introduction of this channel attention mechanism effectively improves the performance of the model in the task of vascular occlusion localization.

[0092] Edge-aware constraint module

[0093] To further improve the accuracy of the segmentation network in boundary areas, we added an auxiliary boundary detection branch at the end of the network. This branch performs auxiliary training by calculating the difference between predicted edges and true edges, thereby introducing edge-aware constraints into the loss function and strengthening the model's ability to locate boundary locations. The edge-aware loss function is defined as follows:

[0094]

[0095] in, represents the predicted edge map of the i-th sample, is the corresponding ground-truth edge map. This design allows the network to explicitly perform edge detection during training to enhance its performance in target edge regions, thereby achieving more accurate boundary prediction in the primary segmentation task. Experimental results show that this design improves the model's Dice coefficient by 2.3%.

[0096] The imbalance between positive and negative samples in medical image segmentation is particularly pronounced in coronary artery analysis. According to experimental data, the average proportion of vascular occlusion in a 512×512 pixel DSA image is only 12.7±3.5%. To address this imbalance, we employed the following three optimization measures:

[0097] Dynamically weighted cross entropy loss

[0098] The traditional cross-entropy loss function can easily cause the model to converge to a local optimum when processing cardiac angiography images. To address this issue, we use a dynamic weighted cross-entropy loss. This method aims to dynamically adjust the loss weights based on the proportion of pixels in each class in the current batch to better cope with sample imbalance. Specifically, the weight calculation formula of the dynamic weighted cross-entropy loss function is as follows:

[0099]

[0100] Among them, N c Indicates the number of pixels from category c, N totalis the total number of pixels. This formula dynamically adjusts the loss weight based on the category distribution in the current batch by inversely weighting the number of pixels in each category, allowing small sample categories to receive more attention and learning opportunities. Compared with the traditional fixed weight strategy, the dynamic weighted cross entropy loss can flexibly respond to changes in categories across batches, avoiding overfitting to mainstream categories in the presence of sample imbalance. Through this optimization strategy, the model can effectively improve its recall ability for small objects. Compared with the fixed weight strategy, this method increases the recall rate of lesion areas by 7.6%.

[0101] Hard sample mining

[0102] To further improve the model's ability to distinguish between difficult and easy samples, especially when dealing with samples with low prediction confidence, we introduced a hard sample mining strategy. In each training cycle, we screen out pixels with prediction confidence between [0.3, 0.7] to form a hard sample set. Applying double loss weight to these hard samples enables the model to pay more attention to these samples with unclear boundaries or difficult to classify, thereby improving its learning ability on difficult samples. The specific loss function is defined as follows:

[0103]

[0104] Among them, γ=2 is the adjustment factor, p i This loss function applies a higher training weight to hard samples, making the model pay more attention to samples with low prediction confidence and difficult to classify during training, thereby accelerating the convergence of the model for samples with higher difficulty.

[0105] Hybrid Data Augmentation

[0106] To address the limitations of cardiac angiography (CA) images, such as low resolution and limited data, we designed an innovative data augmentation strategy. This strategy combines multiple image processing techniques to increase image diversity and enhance model robustness. The formula is as follows:

[0107]

[0108] in, Elastic deformation represents elastic deformation. This deformation method can simulate the diversity of tissue deformation without destroying the heart structure, thereby increasing the changes in the image geometric space.

[0109] Contrast Limited Histogram Equalization (CLAHE) is a technique that enhances the clarity of tiny blood vessel structures and details in low-contrast areas by locally adjusting the contrast of the image.

[0110] Represents a random rotation operation with a rotation angle range of θ∈[-15°,15°] to enhance the performance of the model under different viewing angles.

[0111] In order to synergistically optimize the accuracy of blood vessel segmentation and the ability to locate blocked boundary areas, the present invention constructs a composite loss function system, the overall expression of which is:

[0112]

[0113] Among them, λ1=1.0, λ2=0.5, λ3=0.2 are task weight coefficients, and the optimal combination is determined through grid search.

[0114] in, It is a dynamic weighted cross entropy loss. Combined with the above introduction, the specific expression is:

[0115]

[0116] Where w c Dynamic calculation based on sliding window statistics:

[0117]

[0118] where μ t is the global average category ratio in the current training stage, μ c is the average frequency of occurrence of category c in the most recent B=16 batches.

[0119] For the expression given above:

[0120]

[0121] An auxiliary classification head is introduced in the third stage of the encoder to enhance gradient propagation through feature consistency constraints:

[0122]

[0123] The KL divergence measures the difference in prediction distribution between the main branch and the auxiliary branch. Experiments show that this loss accelerates the model convergence speed by 23%.

[0124] After completing the above optimizations, the experiment used a Frangi filter-processed coronary angiography dataset consisting of 1,200 512×512 pixel DSA images, split into training, validation, and test sets at an 8:1:1 ratio. The data was annotated with two labels: background and vascular occlusion. The overall metric was used to evaluate segmentation accuracy and boundary localization.

[0125] Among them, Overall Acc represents the prediction accuracy of the model on all pixels, and the calculation standard is the ratio of the number of pixels predicted correctly by the model to the total number of pixels:

[0126]

[0127] Where C is the number of categories, TP i is the true number of examples of category i (i.e., the number of correctly predicted pixels), P i is the total number of pixels of class i (including true positives, false positives, true negatives, and false negatives).

[0128] Visualization results such as Figure 4 As shown in Figure 3, the improved model can accurately segment slender vascular branches and clearly define the boundaries of the occluded area.

[0129] 4. Multimodal diagnosis module

[0130] The multimodal diagnosis module uses the LLaVA-Med (Large Language and Vision Assistant for Biomedicine) model, a vision-language multimodal architecture designed to integrate vision and language understanding for biomedical image processing. The LLaVA-Med model is based on the LLaVA framework, which combines a pre-trained visual encoder (CLIP) with a language model (Vicuna) and has been specifically adapted to meet the needs of biomedical image interpretation and conversational AI.

[0131] LLaVA-Med builds on the LLaVA model and further optimizes it for the medical field. This approach is widely applicable to various image-to-text architectures. In the first stage, the model's task is to predict descriptive text from the original image. During this stage, the weights of the image encoder and language model remain fixed and do not participate in training. Only the encoder's subsequent projection matrix is ​​updated. This is intended to align visual concepts with word embeddings. In the second stage, the language model is used for training, guided by multi-turn dialogues generated by GPT-4. LLaVA-Med has achieved significant results on several visual question-answering datasets.

[0132] Specifically, for DSA sequence images from different angles, the program will call LLaVA-Med multiple times to process each image one by one. Subsequently, the DeepSeek API is called to integrate and summarize the analysis results of each image to generate a complete reading report. For subsequent questions raised by users, the program translates them into English through the DeepSeek API and then calls LLaVA-Med for professional answers to meet the dual needs of specialization and localization. By integrating visual and language understanding capabilities to generate standardized diagnostic reports, LLaVA-Med interacts with users in the following ways: Figure 5 shown.

[0133] 5. Model optimization module

[0134] To improve the robustness and accuracy of medical image segmentation and diagnosis tasks, this paper proposes a Proximal Policy Optimization (PPO) algorithm to implement iterative updates of model parameters. To ensure the accuracy and reliability of the diagnosis results, the platform outputs answers that allow users to perform secondary annotation. This framework fully utilizes the lesion area annotations and diagnosis texts that have been manually corrected by experts as standard data, and achieves collaborative optimization of the segmentation network and diagnosis model by constructing a multimodal reward signal.

[0135] In the design of the reward function, a multimodal feedback signal is constructed from two levels: spatial and semantic. First, for the segmentation task of the lesion area, the segmentation accuracy is measured by calculating the Dice coefficient between the lesion mask output by the DeepLabV3+ module and the doctor's correction area, thereby obtaining a spatial consistency reward. Secondly, in order to evaluate the degree of semantic matching between the generated diagnostic description and the expert text, the diagnostic text generated by the LLaVA-Med module is used, and the cosine similarity is calculated with the help of BERT embedding to obtain a semantic matching reward. The above two rewards are weighted and fused according to certain weights to construct the total reward function of the PPO algorithm, whose mathematical expression is:

[0136] R total =αR Dice +βR BERT

[0137] Among them, α and β are hyperparameters, which are optimized by the grid search method to balance the influence of different signals in multimodal feedback.

[0138] To further improve the performance of lesion recognition and diagnosis, a collaborative optimization mechanism based on dual-module interaction was proposed, which achieved iterative enhancement of the model in forward information propagation and reverse strategy optimization.

[0139] During the forward propagation phase, the DeepLabV3+ module first extracts the region of interest (ROI) of the lesion area and passes it as visual input to the LLaVA-Med module, thereby guiding the model to focus on pathological features. Simultaneously, the diagnostic keywords generated by the LLaVA-Med module are passed back to the DeepLabV3+ module, which uses an attention gating mechanism to enhance the segmentation saliency of the corresponding region.

[0140] The reverse optimization stage uses the PPO algorithm to simultaneously update the parameters of the two modules based on the multimodal reward signal. Specifically, the policy gradient method is used to calculate the parameter update direction, and its gradient formula is:

[0141]

[0142] Among them, the advantage function A(s,a) is calculated based on manual feedback and baseline model prediction, ensuring end-to-end optimization effect and improvement of the overall system performance.

[0143] 6. Visual interaction module

[0144] The visualization interaction module is used to intuitively display the location of lesions and their spatial relationships, facilitating preoperative path planning and intraoperative navigation judgment for doctors. The implementation of the three-dimensional interactive interface relies on the WebGL framework and modular design ideas, and realizes accurate rendering and interactive operations of vascular models through dynamic loading and visualization technology. The core of the system uses the Three.js engine to build a three-dimensional scene, and uses GLTFLoader and DRACOLoader to achieve efficient loading and decoding of vascular models. In view of the large amount of medical imaging data, KTX2 texture compression and Meshopt geometry compression technology are introduced to reduce model transmission and rendering overhead. In the scene initialization stage, a perspective camera (PerspectiveCamera) and an orbit controller (OrbitControls) are constructed to support users to observe vascular morphology from multiple angles through translation, rotation and scaling, and generate physically based rendering environment maps through PMREMGenerator to simulate real lighting conditions.

[0145] Furthermore, the present invention also provides a multimodal medical image intelligent analysis method based on a large language model, such as Figure 6 and Figure 7 As shown, the method includes the following steps:

[0146] S1: After the user uploads the digital subtraction angiography sequence, key frames are extracted using a multi-frame quality assessment algorithm;

[0147] Specifically, after the user uploads the digital subtraction angiography (DSA) sequence, the system first extracts key frames through a multi-frame quality assessment algorithm. The key frame extraction adopts a weighted scoring model based on the spatial gradient operator, combined with non-local mean filtering and restricted contrast histogram equalization technology to screen out vascular contrast peak frames from the dynamic sequence, eliminate blurring interference caused by respiratory movement or heart beats, and provide a high-quality image benchmark for subsequent analysis.

[0148] S2: After acquiring the keyframe, multi-scale vascular enhancement is performed to achieve accurate extraction of the coronary artery tree structure;

[0149] Specifically, after acquiring the key frames, the Frangi filter based on the Hessian matrix is ​​used for multi-scale vascular enhancement. By analyzing the eigenvalue combination of the Hessian matrix, it is possible to distinguish between vascular and non-vascular areas, eliminate pseudo-vascular noise, and achieve accurate extraction of the coronary artery tree structure, thereby achieving noise suppression and structural enhancement.

[0150] S3: Automatically identify vascular occlusion areas using the improved DeepLabV3+ network model;

[0151] Specifically, the vascular segmentation results are then fed into the improved DeepLabV3+ pathology recognition network model, which captures multi-scale vascular features through an adaptive dilation rate atrous spatial pyramid pooling (ASPP) module, introduces a channel attention mechanism (SENet) to optimize the feature fusion strategy, and combines dynamic weighted cross entropy loss with hard sample mining technology to significantly improve the positioning accuracy of narrow areas.

[0152] S4: Generate a structured diagnostic report using a multimodal diagnostic model combined with DeepSeek;

[0153] Specifically, the segmentation results are further input into the diagnostic report generation module, and the LLaVA-Med multimodal large language model performs intelligent diagnosis. This module integrates DICOM metadata projection angle parameters and lesion coordinate information through learnable prompt word engineering, constructs an "image feature-text description" cross-modal alignment mechanism, generates a structured diagnostic report that complies with the ACC / AHA guidelines, and calls the DeepSeekAPI engine to achieve real-time conversion of Chinese and English reports.

[0154] S5: Iteratively optimize model parameters through the proximal strategy optimization algorithm.

[0155] Specifically, users can ask clinical questions in natural language through the interactive AI question-and-answer module (such as "Is a stent implant necessary?"). The system uses a semantic parsing engine to extract key terms, combines imaging features with the medical knowledge base to generate professional advice, and then feeds back the Q&A results to the user interface. To ensure diagnostic reliability, the manual correction module allows doctors to manually adjust the mask boundaries or report content generated by the AI. The correction data is stored in the correction data store and input into the model iteration and optimization module. This module uses the proximal policy optimization (PPO) algorithm to iteratively update the model parameters, achieving continuous optimization of model performance and forming a "manual feedback-model iteration" closed loop.

[0156] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and that those skilled in the art may make various changes or modifications within the scope of the claims, without affecting the essence of the present invention. Unless there is a conflict, the embodiments of this application and the features in the embodiments may be combined with each other in any manner.

Claims

1. A multimodal medical image intelligent analysis system based on a large language model, characterized by: The system comprises: Key frame extraction module, which extracts key frames through multi-frame quality assessment algorithm; The coronary artery extraction module achieves accurate extraction of the coronary artery tree structure through multi-scale vessel enhancement; The pathology recognition module uses the improved DeepLabV3+ algorithm to build a segmentation model for cardiovascular blockage areas; The multimodal diagnosis module uses LLaVA-Med multimodal diagnosis, integrates visual and language understanding capabilities, and combines DeepSeek to generate standardized diagnostic reports; Model optimization module, which implements iterative updates of model parameters based on the proximal strategy optimization algorithm; The visual interaction module is used to intuitively display the location of the lesion and its spatial relationship, facilitating doctors' preoperative path planning and intraoperative navigation judgment.

2. The multimodal medical image intelligent analysis system based on a large language model according to claim 1 is characterized in that: The coronary artery extraction module uses a Frangi filter based on the Hessian matrix to perform multi-scale vessel enhancement. By analyzing the eigenvalue combination of the Hessian matrix, it distinguishes between vascular and non-vascular areas, eliminates pseudo-vascular noise, and achieves accurate extraction of the coronary artery tree structure.

3. The multimodal medical image intelligent analysis system based on a large language model according to claim 1, characterized in that: The improved DeepLabV3+ algorithm segmentation model includes: The improved atrous spatial pyramid pooling module captures multi-scale information of vascular structures by introducing multiple convolution kernels with different dilation rates and performing convolution operations at different scales. Hierarchical feature fusion module, which dynamically adjusts the fusion weights of deep and shallow features through the SENet module to adaptively recalibrate the relationship between channels, strengthen the model's focus on important features, and suppress the interference of irrelevant features; The edge-aware constraint module performs auxiliary training by calculating the difference between the predicted edge and the actual edge, thereby introducing edge-aware constraints into the loss function and enhancing the model's positioning ability at the boundary.

4. A multimodal medical image intelligent analysis method based on a large language model, characterized in that: The method comprises the following steps: After the user uploads the digital subtraction angiography sequence, key frames are extracted using a multi-frame quality assessment algorithm; After acquiring the key frames, multi-scale vascular enhancement is performed to achieve accurate extraction of the coronary artery tree structure; The improved DeepLabV3+ network model is used to automatically identify blood vessel obstruction areas; Generate structured diagnostic reports through multimodal diagnostic models and combined with DeepSeek; The model parameters are iteratively optimized through the proximal strategy optimization algorithm.

5. The multimodal medical image intelligent analysis method based on a large language model according to claim 4 is characterized in that: After the user uploads the digital subtraction angiography sequence, the step of extracting key frames using a multi-frame quality assessment algorithm specifically includes: after the user uploads the digital subtraction angiography sequence, the system first extracts key frames using a multi-frame quality assessment algorithm. The key frame extraction adopts a weighted scoring model based on a spatial gradient operator, combined with non-local mean filtering and restricted contrast histogram equalization technology, to screen out vascular contrast peak frames from the dynamic sequence and eliminate blurring interference caused by respiratory movement or heart beats.

6. The multimodal medical image intelligent analysis method based on a large language model according to claim 4 is characterized in that: The step of performing multi-scale vessel enhancement after acquiring the key frame to achieve accurate extraction of the coronary artery tree structure specifically includes: after acquiring the key frame, using a Frangi filter based on the Hessian matrix to perform multi-scale vessel enhancement, distinguishing between vascular and non-vascular areas by analyzing the eigenvalue combination of the Hessian matrix, eliminating pseudo-vascular noise, and achieving accurate extraction of the coronary artery tree structure, thereby achieving noise suppression and structure enhancement.

7. The multimodal medical image intelligent analysis method based on a large language model according to claim 4, characterized in that: The steps of automatically identifying vascular obstruction areas using the improved DeepLabV3+ network model specifically include: the vascular segmentation results enter the improved DeepLabV3+ pathology recognition network model, which captures multi-scale vascular features through a void space pyramid pooling module with an adaptive expansion rate, introduces a channel attention mechanism to optimize the feature fusion strategy, and combines dynamic weighted cross entropy loss with hard sample mining technology to improve the positioning accuracy of stenosis areas.

8. The multimodal medical image intelligent analysis method based on a large language model according to claim 4, characterized in that: The steps of generating a structured diagnostic report through a multimodal diagnostic model in combination with DeepSeek specifically include: the segmentation results are further input into a diagnostic report generation module, and the LLaVA-Med multimodal large language model performs intelligent diagnosis. This module integrates DICOM metadata projection angle parameters and lesion coordinate information through a learnable prompt word project, constructs an "image feature-text description" cross-modal alignment mechanism, generates a structured diagnostic report that complies with the ACC / AHA guidelines, and calls the DeepSeek API engine to achieve real-time conversion of Chinese and English reports.

9. The multimodal medical image intelligent analysis method based on a large language model according to claim 4, characterized in that: The steps of iteratively optimizing model parameters through the proximal strategy optimization algorithm specifically include: users raise clinical questions in natural language through the interactive AI question-and-answer module, the system uses a semantic parsing engine to extract key terms, combines image features with the medical knowledge base to generate professional suggestions, and feeds back the question-and-answer results to the user interface, the manual correction module allows doctors to manually adjust the mask boundaries or report content generated by AI, stores the correction data in the correction data, and uses the proximal strategy optimization algorithm to implement iterative updates of model parameters to achieve continuous optimization of model performance, forming a "manual feedback-model iteration" closed loop.

Images