Intravascular ultrasonic atherosclerotic plaque automatic identification method based on visual language model
By using a hierarchical Transformer plaque segmentation and lesion indication injection module based on a visual language model, the problem of accurate automatic identification and highly clinically relevant description of lesions and plaques in intravascular ultrasound images was solved, enabling fine-grained diagnosis and report generation in multi-plaque scenarios.
Patent Information
- Application Number
- CN202511694694.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-18
- Publication Date
- 2025-12-30
AI Technical Summary
Existing technologies struggle to achieve accurate automatic identification and fine-grained localization of lesions and plaques in intravascular ultrasound images, and the generated clinical reports lack interpretability, failing to meet the need for independent assessment of multiple plaques in a single IVUS image.
A visual language model-based approach is employed, including a hierarchical Transformer plaque segmentation module, an image encoder, a spatial cue encoder, a cue mixer, and a lesion cue injection module. By segmenting, locating, and generating clinical language description text, and combining a multi-scale encoder and a lightweight decoder, the approach achieves accurate segmentation of lesions and plaques and highly clinically relevant text descriptions.
It improves the accuracy and granularity of automatic identification of lesions and plaques. The generated text reflects the visual information and pathological features of lesions and plaques, achieving a tight coupling between visual features and semantic generation, and enhancing the clinical relevance and interpretability of the text.
Smart Images
Figure CN121237296A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a method for automatically identifying atherosclerotic plaques in intravascular ultrasound based on a visual language model. BACKGROUND
[0002] Coronary artery disease, as one of the major causes of death worldwide, its diagnosis and treatment is highly dependent on the accurate assessment of atherosclerotic plaques. Intravascular ultrasound (IVUS) is the gold standard imaging technology for evaluating coronary artery lesions, which can directly present the microstructure of the vessel wall and the plaque burden, providing important reference for risk stratification and interventional treatment. However, the current interpretation of IVUS images mainly relies on professional doctors' manual interpretation, which is tedious and time-consuming, and is easily affected by personal experience and subjective factors, resulting in poor consistency of diagnostic results and low efficiency of diagnosis.
[0003] Some automatic analysis methods and technologies have also emerged, such as methods that use echo intensity, texture, geometric morphology and other features for extraction, combined with classic machine learning algorithms such as support vector machines and decision trees, to realize the analysis of IVUS images. However, these methods are easily affected by image noise and artifacts, cannot adapt to the high heterogeneity of plaque morphology, and have limited generalization ability and interpretability.
[0004] With the development of deep learning, end-to-end models such as convolutional neural networks have made some progress in plaque segmentation and classification tasks, but these models mostly only output discrete class labels, lack precise positioning of spatial details of lesions, and cannot provide clinically interpretable diagnostic evidence. Existing methods can usually only analyze a single overall image, making it difficult to meet the actual clinical needs of a single IVUS image containing multiple plaques that need to be evaluated independently. In addition, there is a gap between the automatic analysis system and the narrative expression of the clinical report, which cannot generate detailed natural language descriptions that fit the medical report, restricting its application in electronic medical records, diagnostic processes and decision support systems.
[0005] Recently, large-scale visual language models (VLM) have made breakthroughs in medical image generation and reporting tasks, but they are mainly aimed at chest X-rays, pathological sections and other overall images, and are mostly general models, making it difficult to directly migrate to the IVUS field. IVUS images have unique noise, artifacts and gray structure characteristics, which pose higher requirements for the spatial positioning and professional context adaptation of the model. The actual clinical needs require the model to focus on specific lesions when generating text, accurately describe plaque types, spatial positions and pathological characteristics, and conform to the reporting standards of the cardiovascular field. Existing automatic systems have not yet solved the problem of how to combine lesion-level visual information with professional natural language descriptions in a refined manner, resulting in a disconnect between the model's results and clinical needs, and a lack of effective explainability mechanisms.
[0006] In summary, existing technologies have significant shortcomings in areas such as automatic plaque identification, spatial localization, and clinically interpretable natural language description, making it difficult to achieve a balance between automation, precision, and interpretability. There is an urgent need for a method for identifying intravascular ultrasound lesions. Summary of the Invention
[0007] The technical problem to be solved by the present invention is to provide a method that improves the accuracy and granularity of automatic identification of lesions and plaques in intravascular ultrasound images, and generates reports with high clinical professionalism and interpretability, achieving a high degree of consistency with clinical reports.
[0008] The technical solution adopted by this invention to solve the above-mentioned technical problems is: an automatic identification method for atherosclerotic plaques by intravascular ultrasound based on a visual language model, wherein the visual language model includes...
[0009] The hierarchical Transformer plaque segmentation module is used to process intravascular ultrasound images to segment and locate lesions and plaques within the intravascular ultrasound images, thereby acquiring image data of all lesions and plaques in the intravascular ultrasound images. Seg Obtain segmentation mask data U for all lesion plaques Mask and bounding box data U Box ;
[0010] An image encoder is used to process intravascular ultrasound images to obtain global visual features of all lesions and plaques in the intravascular ultrasound images. SAM ;
[0011] The spatial cue encoder interacts with the hierarchical Transformer plaque segmentation module to generate spatial cue embedding data P corresponding to each lesion plaque. SAM ;
[0012] The cue mixer, which interacts with the hierarchical Transformer patch segmentation module, image encoder, and spatial cue encoder, is used to generate a lesion-perception hybrid cue P that deeply fuses spatial and visual information of lesion patches. Mix ;
[0013] The lesion indication injection module interacts and communicates with the image encoder and indication mixer to generate clinical language description text for the lesion, which includes the lesion's visual information and pathological features.
[0014] The method for automatic identification of atherosclerotic plaques via intravascular ultrasound based on a visual language model includes:
[0015] Step 1: Acquire intravascular ultrasound images;
[0016] Step 2: Segment the intravascular ultrasound image based on the hierarchical Transformer plaque segmentation module to obtain image data of all lesions and plaques in the intravascular ultrasound image. Seg Obtain segmentation mask data U for all lesion plaques Mask and bounding box data U Box ;
[0017] Image encoders are used to process intravascular ultrasound images to obtain global visual features of all lesions and plaques in the intravascular ultrasound images. SAM ;
[0018] Step 3: Segmentation mask data U for all lesion patches based on the spatial cue encoder. Mask and bounding box data U Box The data is encoded and then converted into spatial cue embedding data P corresponding to each lesion plaque. SAM ;
[0019] Step 4: Fuse image data I based on prompt mixer Seg Global visual features I SAM Embedded data P with spatial cues SAM This leads to the generation of mixed lesion perception prompts P Mix ;
[0020] Step 5: Inject global visual features I based on lesion indication module SAM Mixed lesion perception prompts P Mix The process involves processing the data to generate a clinical description of the lesion.
[0021] To avoid complex parameter adjustments and reduce computational resources and time consumption, the image encoder in the visual language model is a frozen SAM image encoder, and the spatial cue encoder is a frozen SAM cue encoder.
[0022] In order to dynamically inject spatial information and visual cues into the visual language bridge module, a tight coupling between visual features and semantic generation is achieved, which greatly improves the clinical relevance and interpretability of the text. The lesion cues injection module includes an interactive cues perception query converter and a large language model decoder.
[0023] In step 5, the cue-aware query converter utilizes a set of learnable query embeddings, Q-learning, and connects them with global visual features via a cross-attention layer. SAM Interactive processing allows for the extraction of visual information describing the lesions and plaques;
[0024] Simultaneously, mixed lesion perception suggests P MixThe key K and value V are injected into the projection matrix of the cross-attention layer in the cue-aware query converter, and the weights of the key K and value V of the cross-attention layer are modulated using LoRA low-rank adaptive technology, so that the cue-aware query converter focuses on the view information of lesion plaques.
[0025] The cue-aware embedding output by the cue-aware query converter is used as a soft cue input to a large language model decoder, and then the decoding process is used to obtain clinical language description text that reflects the view information and pathological features of lesions and plaques.
[0026] To make the narrative of clinical reports more natural and enhance their interpretability, the large language model decoder uses an autoregressive decoding strategy to generate natural language descriptions word by word, so that the text reflects the visual information and pathological features of lesions and plaques.
[0027] To optimize the task by training on a small number of parameters and reduce the consumption of computing resources and time, the prompt-aware query converter adopts the Q-Former query converter, and the large language model decoder is a frozen LLM decoder based on BLIP-2.
[0028] To effectively achieve multi-scale accurate segmentation of lesions and plaques in intravascular ultrasound images, support complex scenarios with multiple lesions and plaques coexisting, and overcome the limitations of single-size processing, the hierarchical Transformer plaque segmentation module includes an interactive multi-scale encoder and decoder. The multi-scale encoder adopts a hierarchical Transformer architecture inspired by SegFormer, and the decoder is a lightweight MLP decoder.
[0029] Step 2 includes steps 2.1 to 2.5;
[0030] Step 2.1: After preprocessing the intravascular ultrasound images, they are sent to a multi-scale encoder;
[0031] Step 2.2: The multi-scale encoder divides multiple lesions and plaques in the intravascular ultrasound image into multiple non-overlapping patches, and linearly embeds these patches into a high-dimensional space to form an embedding sequence.
[0032] Step 2.3: The embedded sequence is sequentially passed through the self-attention layer and hybrid feedforward network of each Transformer coding layer in the hierarchical Transformer architecture to realize multi-granularity processing of intravascular ultrasound images and generate multi-scale feature maps.
[0033] Step 2.4: After the multi-scale feature maps are passed through the unified channel of the MLP layer of the MLP decoder, they are upsampled to a unified resolution and stitched together to form the aggregated feature F_agg;
[0034] Step 2.5: The MLP decoder outputs a segmentation mask U based on the decoding processing of the aggregated feature F_agg. Mask Simultaneously, the bounding box U is calculated based on the mask contour. Box .
[0035] Preferably, in step 2.1, the method for preprocessing the intravascular ultrasound image is to adjust the intravascular ultrasound image to 224×224 pixels and perform standardization and normalization using ImageNet pre-trained statistics.
[0036] Compared with existing technologies, the advantages of this invention are as follows: The automatic identification method for atherosclerotic plaques based on a visual language model in this invention improves the automatic identification rate, accuracy, and fine-grained level of description of each lesion plaque in the intravascular ultrasound image by segmenting the intravascular ultrasound image. Simultaneously, when generating clinical language description text, it guides attention to the visual features of the lesion plaque, ensuring that the generated clinical language description text reflects the visual information and pathological features of the lesion plaque, achieving a tight coupling between visual features and semantic generation, greatly improving the clinical relevance and interpretability of the text. In particular, it overcomes the technical bottleneck of achieving fine-grained description at the lesion level and the fusion of clinical diagnostic language in complex lesion scenarios with multiple plaques by innovatively introducing a high-performance hierarchical Transformer segmentation module and a lesion prompt injection module, effectively improving the ability to focus on the pathological features of the lesion plaque and the clinical relevance of the generated text. Attached Figure Description
[0037] Fig. 1 This is an architecture diagram of the visual language model in an embodiment of the present invention.
[0038] Fig. 2 This is an architecture diagram of the layered Transformer patch segmentation module in an embodiment of the present invention.
[0039] Fig. 3 This is an architectural diagram of the lesion indication injection module in an embodiment of the present invention. Detailed Implementation
[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0041] like Figs. 1 to 3 As shown, this embodiment constructs a visual language model to achieve automatic identification of intravascular ultrasound atherosclerotic plaques, and finally automatically generates clinical language description text with high clinical interpretability and in line with medical report standards, which is convenient for widespread use in electronic medical records, automated diagnostic processes and other fields.
[0042] In this embodiment, the visual language model includes a hierarchical Transformer patch segmentation module, an image encoder, a spatial cue encoder, a cue mixer, and a lesion cue injection module.
[0043] The hierarchical Transformer plaque segmentation module is used to process intravascular ultrasound images to segment and locate lesions and plaques within the intravascular ultrasound images, thereby acquiring image data of all lesions and plaques in the intravascular ultrasound images. Seg Obtain segmentation mask data U for all lesion plaques Mask and bounding box data U Box .
[0044] An image encoder is used to process intravascular ultrasound images to obtain global visual features of all lesions and plaques in the intravascular ultrasound images. SAM .
[0045] The spatial cueing encoder interacts with the hierarchical Transformer plaque segmentation module to generate spatial cueing embedding data P corresponding to each lesion plaque. SAM .
[0046] The cue mixer interacts with the hierarchical Transformer patch segmentation module, image encoder, and spatial cue encoder to generate a lesion-perception hybrid cue P that deeply fuses spatial and visual information of lesion patches. Mix .
[0047] The lesion indication injection module interacts and communicates with the image encoder and indication mixer to generate clinical language description text for the lesion. The clinical language description text includes the lesion's view information and pathological features.
[0048] To avoid complex parameter adjustments and reduce computational resources and time consumption, the image encoder in the visual language model is a frozen SAM image encoder, and the spatial cue encoder is a frozen SAM cue encoder.
[0049] Based on the aforementioned visual language model, automatic identification of intravascular ultrasound atherosclerotic plaques can be performed. The corresponding method for automatic identification of intravascular ultrasound atherosclerotic plaques based on the visual language model includes the following steps 1 to 5.
[0050] Step 1: Acquire intravascular ultrasound images. These images are acquired using existing ultrasound imaging equipment and then fed into a visual language model for processing.
[0051] Step 2: Segment the intravascular ultrasound image based on the hierarchical Transformer plaque segmentation module to obtain image data of all lesions and plaques in the intravascular ultrasound image.Seg Obtain segmentation mask data U for all lesion plaques Mask and bounding box data U Box Image encoders are used to process intravascular ultrasound images to obtain global visual features of all lesions and plaques in the intravascular ultrasound images. SAM .
[0052] Some intravascular ultrasound images contain multiple plaque lesions of varying sizes. Using a single-scale encoder makes it difficult to achieve multi-level fine-grained description of these plaque lesions. In this embodiment, to effectively achieve multi-scale accurate segmentation of plaque lesions in intravascular ultrasound images, supporting complex scenarios with multiple plaque lesions coexisting, and overcoming the limitations of single-size processing, this method addresses these issues. Fig. 2 As shown, the hierarchical Transformer patch segmentation module includes an interactive multi-scale encoder and decoder. The multi-scale encoder adopts a hierarchical Transformer architecture inspired by SegFormer, and has multiple Transformer encoding layers, enabling the extraction of multi-scale features. The decoder in this embodiment is a lightweight MLP decoder. The MLP decoder directly fuses multi-scale features at multiple resolutions, such as 1 / 4, 1 / 8, 1 / 16, and 1 / 32, output by the Transformer encoder. By aggregating feature information from different levels and combining local and global attention mechanisms, it achieves efficient and accurate semantic segmentation without the need for complex decoding structures. Multi-scale feature fusion enhances the model's resistance to image perturbations, while retaining the advantages of non-local attention mechanisms and reducing computational load.
[0053] Step 2 in this embodiment includes steps 2.1 to 2.5.
[0054] Step 2.1: After preprocessing the intravascular ultrasound image, it is sent to the multi-scale encoder.
[0055] The preprocessing method for intravascular ultrasound images is to adjust the intravascular ultrasound images to 224×224 pixels and perform standardization and normalization using ImageNet pre-trained statistics.
[0056] Step 2.2: The multi-scale encoder divides multiple lesions and plaques in the intravascular ultrasound image into multiple non-overlapping patches, and linearly embeds these patches into a high-dimensional space to form an embedding sequence.
[0057] Step 2.3: The embedded sequence is sequentially passed through the self-attention layer and hybrid feedforward network of each Transformer coding layer in the hierarchical Transformer architecture to realize multi-granularity processing of intravascular ultrasound images and generate multi-scale feature maps.
[0058] Step 2.4: After the multi-scale feature maps are passed through the unified channel of the MLP layer of the MLP decoder, they are upsampled to a unified resolution and stitched together to form the aggregated feature F_agg.
[0059] Step 2.5: The MLP decoder outputs a segmentation mask U based on the decoding processing of the aggregated feature F_agg. Mask Simultaneously, the bounding box U is calculated based on the segmented mask contour. Box Once this step is completed, the spatial information of the lesion plaque in the intravascular ultrasound image is obtained accordingly.
[0060] Step 3: Segmentation mask data U for all lesion patches based on the spatial cue encoder. Mask and bounding box data U Box The data is encoded and then converted into spatial cue embedding data P corresponding to each lesion plaque. SAM .
[0061] Step 4: Fuse image data I based on prompt mixer Seg Global visual features I SAM Embedded data P with spatial cues SAM This leads to the generation of mixed lesion perception prompts P Mix .
[0062] Step 5: Inject global visual features I based on lesion indication module SAM Mixed lesion perception prompts P Mix The process involves processing the data to generate a clinical description of the lesion.
[0063] To dynamically inject spatial information and visual cues into the visual-language bridge module, a tight coupling between visual features and semantic generation is achieved, greatly enhancing the clinical relevance and interpretability of the text, such as... Fig. 3 As shown, the lesion suggestion injection module in this embodiment includes a suggestion-aware query converter and a large language model decoder for interactive communication. To optimize the task through training with a small number of parameters and reduce computational resource and time consumption, the suggestion-aware query converter uses the Q-Former query converter, and the large language model decoder is a frozen LLM decoder based on BLIP-2. The Q-Former query converter is responsible for converting the frozen image encoder output into a format understandable by the language model, achieving cross-modal alignment through a self-attention mechanism. It is a method that integrates visual and language models, used in BLIP-2 to combine a pre-trained image encoder and a large language model. Q-Former transforms visual semantic representations into textual semantics through learnable queries (soft cues). BLIP-2 is an efficient visual-language pre-training strategy that reduces training costs and avoids catastrophic forgetting by freezing the LLM decoder.
[0064] In step 5, the cue-aware query converter utilizes a set of learnable query embeddings, Q-learning, and connects them with global visual features via a cross-attention layer. SAM Interactive processing allows for the extraction of visual information describing the lesions and plaques.
[0065] Simultaneously, mixed lesion perception suggests P Mix The key K and value V are injected into the projection matrix of the cross-attention layer in the cue-aware query converter, and the weights of key K and value V in the cross-attention layer are modulated using LoRA low-rank adaptive technology, enabling the cue-aware query converter to focus on the visual information of lesions and plaques. By employing LoRA low-rank adaptive technology, spatial and visual cues are dynamically injected into the visual-language bridge module, achieving tight coupling between visual features and semantic generation, greatly improving the clinical relevance and interpretability of the text.
[0066] The cue-aware embedding output by the cue-aware query converter is used as a soft cue input to a large language model decoder, and then the decoding process is used to obtain clinical language description text that reflects the view information and pathological features of lesions and plaques.
[0067] To make the narrative of clinical reports more natural and enhance interpretability, the large language model decoder uses an autoregressive decoding strategy to generate natural language descriptions word by word, so that the text reflects the visual information and pathological features of lesions and plaques.
[0068] The automatic identification method for atherosclerotic plaques based on visual language models in this invention performs excellently on the MICCAI 2011 IVUS public dataset: the Dice coefficient of the segmentation module reaches 0.925, which is significantly better than mainstream models such as U-Net and UNETR; the text description achieves BLEU-4 of 33.8 and CIDEr of 82.4, leading many advanced visual language models; the pathological terminology achieves F1 score of 0.781 and expert score of 4.4 / 5, verifying the clinical applicability of the generated text.
[0069] This invention presents an automatic identification method for atherosclerotic plaques using intravascular ultrasound based on a visual language model. By segmenting intravascular ultrasound images, it improves the automatic identification rate, accuracy, and fine-grained level of description of individual lesions and plaques within the images. Simultaneously, when generating clinical descriptive text, it guides attention to the visual features of the lesions and plaques, ensuring that the generated text reflects both visual information and pathological characteristics. This achieves a tight coupling between visual features and semantic generation, significantly enhancing the clinical relevance and interpretability of the text. In particular, it overcomes the technical bottleneck of achieving fine-grained lesion-level description and clinical diagnostic language integration in complex multi-plaque lesion scenarios by innovatively introducing a high-performance hierarchical Transformer segmentation module and a lesion prompt injection module, effectively improving the ability to focus on the pathological features of lesions and plaques and the clinical relevance of the generated text.
[0070] This invention presents an automatic identification method for atherosclerotic plaques using intravascular ultrasound based on a visual language model. Primarily aimed at assisting in the diagnosis and report generation of clinical intravascular ultrasound images, it is applicable to various scenarios including cardiovascular interventional surgery, coronary artery lesion assessment, and atherosclerotic plaque monitoring. By automatically generating fine-grained plaque descriptions with clinical diagnostic value, it significantly reduces the workload of physicians and improves diagnostic efficiency and report quality. Furthermore, the system architecture possesses good scalability, allowing for future integration with real-time interventional systems for online assisted diagnosis, or integration into intelligent cardiovascular imaging management platforms, thus promoting the intelligent upgrading of healthcare.
[0071] This method for automatic identification of atherosclerotic plaques using intravascular ultrasound based on a visual language model achieves seamless integration of high-precision localization of plaques in IVUS images and professional clinical language descriptions through an innovative architecture that combines visual segmentation and cue injection. This significantly improves the accuracy and clinical interpretability of automated diagnosis, providing strong technical support for the intelligent diagnosis and treatment of cardiovascular diseases.
Claims
1. An intravascular ultrasound atherosclerotic plaque automatic identification method based on a visual language model, characterized in that: The visual language model comprises The hierarchical Transformer plaque segmentation module is used for processing the intravascular ultrasound image to realize segmentation and positioning of the lesion plaque in the intravascular ultrasound image, and then image data I of all the lesion plaques in the intravascular ultrasound image is obtained Seg , segmentation mask data U of all the lesion plaques is obtained Mask , and boundary box data U Box An image encoder for processing an intravascular ultrasound image to obtain global visual features I of all lesion plaques in the intravascular ultrasound image SAM ; The spatial cue encoder, in interactive communication with the hierarchical Transformer patch segmentation module, is configured to generate spatial cue embedding data P corresponding to each lesion patch SAM ; The prompt mixer is in interactive communication with the layered Transformer patch segmentation module, the image encoder and the spatial prompt encoder, and is used for generating a lesion perception mixed prompt P of the spatial information and the visual information of the lesion patch being deeply fused Mix ; The lesion prompt injection module is in interactive communication with the image encoder and the prompt mixer, and is configured to generate a clinical language description text for the lesion, the clinical language description text comprising view information of the lesion and pathological characteristics of the lesion; The intravascular ultrasound atherosclerotic plaque automatic identification method based on the visual language model comprises Step 1: obtaining an intravascular ultrasound image of a blood vessel in a tube; Step 2, segmenting the intravascular ultrasound image based on the hierarchical Transformer plaque segmentation module to obtain image data I of all lesion plaques in the intravascular ultrasound image Seg , obtaining segmentation mask data U of all lesion plaques Mask and bounding box data U Box ; The image-based encoder processes the intravascular ultrasound image to obtain global visual features I of all lesion plaques in the intravascular ultrasound image SAM ; Step 3, encode the segmentation mask data U of all plaque lesions based on the spatial cue encoder Mask and the bounding box data U Box to perform encoding processing, and then convert into the spatial cue embedding data P corresponding to each plaque lesion SAM ; Step 4, fusing image data I based on the hint Seg global visual features I SAM with the spatial hint embedding data P SAM , and further generating lesion-aware mixed hints P Mix ; Step 5, the global visual feature I is processed based on the lesion prompt injection module to generate a clinical language description text for the lesion. SAM , the lesion perception mixed prompt P Mix is processed to generate a clinical language description text for the lesion.
2. The method of claim 1, wherein the method further comprises: determining a visual language model based on the extracted features; and classifying the plaque as atheromatous or non-atheromatous based on the visual language model. The image encoder in the visual language model is a frozen SAM image encoder, and the spatial prompt encoder is a frozen SAM prompt encoder.
3. The intravascular ultrasound atherosclerotic plaque automatic identification method based on a visual language model according to claim 1 or 2, characterized in that: The lesion prompt injection module comprises an interactive communication prompt-aware query converter and a large language model decoder. In step 5, the perception query translator is prompted to utilize a set of learnable query embeddings Q-learning, through cross-attention layers with global visual features I SAM interaction, and further extract view information describing the lesion patch; Meanwhile, the lesion-aware hybrid prompt P Mix The key K and the value V of the cross-attention layer in the prompt-aware query converter are projected in the key K and the value V projection matrix, and the LoRA low-rank adaptive technology is used to modulate the key K and the value V weight of the cross-attention layer, so that the prompt-aware query converter pays attention to the view information of the lesion plaque. The prompt-aware embedding output by the prompt-aware query converter is input to the large language model decoder as a soft prompt, and then a clinical language description text reflecting the view information and pathological characteristics of the lesion plaque is obtained through decoding processing.
4. The method of claim 3, wherein the method further comprises: determining a visual language model based on the visual features of the image; and determining the presence of the atherosclerotic plaque based on the visual language model. The large language model decoder adopts a word-by-word generation strategy to generate a natural language description, so that the text reflects the view information and pathological characteristics of the lesion plaque.
5. The method of claim 3, wherein the method further comprises: determining a visual language model based on the extracted features; and classifying the plaque as atheromatous or non-atheromatous based on the visual language model. The prompt-aware query converter adopts a Q-Former query converter, and the large language model decoder is a frozen LLM decoder based on BLIP-2.
6. The intravascular ultrasound atherosclerotic plaque automatic identification method based on a visual language model according to claim 1 or 2, characterized in that: The hierarchical Transformer plaque segmentation module comprises an interactive communication multi-scale encoder and a decoder, the multi-scale encoder adopts a hierarchical Transformer architecture based on SegFormer inspiration, and the decoder is a lightweight MLP decoder. Step 2 comprises steps 2.1 to 2.5; Step 2.1: after pre-processing the intravascular ultrasound image of the blood vessel in the tube, the image is sent to the multi-scale encoder; Step 2.2: the multi-scale encoder divides the multiple lesion plaques in the intravascular ultrasound image of the blood vessel in the tube into multiple non-overlapping patches, and linearly embeds these patches into a high-dimensional space to form an embedding sequence; Step 2.3: the embedding sequence sequentially passes through the self-attention layer and the mixed feedforward network of each Transformer encoding layer in the hierarchical Transformer architecture, realizing multi-granularity processing of the intravascular ultrasound image of the blood vessel in the tube, and then generating a multi-scale feature map; Step 2.4: the multi-scale feature map is uniformly channeled through the MLP layer of the MLP decoder, upsampled to a uniform resolution, and spliced to form an aggregated feature F_agg; Step 2.5, the MLP decoder outputs a segmentation mask U based on a decoding process of the aggregated features F agg Mask At the same time, a bounding box U is calculated according to the mask contour Box .
7. The intravascular ultrasound atherosclerotic plaque automatic identification method based on visual language model according to claim 6, characterized in that: In step 2.1, the pre-processing method of the intravascular ultrasound image of the blood vessel in the tube is to adjust the intravascular ultrasound image of the blood vessel in the tube to 224x224 pixels, and to normalize the image using ImageNet pre-training statistics.