Liver ultrasound contrast lesion classification method with multi-modal information and perfusion kinetics

Abstract

The application discloses a liver ultrasound contrast lesion classification method based on multi-modal information and perfusion kinetics, and the method comprises the following steps: acquiring video image data with perfusion contrast images and standard gray-scale images and preprocessing, and structuring a clinical data vector of a patient; identifying a liver lesion ROI region and obtaining a TIC curve for analysis to obtain blood flow kinetics key phase characteristics; based on a double-head input time sequence classification model of the gray-scale images and the contrast images, first spatial characteristics and second spatial characteristics are extracted and fused to obtain visual fusion characteristics; a clinical data vector is projected to an embedding space aligned with image characteristics to obtain a clinical feature vector; finally, an explainability heat map is obtained, blood flow kinetics key phase characteristics are marked on the TIC curve, the most important clinical indicators in classification decision are listed according to category probability, and the summary is used as an auxiliary diagnosis result.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence technology, specifically involving a liver ultrasound contrast imaging lesion classification method based on multimodal information and perfusion dynamics. It is an auxiliary classification method for liver focal lesions that combines ultrasound contrast imaging video, contrast agent time-intensity curve dynamics, and patient clinical text information. Background Technology

[0002] In the field of video classification technology, existing techniques have gradually evolved from traditional schemes based on manual feature extraction and shallow models to multimodal, multi-level feature fusion models centered on deep learning. Some researchers use traditional spatiotemporal feature extraction methods based on Convolutional Neural Networks (CNNs) for video classification. Typical frameworks such as C3D (3D ConvNet) and I3D (Inflated 3D ConvNet) significantly improve classification accuracy by directly learning feature representations in the spatial and temporal dimensions from video data.

[0003] To better model the spatiotemporal information in videos, some studies have proposed a two-stream architecture, which inputs RGB frames and optical flow information into two parallel networks respectively, and then performs feature fusion in the later stages to obtain a more comprehensive spatiotemporal representation. At the same time, multimodal data such as audio signals and text descriptions are also gradually being incorporated into video classification frameworks, enabling models to make judgments with rich information sources, thereby obtaining more robust and discriminative feature representations.

[0004] In recent years, Transformer has made significant progress in natural language processing tasks such as machine translation and text classification. Researchers have begun to introduce its self-attention mechanism into the field of video understanding. Transformer-based video classification models can capture long-range dependencies flexibly by focusing on the correlations between different frames and spatial regions in a video, without relying on strict convolutional structures. These models are often structurally divided into temporal and spatial domains, or they use a pure Transformer structure to jointly encode spatiotemporal features. Typical examples include TimeSformer and ViViT.

[0005] The classification and analysis of medical videos (such as ultrasound, endoscopy, and MRI dynamic sequences) is an important branch of video understanding in professional fields. Existing research mostly uses CNN or RNN-based methods (such as LSTM) for medical video classification and diagnostic support, but in recent years, some works have explored using Transformer structures to model the spatiotemporal features of medical video data. These attempts aim to identify lesions and classify diseases with higher accuracy and robustness, thereby providing stronger support for clinical decision-making.

[0006] However, existing liver ultrasound contrast imaging techniques still have the following significant drawbacks:

[0007] First, it lacks the ability to deeply fuse multimodal data. Ultrasound contrast imaging data typically includes left grayscale images and right contrast imaging, which differ significantly in feature space. Traditional Transformer or CNN models often struggle to establish a sufficient organic connection between these two modes. Simple stitching or parallel input cannot effectively fuse grayscale anatomical information with contrast perfusion information, resulting in insufficient lesion feature extraction.

[0008] Second, it neglects the dynamics of contrast agent perfusion. Existing AI analysis systems mostly focus on extracting spatial features from single frames or short time series images, lacking quantitative temporal dynamic modeling (TIC curves) of the contrast agent perfusion process such as "rapid in and out," and are unable to explicitly capture key medical semantics such as "rapid enhancement in the arterial phase," resulting in insufficient sensitivity of the model to temporal features.

[0009] Third, the utilization rate of clinical background information is low. Existing systems are mostly pure image models, failing to effectively integrate clinical text data such as patient age, medical history, and biochemical indicators (e.g., AFP, hepatitis B antigen). In actual clinical diagnosis, this non-imaging information is often an important basis for doctors to make differential diagnoses. The lack of this information greatly limits the diagnostic accuracy and clinical applicability of the model. Summary of the Invention

[0010] The technical problem solved by the present invention is that, although existing technologies have explored the use of Transformer structures to model the spatiotemporal features of medical video data, and Transformer-based video classification models can flexibly capture long-range dependencies by focusing on the correlation between different frames and spatial regions in the video, in the field of liver ultrasound contrast imaging lesion classification, CNN or RNN (such as LSTM) methods are still used to classify and diagnose medical videos. The use of Transformer structures to model the spatiotemporal features of liver ultrasound contrast imaging lesion video data still needs further research. Moreover, existing technologies mostly only process lesion videos and lack the combination of clinical text and lesion video analysis, resulting in inaccurate diagnostic criteria.

[0011] To address the aforementioned technical problems, the present invention provides a method for classifying liver lesions using contrast-enhanced ultrasound based on multimodal information and perfusion dynamics, comprising the following steps: Acquire video image data with perfusion angiography images and standard grayscale images and preprocess them, as well as structured clinical data vectors of patients; Based on the preprocessed video image data, the ROI regions of liver lesions were determined in the perfusion angiography images and standard grayscale images. The TIC curve was obtained by combining the average brightness-time relationship and analyzed to obtain the key phase characteristics of hemodynamics. A dual-head input temporal classification model based on grayscale images and contrast images captures the first spatial features of grayscale images and extracts the second spatial features of the ROI region of liver lesions. The clinical data vector is projected onto the embedding space aligned with the image features to obtain the clinical feature vector. The first spatial features and the second spatial features are fused to generate visual fusion features. The TIC curve, hemodynamic key phase features and visual fusion features are fused to obtain time-aware features. Multimodal temporal features are generated by using a cross-modal cross-attention fusion mechanism and combining clinical feature vectors. Modeling the dependencies between different time frames outputs global temporal features. Combining these with the final multimodal temporal features, the model is mapped to the target category probability distribution to generate category probabilities. This results in an interpretable heatmap, marking key phase features of hemodynamics on the TIC curve, listing the clinical indicators that contribute most to the classification decision based on the category probabilities, and summarizing these findings as auxiliary diagnostic results.

[0012] Preferably, the clinical text data is structured into the clinical data vector, which includes hepatitis B surface antigen, hepatitis B e antigen, antibody indicators, alpha-fetoprotein levels, liver function indicators, years of alcohol abuse, and past medical history.

[0013] Preferably, sparse frame detection and dense frame tracking are used to process the preprocessed video image data to obtain the ROI region of the liver lesion.

[0014] Preferably, the input layer of the dual-head input temporal classification model based on grayscale images and angiography images assigns a base value of highest importance for the arterial phase, a base value of moderate importance for the portal venous phase, and a base value of low importance for the delayed phase, based on the TIC curve and key phase features of hemodynamics.

[0015] Preferably, the key hemodynamic phase features include the arterial phase start point, the arterial phase peak point, and the regression phase start point.

[0016] Preferably, the dual-head input temporal classification model uses multi-layer ViT processing to obtain the first spatial features and the second spatial features.

[0017] This invention provides a method for classifying liver lesions using multimodal information and perfusion dynamics via contrast-enhanced ultrasound. It acquires and preprocesses video image data containing both perfusion-enhanced and standard grayscale images, along with structured clinical data vectors of the patient. The method identifies the ROI region of the liver lesion and obtains TIC curves for analysis to derive key hemodynamic phase features. A dual-input temporal classification model based on grayscale and contrast-enhanced images extracts first and second spatial features, which are then fused to obtain visual fusion features. The clinical data vectors are projected into an embedding space aligned with the image features to obtain clinical feature vectors. Finally, an interpretable heatmap is obtained, key hemodynamic phase features are marked on the TIC curves, and clinical indicators contributing most to the classification decision are listed according to category probability. These results are then summarized as auxiliary diagnostic findings. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the liver ultrasound contrast imaging lesion classification method based on multimodal information and perfusion dynamics provided in this embodiment of the invention; Figure 2 This is a schematic diagram of a lesion tracking pipeline provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the TIC processing flow provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a dual-head input temporal classification model based on grayscale images and angiographic images provided in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0020] like Figure 1 As shown in the embodiment of the present invention, the method for classifying liver lesions by multimodal information and perfusion dynamics using contrast-enhanced ultrasound includes the following steps: The video image data originates from medical ultrasound machines. These machines typically provide B-mode (grayscale) ultrasound video output, which can simultaneously display perfusion imaging images (usually in a specific mode) and standard grayscale images on the screen. A video capture card is used to capture the real-time video output from the ultrasound machine at a high frame rate.

[0021] For textual data, patient clinical data is obtained from the hospital information system, including but not limited to: age, gender, hepatitis B surface antigen, hepatitis B e antigen, antibody indicators, alpha-fetoprotein levels, liver function indicators, years of alcohol consumption, and past medical history. This data is structured into a numerical vector or a category label vector C_patient.

[0022] The acquired video images are preprocessed, including frame extraction, linear normalization, and size scaling.

[0023] Step 1. Use FFmpeg to extract frames from the video and assign a precise timestamp to each frame to ensure the correct order of video frames and guarantee the accuracy of subsequent timing analysis.

[0024] Step 2. Assuming the frame width is W and the middle position of the image is W / 2, the left half [0:W / 2] is the perfusion imaging image and the right half [W / 2:W] is the grayscale image. Store the two sub-images as two independent image tensors.

[0025] Step 3. Perform linear normalization on the pixel values. Step 4. Resize to scale both left and right sub-images to the same resolution.

[0026] Extract the ROI region of the preprocessed video image and calculate the TIC curve.

[0027] Step 1. Train the detection network.

[0028] Doctors were instructed to mark detection boxes for lesion areas in a small portion of the contrast-enhanced video. The detection network chosen was YOLOv11, a lightweight detection network that also met the accuracy requirements of the invention. This small portion of data was used to fine-tune the model, training a detection network for liver lesion detection in contrast-enhanced video.

[0029] Step 2. Extract the ROI region of liver lesions from the preprocessed video image.

[0030] Considering that the preprocessed video image data contains both perfusion imaging images and standard grayscale images, and that the two have a strict physical spatial registration relationship when output by ultrasound equipment, when extracting the region of interest (ROI) of liver lesions, one modality (such as the perfusion imaging image with more clearly defined features) is selected for extracting the spatial coordinates of the lesion. Subsequently, the obtained spatial bounding box or contour coordinates of the lesion are synchronously mapped to another modality (standard grayscale image), thereby parallelly cropping and outputting the perfusion imaging ROI sequence and the standard grayscale ROI sequence, which are completely aligned in space.

[0031] To obtain the spatial coordinates of the lesion under different computing power environments and accuracy requirements, this invention provides multiple target acquisition strategies. Before applying any of the following strategies, the sparse frame extraction interval is pre-set to be... Frame (preferred value range) ), then every ) in the video sequence A frame of an image is defined as a sparse frame, and the interval between two adjacent sparse frames is... A continuous frame image is defined as an intermediate frame.

[0032] Specific methods for obtaining the ROI of lesions include, but are not limited to, the following three: Implementation Method 1 (Based on Sparse Frame Detection and Dense Frame Tracking Collaboration): When computing resources are limited or extremely high real-time performance is required, a fine-tuned target detection network is used to detect the sparse frames (e.g., set...). (That is, 1 frame is taken every 100 frames) to detect lesions, and the output includes the initial bounding box. The detection frame; for a large number of intermediate frames of standard grayscale images, a dynamic tracking algorithm (such as optical flow) is used to locate the bounding box of the most recent detection frame. Using the initial template, target search and location prediction are performed in subsequent intermediate frames, and the lesion bounding box is output in real time for each frame.

[0033] Implementation Method 2 (Based on Frame-by-Frame Dense Deep Learning Prediction): In scenarios with high-performance cloud or edge GPU computing power, pre-trained deep learning object detection networks (such as YOLO series networks) or instance segmentation networks are directly used to perform independent end-to-end inference on each consecutive frame of the video stream, directly outputting the accurate lesion boundary rectangle or lesion mask contour for each frame.

[0034] Implementation Method 3 (Temporal Contour Evolution Based on Interactive Initialization): Receive interactive instructions (such as selecting the center point of the lesion or defining the initial bounding box) input by the operator on the initial frame or key frame of the video sequence, and generate the initial lesion ROI contour using a region growing algorithm or active contour model; then, along the time dimension, automatically track and evolve the ROI contour to the preceding and following consecutive video frames using inter-frame registration or deformation field technology.

[0035] Uniform constraints for the ROI region of liver lesions: Regardless of the implementation method used to extract the spatial coordinates of the lesion, the final extracted and output liver lesion ROI region sequence must meet the following preset conditions to ensure the accuracy of subsequent time series and dynamic analysis: A. Spatial Inclusion and Adaptive Expansion Conditions: The boundary of the ROI region must completely cover the morphological edge of the target lesion, and on this basis, adaptive area enlargement is performed according to a preset ratio (preferably expanding outward by 20%). This ensures that each frame of the image can completely capture the contrast agent perfusion characteristics of the core area of ​​the lesion and its surrounding microvessels.

[0036] B. Spatiotemporal smoothness condition: The offset of the ROI coordinate center point and the rate of change of the same target lesion between consecutive adjacent video frames must be lower than the preset temporal smoothing threshold. This is to suppress high-frequency jitter of the bounding box caused by the patient's breathing or small displacement of the probe, and to prevent non-physiological abrupt noise from being generated in the subsequently generated TIC curve.

[0037] C. Modal alignment condition: Under the same timestamp, the output perfusion angiography ROI and the standard grayscale ROI must maintain strictly consistent center coordinates and scale ratios.

[0038] Step 3. Calculate the TIC curve.

[0039] like Figure 3 As shown, since the lesion results are not entirely derived from model detection and inference, the lesion area generated by each contrast-enhanced video in the dataset is expanded by 20% to ensure that the entire lesion is captured in each frame. The brightness of the lesion area is calculated, and the average brightness-time relationship of the entire video is calculated to obtain a TIC curve (Time-Signal Intensity curve) of length T. This curve reflects the blood perfusion and outflow rates of the lesion and is an important basis for doctors' diagnosis. Analysis of the TIC curve automatically detects and outputs key hemodynamic phase features, including three key time points: t_arterial: the onset of the arterial phase (the inflection point where the contrast agent first rapidly rises); t_peak: the peak point of the arterial phase (the curve reaches its highest point); t_washout: the onset of the resolution phase (the inflection point where the curve begins to significantly decline from the peak).

[0040] Because the input image is a special ultrasound contrast perfusion image, this invention designs a dual-input temporal classification model based on grayscale images and contrast images. The model structure is as follows: Figure 4 The detailed structure and flow of the model are as follows: A. Input Layer: a. The input image data consists of two image channels, one of which is a grayscale ultrasound image with a shape of... One is a contrast-enhanced ultrasound image, shaped as ,in The input is a grayscale ultrasound image / video. This is a video image of contrast-enhanced ultrasound, where T is the frame rate, H is the image height, and W is the image width. The number of channels in a grayscale image. This represents the number of imaging image channels.

[0041] Each frame's two images are segmented and embedded into non-overlapping dimensions of size P (e.g., 16). 16) Image patches are used to form the initial token. The token is mapped linearly to a fixed dimension d.

[0042] b. The obtained TIC curve and three time points are input as a time importance weight vector, and the generation strategy is as follows: The arterial phase [t_arterial:t_peak] is assigned the highest importance weight base value (e.g., 1.0).

[0043] The portal pulsation period [t_peak:t_washout] is assigned a moderately weighted base value (e.g., 0.7).

[0044] The delay period [t_washout] and other periods are assigned lower base weights (e.g., 0.5).

[0045] This base weight vector will be smoothed and adaptively adjusted through a learnable linear layer to generate the final time-aware encoding P_time. In other words, the weights will be optimized through subsequent model iterations.

[0046] c. The structured clinical data vector C_patient.

[0047] B. Dual-channel spatial encoder and text multilayer perceptron.

[0048] Left channel (grayscale image encoder): Input grayscale image tokens; Multi-layer Vision Transformer (ViT) processing captures the first spatial features: positional encoding is added, and a self-attention mechanism is used to model the relationship between patches. The first spatial features are mainly used to characterize the anatomical structure and morphological priors of liver parenchyma and lesions, including but not limited to: the two-dimensional boundary contour of the lesion, the uniformity of the internal echo, the texture features of the tissue base, and the surrounding halo sign. Output the spatial features of each frame: ,in This refers to the spatial features of a grayscale image.

[0049] Right channel (contrast image encoder): Input the imaging image tokens; The lesion region ROI obtained when calculating the TIC curve is also split into tokens and added to the original tokens, so that the model pays more attention to the lesion region; The VIT module is also used to extract the second spatial features. The second spatial features are mainly used to characterize the spatial distribution of microvascular blood flow perfusion in the lesion target area, including but not limited to: the local enhancement intensity distribution of the contrast agent at a specific time frame, the non-uniform enhancement pattern (such as ring enhancement, nodular enhancement) and the features of necrotic areas with no blood flow perfusion inside. Output the spatial features of each frame: ,in Spatial features of the imaging image.

[0050] Textual multilayer perceptron: Input the clinical data vector C_patient; Clinical data is projected into an embedding space aligned with image features using a multilayer perceptron. Output clinical feature vector ∈R^d.

[0051] C. Fusion Module: a. Generate dynamic fusion weights using a self-attention mechanism:

[0052] Output unified visual fusion features ,in The fused temporal characteristics.

[0053] b. By incorporating information from TIC curves, visual fusion features, and key hemodynamic phase features, a priori understanding of time medicine is injected, resulting in time-perception features:

[0054] Where P_time is the time-aware encoding of that time.

[0055] c. Perform cross-modal attention fusion on the text information, Treat it as a set of sequence queries. Consider the key contextual information (Key and Value) and calculate cross-attention:

[0056] This crossover allows each frame of the image to obtain clinical textual information in the model.

[0057] d. Residual fusion: The cross-modal attention output is merged with the original features to generate the final multimodal temporal features.

[0058] D. Time encoder.

[0059] Input fusion features Temporal features are captured through a multi-layer temporal transformer encoder: a self-attention mechanism is used to model the dependencies between different time frames and output global temporal features. ,in For the global time features of the time encoder, The dimension of the fusion feature.

[0060] E. Classification Header Multimodal temporal features are passed through a fully connected layer and a Softmax (classification activation function) layer to generate class probabilities.

[0061] The specific steps are as follows: A. Input Layer After the input video is processed by this model, the output will be the auxiliary diagnostic result for that video. In step A, the segmentation embedding is Tubelet Embedding (embedding for spatiotemporal blocks), where a Tubelet is a three-dimensional block containing the temporal and spatial patches of a video. Specifically, for the input video segment... Set the size of the Tubelet and quantity t is the time frame length, P is the length and width of each Tubelet, T is the time frame length of the input video, H is the height of a frame, and W is the width of a frame.

[0062] After setting, the following formula is used: [1,M] in, This represents the i-th Tubelet. To expand the Tubelet into a vector, It is the linear mapping matrix of the Tubelet. This is the bias for Tubelet embedding. After the formula, the video stream can be segmented and embedded into a Tebelet feature embedding sequence. , where M is the number of tubelets, that is, the number of tubelets initially set.

[0063] Because the Transformer is not sensitive to the order of the input sequence, positional encoding needs to be added to the Embedding after the TubeletEmbedding to preserve positional information.

[0064]

[0065] in, It is the position encoding of the i-th token. This invention uses a sine-cosine function.

[0066] B. Dual-channel space encoder For the input It requires processing through multiple VIT layers, as detailed below: Self-attention mechanism In each layer, multi-head self-attention is calculated: Calculation of Query, Key, and Value:

[0067] Here, Query(Q), Key(K), and Value(V) are all special names for intermediate variables in the model, and they are all vectors. It is the input of the l-th layer. These are all learnable parameter matrices.

[0068] Attention score and weighted output ( ): This operation is called Attention, and the formula is:

[0069] in, Let K be the transpose of matrix K. It represents the dimensions of the Q and K vectors for each attention head. It is a normalization factor used to scale attention scores, because Q and The dot product value can be very large, causing the Softmax output to approach its limit, so it is necessary to use... To perform scaling.

[0070] Multi-head attention: That is, the results of multiple heads are concatenated and then subjected to a linear transformation, the formula is:

[0071] The output of each head , It is a linear projection matrix.

[0072] Residual Linking and Layer Normalization (Residual + LayerNorm) The formula is as follows:

[0073] Feed-Forward Network (FFN) Each layer also contains a feedforward network, which consists of two linear layers and a nonlinear activation, as shown in the following formula:

[0074] Where W is a learnable parameter matrix, and b is a learnable bias. C. Fusion Module Calculation of attention weights For each time step t, the attention weights for the left and right paths are calculated based on the relevance of the input features. The features for the left and right paths are defined as Q and K, respectively. The relevance is calculated through a dot product, and then scaled to obtain the normalized weights for the left and right paths. :

[0075] Dynamic fusion features The left and right features are weighted and summed using attention weights to generate fused features:

[0076] in, and These are the weights assigned to the left and right paths respectively. and These are the characteristics of the left and right paths at time step t.

[0077] D. Time encoder The structure of the time transfactor is modified by adding temporal positional encoding: Since the transformer is not sensitive to the order of the sequence, positional encoding is added to explicitly represent the order of time frames:

[0078] Wherein, P is a fixed-time position code based on a sine-cosine function.

[0079] The self-attention mechanism, residual links, and layer normalized feedforward network are the same as those in the B operation, and are referred to here as the temporal transfactor layer.

[0080] A time encoder is constructed by stacking multiple time transformer layers. The input of each layer is the output of the layer above it.

[0081] For temporal global feature extraction, after passing through an L-layer Transformer, the features from all time steps are averaged and pooled to extract global features.

[0082] E. Classification Header The classification head maps features to the probability distribution of the target class through a series of linear transformations, activation functions, and normalization operations. First, it passes through a fully connected layer:

[0083] Where W is the weight matrix of the classifier head, and b is the bias of the classifier head.

[0084] Then, the output of the classification head is converted into class probabilities through the Softmax layer.

[0085] After going through the entire process of the above system, the final algorithm can obtain the following output: Interpretable heatmaps: Transformer-based models can generate image regions that support decision-making based on the weights of different patches when making decisions.

[0086] TIC Curve and Phase Division: Visualize the automatically generated TIC curve and mark the key hemodynamic phase features detected by the system, intuitively demonstrating the hemodynamic analysis process.

[0087] Key clinical evidence: List the top clinical indicators that contributed most to this classification decision (e.g., "high AFP levels, long-term alcohol consumption history, and rapid arterial phase enhancement are the main criteria").

[0088] This system adopts a hybrid architecture of "edge-cloud collaboration" to balance real-time performance with data backtracking and analysis capabilities.

[0089] Edge computing nodes are high-performance workstations or embedded AI computing boxes deployed next to ultrasound equipment. They handle the most real-time demanding tasks, including video stream reception, real-time lesion tracking, and lightweight TIC curve generation and keyframe extraction. This ensures immediate feedback of diagnostic results, avoiding disruptions to doctors' operational smoothness due to network latency; all raw video data is processed at the edge, protecting patient privacy and complying with medical data security regulations.

[0090] The cloud-based central service platform runs the most complex multimodal fusion classification models, stores structured data, and performs model updates and big data analysis. It fully leverages the powerful GPU computing power of the cloud to run larger and more accurate models; enables data and model synchronization across multiple centers and devices; and facilitates continuous learning and model iteration.

[0091] This invention provides a fully automated classification system for focal lesions on liver ultrasound contrast imaging that integrates real-time lesion tracking and multimodal data, achieving a leap from "semi-automatic" to "fully automatic" and improving diagnostic efficiency and consistency. The invention designs a TIC (Time-Intensity Coefficient) module, which automatically generates time-intensity curves for lesion regions and intelligently identifies key phase points such as the onset of the arterial phase, peak value, and the onset of the decay phase, thereby generating time-aware codes and integrating them into the core classification model. To address the dual-path influence of grayscale and contrast imaging, the invention designs a dedicated dual-path spatial encoder, extracting spatial features from grayscale image sequences and contrast imaging sequences based on Vision Transformer (ViT), and innovatively introduces a self-attention-based dynamic fusion module to generate adaptive fusion weights for the two image features at each time step. To incorporate the patient's clinical background, the invention constructs a clinical data encoder and a cross-modal attention fusion mechanism, transforming the patient's textual clinical data (such as hepatitis B antigen / antibody, AFP, and alcohol history) into feature vectors and deeply interacting with the temporal features of the images.

[0092] Spatial feature extractors can be replaced with other feature extraction networks, such as CNN-based models like ResNet, VGG, and Convnext, or some variants of VIT. Models for real-time lesion monitoring can use other monitoring networks. The core is a comprehensive solution of monitoring and tracking.

[0093] The key technical points of the embodiments of the present invention are as follows: 1. Core Architecture: A grayscale-imaging dual-path spatiotemporal fusion model based on dynamic attention weighting; 2. Embedded Medical Knowledge: A Time-Aware Encoding Mechanism Based on Key Phase Point Detection of TIC Curves; 3. Information Upgrading: A Cross-Modal Attention Fusion Method for Imaging Features and Clinical Data; 4. Lightweight Innovation: A lightweight real-time lesion localization pipeline based on "detection-tracking" collaboration.

Claims

1. A method for classifying liver lesions using multimodal information and perfusion dynamics via contrast-enhanced ultrasound, characterized in that, Includes the following steps: Acquire ultrasound video image data including perfusion angiography images and standard grayscale images; The region of interest (ROI) of the liver lesion is determined based on the ultrasound video image data, and a time-intensity curve is extracted based on the ROI to obtain key hemodynamic phase features. Extract the first spatial features of the standard grayscale image and the second spatial features of the perfusion angiography image, and spatially fuse the first spatial features and the second spatial features to obtain visual fusion features; The time-intensity curve, the key phase features of hemodynamics, and the visual fusion features are fused temporally to obtain multimodal temporal features; Based on the multimodal temporal features, the target category probability distribution of liver lesions is generated and the classification results are output.

2. The method for classifying liver lesions using multimodal information and perfusion dynamics based on contrast-enhanced ultrasound as described in claim 1, characterized in that, The acquisition step also includes: The patient's clinical text data is acquired, and the clinical text data is structured and projected into the embedding space to obtain the clinical feature vector.

3. The method for classifying liver lesions using multimodal information and perfusion dynamics based on contrast-enhanced ultrasound according to claim 2, characterized in that, In the step of obtaining the multimodal temporal features: By utilizing a cross-modal cross-attention mechanism, the clinical feature vector is combined with the visual fusion features, time-intensity curves, and hemodynamic key phase features for feature interaction and fusion.

4. The liver ultrasound contrast-enhanced lesion classification method based on multimodal information and perfusion dynamics according to claim 2, characterized in that: The clinical text data includes at least one or more of the following: hepatitis B surface antigen, hepatitis B e antigen, antibody indicators, alpha-fetoprotein levels, liver function indicators, years of alcohol consumption, or past medical history.

5. The method for classifying liver lesions using multimodal information and perfusion dynamics based on contrast-enhanced ultrasound according to claim 1, characterized in that, The steps for determining the region of interest (ROI) of a liver lesion specifically include: A sparse frame detection network is used to detect lesions in the ultrasound video image data to obtain initial bounding boxes, and a dynamic tracking algorithm is used to track the target in dense frames to output the lesion bounding boxes of each video frame.

6. The method for classifying liver lesions using multimodal information and perfusion dynamics based on contrast-enhanced ultrasound according to claim 1, characterized in that, In the steps of extracting the first spatial features and the second spatial features: A dual-stream input network containing multiple layers of visual Transformer is used to extract the first spatial features and the second spatial features respectively; A self-attention mechanism is used to generate dynamic fusion weights, and the first spatial feature and the second spatial feature are weighted and summed to generate the visual fusion feature.

7. The method for classifying liver lesions using multimodal information and perfusion dynamics based on contrast-enhanced ultrasound according to claim 1, characterized in that, The key hemodynamic phase features include the arterial phase onset, arterial phase peak, and resolution phase onset. During time-series fusion, a base value for the time importance weight corresponding to different times is assigned based on the key hemodynamic phase characteristics, wherein the base value for the importance weight of the arterial phase is higher than that of the portal venous phase and the delayed phase.

8. The method for classifying liver lesions using multimodal information and perfusion dynamics based on contrast-enhanced ultrasound according to claim 1 or 2, characterized in that, The step of outputting the classification result also includes at least one of the following: Generate and output interpretable heatmaps corresponding to the liver lesions; Mark the key hemodynamic phase features on the visualized time-intensity curve; When the clinical feature vector is combined, the output is the clinical indicator that contributes the most to the classification decision.

Images