CEUS image analysis method based on key frame segmentation and time sequence tracking fusion

Abstract

The application provides a CEUS image analysis method based on key frame segmentation and time sequence tracking fusion, a CEUS video risk grading overall framework based on key frame screening, segmentation and time sequence fusion, integrates video preprocessing, key frame extraction, plaque segmentation, time sequence feature fusion and multi-frame risk decision and the like links, forms a complete risk grading process, and can be used for automatic grading of carotid plaque rupture risk. The application has the advantages of significantly improving efficiency, enhancing time sequence sensitivity, stabilizing grading results, being strong in clinical interpretability and being good in universality, and can be popularized to CEUS image analysis of organs such as liver and kidney.

Description

Technical Field

[0001] This invention relates to the field of image analysis and recognition, and in particular to a CEUS image analysis method based on keyframe segmentation and temporal tracking fusion. Background Technology

[0002] The formation and rupture of carotid atherosclerotic plaques are the main pathological basis for ischemic stroke. Clinical studies have shown that plaque stability is closely related to its internal neovascularization density, fibrous cap thickness, and dynamic morphological changes. Traditional two-dimensional grayscale ultrasound imaging can be used to detect plaque morphology, but it has insufficient sensitivity to minute blood flow perfusion characteristics.

[0003] CEUS (Continuous Ultrasound Imaging) enhances blood flow perfusion signals by intravenously injecting an ultrasound contrast agent (usually gas-filled microbubbles), dynamically displaying the distribution and perfusion patterns of neovascularization within plaques in ultrasound images, thus becoming an important tool for assessing plaque stability. Currently, clinicians primarily rely on subjective judgment by manually observing the continuity, uniformity, and intensity of plaque perfusion in CEUS videos. However, this method depends on operator experience and suffers from high subjectivity, poor repeatability, and low efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a CEUS image analysis method based on keyframe segmentation and temporal tracking fusion.

[0005] To address the aforementioned problems, this invention provides a CEUS image analysis method based on keyframe segmentation and temporal tracking fusion, comprising:

[0006] The original CEUS video is preprocessed to obtain a preprocessed frame sequence;

[0007] Extract keyframes from the preprocessed frame sequence to obtain a keyframe sequence;

[0008] The patch region segmentation module inputs keyframe images from the keyframe sequence into the trained segmentation network to segment patch regions and obtain a patch binary mask. ;

[0009] The temporal feature tracking and fusion module is based on the preprocessed frame sequence and the image of each keyframe. The patch binary mask obtained by the segmentation network High-level feature map Obtain the intermediate frames between keyframes. Image The corrected mask ;

[0010] Binary mask based on patchy dimensions of keyframe images Corrected mask for non-keyframes Risk level prediction is performed on all frame images, including keyframes and non-keyframes, and then the final classification of the corresponding patches is obtained based on a preset decision mechanism.

[0011] Furthermore, in the above method, the original CEUS video is preprocessed to obtain a preprocessed frame sequence, including:

[0012] Convert the original CEUS video (non-standard format) to DICOM or AVI format to obtain a resolvable frame sequence;

[0013] Z-score normalization is performed on the parsable frame sequence to obtain a frame sequence with uniform brightness;

[0014] A motion stabilization algorithm based on ORB feature point matching is used to eliminate inter-frame motion artifacts in a frame sequence with uniform brightness, resulting in a stable frame sequence with motion artifacts eliminated, which is then used as the preprocessed frame sequence.

[0015] Furthermore, in the above method, keyframes are extracted from the preprocessed frame sequence to obtain a keyframe sequence, including:

[0016] The first frame of the preprocessed frame sequence is used as the initial keyframe, and then a frame is selected as the keyframe every 10 frames.

[0017] Alternatively, calculate the information entropy difference between adjacent frames in the preprocessed frame sequence, and select key frames from adjacent frames when the difference is greater than a threshold.

[0018] Alternatively, the inter-frame motion can be calculated using optical flow, and the corresponding frame can be selected as a keyframe when the inter-frame motion exceeds a threshold.

[0019] Furthermore, in the above method, the segmentation network includes:

[0020] The multi-scale convolution and feature aggregation module is used to take keyframe images from the keyframe sequence as input features. Based on input features The input feature X is processed in parallel using 3×3, 5×5, and 7×7 convolutional branches, and the features obtained from the parallel processing are concatenated to obtain multi-scale fused features. ;

[0021] A dilated spatial pyramid pooling module is used to process multi-scale fused features with dilated convolutions of dilation rates of 1, 6, 12, and 18. By fusing features from different receptive fields, global and local fused features are obtained. ;

[0022] Channel attention and skip attention gate modules, based on global and local fusion features This yields a patchy binary mask with the same size as the keyframe image. .

[0023] Furthermore, in the above method, the channel attention and skip attention gate modules are used to fuse global and local features. This yields a patchy binary mask with the same size as the keyframe image. ,include:

[0024] Channel attention and skip attention gate modules, based on global and local fusion features Calculate channel weights ;

[0025] Channel weights Features of global and local fusion The channel-wise multiplication yields the channel-enhanced feature F. se ;

[0026] The raw encoder features of each layer in the encoding stage are obtained from the keyframe images. Based on the original encoding end features and the channel-enhanced feature F, which is part of the decoding end feature. se By using additive attention to control the feature fusion of skip connections, a spatial attention weight map is obtained. ;

[0027] Based on spatial attention weight graph and original encoding end features The fusion characteristics of the focused patch region were obtained. ;

[0028] Focusing on the fusion characteristics of patch regions The decoded features are concatenated with the guiding features of the decoding process to update the decoding features, resulting in the final feature map F for the decoding stage. dec ;

[0029] The final feature map F in the decoding stage dec A 1×1 convolution is performed to compress the number of channels to 1, generating a single-channel score map. Sigmoid activation is applied to the values ​​at each position in the single-channel score map, mapping the values ​​to the 0~1 range to obtain a normalized score map. A segmentation threshold is set, and the normalized score map is evaluated pixel-by-pixel: if the score is greater than or equal to the segmentation threshold, it is marked as 1; if the score is less than the segmentation threshold, it is marked as 0, thus obtaining a patchy binary mask with the same size as the keyframe image. .

[0030] Furthermore, in the above method, the overall loss of the segmentation network is: ,in, , For Dice's loss, For binary classification, cross-entropy loss, This is the boundary loss.

[0031] Furthermore, in the above method, the temporal feature tracking and fusion module is based on the preprocessed frame sequence and the image of each key frame. The patch binary mask obtained by the segmentation network High-level feature map Obtain the intermediate frames between keyframes. Image The corrected mask ,include:

[0032] Suppose the preprocessed frame sequence is as follows: The total number of frames is [number], and the set of keyframes is: The image of each keyframe A patch binary mask is obtained through the segmentation network. High-level feature map Among them, high-level feature maps That is, global and local fusion features ;

[0033] Images in adjacent keyframes and First, the displacement field of dense pixels is estimated using the optical flow network RAFT. : ;

[0034] Based on dense pixel displacement field Intermediate frames between adjacent keyframes Image Interpolation yields the image from the keyframe to the intermediate frame. displacement field : ,in, ;

[0035] Based on displacement field mask the keyframe High-level feature map Mapping to intermediate frames To obtain the image of the intermediate frame Forward mask The image of the middle frame propagation characteristics ,in, The optical flow warping operator for bilinear interpolation;

[0036] From the next keyframe Propagate back to intermediate frame To obtain the image of the intermediate frame Reverse mask ;

[0037] The intermediate frame is obtained by weighted summation based on confidence level. initial fusion mask : Among them, confidence weight and Adaptive calculation based on optical flow consistency and image similarity:

[0038] , Similarly;

[0039] The corrected mask is obtained using a lightweight U-Net mask correction module. , The input includes intermediate frames. Image Initial mask With propagation characteristics ;

[0040] Based on intermediate frames Image and the previous intermediate frame -1 image Calculate and define the confidence index : ;

[0041] when or This triggers a rollback, changing the patch binary mask image. As the updated and revised mask Otherwise, keep the original modified mask. .

[0042] Furthermore, in the above method, the training loss function of the lightweight U-Net-type mask correction module includes cross-entropy and boundary consistency loss. : ,in, For real label masking, For gradient operators, .

[0043] Furthermore, in the above method, during the training of the temporal feature tracking and fusion module, temporal consistency constraints are applied. Cross-entropy and boundary consistency loss Together, they form the overall training loss function, which is used for backpropagation to optimize the parameters of the entire temporal feature tracking and fusion module; wherein, based on intermediate frames Image The corrected mask The previous intermediate frame -1 image The corrected mask Compute timing consistency constraints : .

[0044] Furthermore, in the above method, a patchy binary mask with consistent keyframe image size is used. Corrected mask for non-keyframes Risk level prediction is performed on all frame images, including keyframes and non-keyframes. Then, based on a preset decision mechanism, the final classification of the corresponding patches is obtained, including:

[0045] For each frame of the image and patch binary mask Or a modified mask Perform convolutional encoding to extract frame-level embeddings and obtain frame-level feature vectors. : ,in, It is a frame-level feature vector;

[0046] Frame-level feature sequences A Transformer encoder is used to model the input temporal features, resulting in a feature sequence that fuses spatiotemporal information. ,in, To enable learnable positional encoding, the Transformer layer uses a multi-head self-attention mechanism;

[0047] Feature sequences based on fusion of spatiotemporal information The time-series representation is obtained through global pooling. : ;

[0048] Based on time-series representation Ultimately, the risk level is output through two layers of sensors. : , ( ) is the activation function. This is the weight matrix of the first-layer perceptron, used for temporal representation. Perform a linear transformation; This is the weight matrix of the second-layer perceptron, used to perform a quadratic linear transformation on the activated features;

[0049] loss function A combination of cross-entropy and focus loss is used: ,in, Weighting coefficients , For cross-entropy loss, Loss of focus;

[0050] Risk level prediction results across all frames in the entire CEUS video sequence Based on a pre-defined decision-making mechanism, the final classification of the corresponding plaque is obtained.

[0051] According to another aspect of the present invention, a computer-readable storage medium is also provided, having stored thereon computer-executable instructions, wherein when executed by a processor, the computer-executable instructions cause the processor to perform the method of any of the above embodiments.

[0052] According to another aspect of the present invention, a calculator device is also provided, comprising:

[0053] Processor; and

[0054] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any of the above embodiments.

[0055] Compared with existing technologies, the CEUS (Contrast-Enhanced Ultrasound) image analysis method based on keyframe segmentation and temporal tracking fusion of the present invention integrates video preprocessing (including Z-Score brightness normalization and ORB feature motion stabilization), keyframe extraction (fixed interval / adaptive strategy), plaque segmentation (MS-Att U-Net), temporal feature fusion (bidirectional optical flow propagation + confidence weighting), and multi-frame risk decision (highest risk / weighted average / confidence fusion) to form a complete risk grading process that can be used for automated grading of carotid plaque rupture risk.

[0056] Specifically, the carotid plaque CEUS video analysis method of the present invention, based on keyframe segmentation and temporal tracking fusion, is based on a CEUS video plaque neovascularization classification framework that integrates keyframe screening, segmentation and temporal fusion. It integrates video preprocessing (including Z-Score brightness normalization and ORB feature motion stabilization), keyframe extraction (fixed interval / adaptive strategy), plaque segmentation (MS-Att U-Net), temporal feature fusion (bidirectional optical flow propagation + confidence weighting) and multi-frame classification decision (highest level / weighted average / confidence fusion), forming a complete automated classification process for intraplaque neovascularization, which can be used to determine the stability and rupture risk of carotid plaques.

[0057] The present invention has the following beneficial effects:

[0058] 1) Significantly improved efficiency: Keyframe filtering (selecting the first frame and every 10 subsequent frames, selecting all if there are fewer than 10 frames) reduces the number of input frames by about 90%, which greatly reduces the number of frames the model needs to calculate when processing video, shortening the inference time to about 1 / 5 of the original time. This allows for faster provision of risk grading results to clinicians, such as the grading results of neovascularization within plaques. It also supports mainstream CEUS video formats such as DICOM and AVI, and is compatible with a frame rate range of 15-60fps, demonstrating strong compatibility.

[0059] 2) Enhanced temporal sensitivity: The temporal feature modeling module (optional bidirectional LSTM, Transformer, TCN) captures the full-process characteristics of blood perfusion (such as capturing the full-process characteristics of ultrasound microbubble contrast agent perfusion in plaque neovascularization). The temporal features extracted from the keyframe sequence can reflect the dynamic changes of microblood flow perfusion in plaque from start to finish. Compared with analyzing only a single frame or short segment, it can capture dynamic information related to plaque stability more comprehensively and accurately, and improve the accuracy of risk assessment, such as improving the accuracy of neovascularization classification in plaque.

[0060] 3) Stable grading results: By using a decision-making mechanism that predicts the risk level of all frames and takes the highest risk (or weighted average, confidence fusion), such as predicting the neovascularization level of all frames and taking the highest level, the impact of random fluctuations in the prediction results of a single frame or some frames on the final grading is avoided, making the grading results more stable and reliable.

[0061] 4) Strong clinical interpretability: Since risk level prediction, such as neovascularization grading prediction, is performed for each frame, high-risk key frames or non-key frames, such as high-level ones, can be located. The model is presented to doctors by labeling frame indexes, generating perfusion feature heatmaps (highlighting high-risk areas), outputting frame-level risk scores (0-100 points) or outputting frame-level neovascularization grades within plaques (grade 0, grade 1, grade 2), etc., which helps doctors understand the specific image frames and perfusion features on which the model is based, and improves clinical trust in the model.

[0062] 5) Good versatility: The core of the method is keyframe extraction, temporal fusion and multi-frame risk decision-making, which can be extended to CEUS image analysis of organs such as liver and kidney for risk classification of related lesions in these organs. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a CEUS image analysis method based on keyframe segmentation and temporal tracking fusion according to an embodiment of the present invention. Detailed Implementation

[0064] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] With the development of artificial intelligence and medical imaging fusion technology, automatic plaque recognition and risk classification based on deep learning has gradually become a research hotspot. Current research uses 2D convolutional neural networks (CNNs) for plaque region segmentation in CEUS image frames, or utilizes 3D convolutional and recurrent neural networks (RNNs, LSTMs) for classifying short temporal segments. However, these methods have significant problems when processing long video sequences:

[0066] 1) The number of input frames for the model is limited by video memory, making it impossible to efficiently process the entire sequence;

[0067] 2) Insufficient utilization of time-series information easily leads to the loss of dynamic changes in plaque perfusion characteristics;

[0068] 3) A large number of redundant frames in the video increase computational costs and affect the stability of risk classification.

[0069] Therefore, how to effectively screen key frames and fuse spatial and temporal features while ensuring the integrity of temporal information, so as to achieve objective quantitative classification of carotid plaque rupture risk, has become a technical problem that urgently needs to be solved in this field.

[0070] Currently, the publicly available CEUS plaque analysis techniques mainly include the following three categories:

[0071] (1) Static frame segmentation method

[0072] Researchers used U-Net or ResNet architectures to segment patch regions in single-frame CEUS images, and then performed risk estimation based on regional average brightness, texture features, and other factors. This approach ignores the dynamic changes in blood flow over time and can only perform static assessments.

[0073] (2) Short-time modeling method

[0074] Some studies divide CEUS videos into fixed-length segments (e.g., 10–20 frames) and use 3D-CNN or bidirectional LSTM to extract short-term features. Other researchers have proposed using 3D convolutional networks to predict risks in local temporal segments. This method models local temporal relationships well, but struggles to cover the entire video information.

[0075] (3) End-to-end video classification method

[0076] Recent research has attempted to directly input complete CEUS video sequences and learn global temporal relationships through a Video Transformer structure. While this method can theoretically obtain optimal global features, it faces significant computational limitations in practical applications and is insensitive to inter-frame redundancy, leading to a decline in the model's generalization ability.

[0077] The above methods generally suffer from the following drawbacks:

[0078] Insufficient use of time series: Most models only analyze short, continuous segments and cannot capture the dynamic evolution of the entire plaque perfusion process.

[0079] Excessive frame redundancy: CEUS video frame rate is typically 25–30 fps, with minimal differences between adjacent frames. Inputting all frames results in excessive computational overhead and feature redundancy.

[0080] Unstable risk assessment: Fluctuations in the results of single frames or short segments lead to unstable final classification results.

[0081] Lack of keyframe mechanism: Failure to distinguish critical moments of plaque perfusion (such as peak, initial perfusion, and washout phase).

[0082] Lack of interpretability: Model results are difficult to correlate with specific frames or physiological phenomena, reducing clinical credibility.

[0083] To address the aforementioned shortcomings, this invention aims to provide a CEUS carotid plaque rupture risk classification method that integrates keyframe extraction, plaque segmentation, and temporal feature tracking, thereby solving the following problems: computational efficiency issues in full-video analysis; insufficient utilization of temporal information; and fluctuations and poor interpretability of risk classification results. This invention significantly reduces input redundancy through a keyframe filtering mechanism; extracts dynamic change patterns of plaques through a temporal feature fusion module (including optional structures such as bidirectional LSTM); and finally, performs comprehensive risk assessment based on multi-frame inference results, improving classification accuracy and stability.

[0084] like Figure 1 As shown, this invention provides a CEUS image analysis method based on keyframe segmentation and temporal tracking fusion, comprising:

[0085] Step S1: Preprocess the original CEUS video to obtain a preprocessed frame sequence;

[0086] Specifically, this invention can be a carotid plaque CEUS video analysis method based on keyframe segmentation and temporal tracking fusion. Carotid plaque rupture is a major pathological basis for ischemic stroke. Studies have shown that intraplaque neovascularization is an important indicator of unstable plaques and is closely related to plaque rupture. Traditional two-dimensional grayscale ultrasound imaging can be used to detect plaque morphology and size, but cannot detect intraplaque neovascularization. Carotid plaque CEUS observes the perfusion characteristics of microbubble contrast agents throughout the entire process of microbubble perfusion in intraplaque neovascularization to evaluate the density and distribution of neovascularization, thereby diagnosing the grading of intraplaque neovascularization and plaque stability. Currently, this examination mainly relies on subjective judgment by ultrasound physicians through visual observation, which is highly dependent on physician experience and suffers from problems such as strong subjectivity, poor repeatability, and low efficiency.

[0087] Preferably, step S1 includes:

[0088] Step S11: Convert the original CEUS video in non-standard format to DICOM or AVI format to obtain a parseable frame sequence;

[0089] Step S12: Z-score normalization is performed on the parsable frame sequence to obtain a frame sequence with uniform brightness.

[0090] Step S13: Based on the motion stabilization algorithm of ORB feature point matching, eliminate inter-frame motion artifacts in the frame sequence with uniform brightness to obtain a stable frame sequence with motion artifacts eliminated, which is used as the preprocessed CEUS stable frame sequence.

[0091] Here, video frame sequences can be extracted from the original CEUS video. Z-Score normalization is used to normalize the brightness to reduce brightness differences under different devices or imaging conditions. A motion stabilization algorithm (motion compensation) based on ORB feature point matching is used to eliminate inter-frame motion artifacts caused by patient movement, resulting in a preprocessed stable CEUS frame sequence. In addition, if the original video is in a non-standard format, the frame sequence needs to be converted to DICOM or AVI format first to ensure that the frame data can be parsed.

[0092] Step S2: Extract keyframes from the preprocessed CEUS stable frame sequence to obtain a keyframe sequence. A keyframe can refer to the first frame of the video and subsequent frames taken at intervals of 10 frames (this interval was determined through grid search optimization of 500 sets of clinical CEUS videos (frame rate 25-30fps), achieving an optimal balance between redundancy elimination rate and temporal information preservation rate). Specifically, let the total number of video frames be... If the frame index starts from 1, then the keyframe set... ,in satisfy If the total number of video frames Then all frames are directly used as keyframes to form a keyframe sequence.

[0093] Here, keyframe selection strictly follows the fixed-interval sampling principle, that is, the first frame of the video is determined as the initial keyframe, and then one frame is selected as the keyframe every 10 frames (this interval was determined through grid search optimization of 500 sets of clinical CEUS videos (frame rate 25-30fps), achieving an optimal balance between redundancy elimination rate and temporal information preservation rate). For example, if the video has 101 frames, then the keyframes are frames 1, 11, 21, ..., 101, a total of 11 frames; if the total number of frames in the video is... In this case, all frames are used as keyframes. This fixed-interval approach ensures uniform coverage of the video timeline, reducing redundant frames while guaranteeing the capture of information from different stages of patch perfusion.

[0094] Optional strategies: Adaptive keyframe selection can be used instead of fixed-interval sampling, including:

[0095] 1) Inter-frame entropy method: Calculate the information entropy difference between adjacent frames in the preprocessed frame sequence. When the difference is greater than the threshold (the threshold is determined to be 0.15 through statistical analysis of 300 sets of clinical data), it is selected as a key frame. Specifically, when the information entropy difference between adjacent frames t and t+1 is greater than the threshold, the next frame (frame t+1) corresponding to the difference can be used as a key frame.

[0096] 2) Motion Quantity Method: Inter-frame motion (average pixel displacement) is calculated using optical flow. When the inter-frame motion exceeds a threshold (defined as 5 pixels based on statistical analysis of 300 sets of clinical data), the corresponding frame is selected as the keyframe. Specifically, consistent with the inter-frame entropy method, the frame following the motion (frame t+1) is typically selected as the keyframe.

[0097] Step S3: The patch region segmentation module inputs the keyframe images from the keyframe sequence into the trained segmentation network to segment the patch regions, obtaining a patch mask, i.e., a patch binary mask. The segmentation network takes keyframe images as input and outputs a binary mask image of the same size as the input keyframe images, which serves as the patch features obtained from the segmentation. In the binary mask image, the foreground represents the patch region and the background represents the non-patch region.

[0098] This invention proposes an improved U-Net (MS-Att U-Net) network with multi-scale context aggregation and attention mechanisms as a segmentation network for accurate segmentation of CEUS patch regions (the core difference from the original U-Net lies in the introduction of multi-scale convolutional branches, ASPP modules, SE channel attention, and skip attention gates). Key innovations include: multi-scale convolutional coding, multi-scale dilated spatial pyramid pooling (ASPP), channel attention (SE), and skip attention gate mechanisms.

[0099] Preferably, the segmentation network includes: a multi-scale convolution and feature aggregation module, a dilated spatial pyramid pooling module, a channel attention and skip attention gate module, and a joint loss function module.

[0100] Step S3 includes:

[0101] Step S31, the multi-scale convolution and feature aggregation module, uses the keyframe images in the keyframe sequence as input features. The multi-scale convolution and feature aggregation module is based on the input features. The input feature X is processed in parallel using 3×3, 5×5, and 7×7 convolutional branches, and the features obtained from the parallel processing are concatenated to obtain multi-scale fused features. .

[0102] Here, the multi-scale convolution and feature aggregation module introduces multi-scale convolution branches at each encoding stage to enhance the perception of patches and perfusion textures at different scales; keyframe images in the keyframe sequence are used as input features. Based on input features The formula for parallel convolution aggregation is: ,in Indicates feature concatenation operation; This represents convolution.

[0103] Step S32, the dilated spatial pyramid pooling (ASPP) module processes multi-scale fused features using dilated convolutions with dilation rates of 1, 6, 12, and 18. By fusing features from different receptive fields, global and local fused features are obtained. ;

[0104] Introduce different expansion rates at the bottleneck layer The dilated convolutional branch (the dilation rate combination was determined through comparative experiments to optimally cover global and local features of the plaque perfusion region) is used to capture global and local fused features under different receptive fields. : This module can effectively enhance the response capability to large-scale perfusion changes.

[0105] To highlight the salient features of the patch region, a Squeeze-and-Excitation (SE) submodule and an Attention Gate submodule are introduced into the channel attention and jump attention gate modules.

[0106] Step S33, Channel Attention and Skip Attention Gating Module, based on global and local fusion features Calculate channel weights ; channel weight Features of global and local fusion The channel-wise multiplication yields the channel-enhanced feature F. se Based on the keyframe images, the original coding end features of each layer in the coding stage are obtained. Based on the original encoding end features and the channel-enhanced feature F, which is part of the decoding end feature. se By using additive attention to control the feature fusion of skip connections, a spatial attention weight map is obtained. Based on spatial attention weight graph and original encoding end features The fusion characteristics of the focused patch region were obtained. The focus will be on the fusion characteristics of the patch regions. The decoded features are concatenated with the guiding features of the decoding process to update the decoding features, resulting in the final feature map F for the decoding stage. dec (Size restored to match the input keyframe image); the final feature map F from the decoding stage... dec

[0107] Perform a 1×1 convolution to compress the number of channels to 1, generating a single-channel score map. Apply Sigmoid activation to the values ​​at each position in the single-channel score map, mapping the values ​​to the 0~1 range to obtain a normalized score map. Set a segmentation threshold (e.g., 0.5), and perform a pixel-by-pixel judgment operation on the normalized score map: if the score is ≥0.5, mark it as 1 (patch region); if the score is <0.5, mark it as 0 (background region), to obtain a patch binary mask with the same size as the keyframe image. .

[0108] Preferably, the SE module calculates channel weights using global average pooling. : ,in, Indicates global average pooling. It is the ReLU activation function. This is the Sigmoid activation function.

[0109] Here, The final output is the channel weights, with each channel corresponding to a weight value between 0 and 1;

[0110] : Input features Perform global average pooling to compress the spatial dimensions of features (such as width and height) to 1, retaining only the statistical information of the channel dimension;

[0111] , : The weight matrix of the fully connected layer, where, Responsible for compressing channel dimensions, Responsible for restoring the channel dimension;

[0112] ( ): ReLU activation function (introduces non-linearity into the network, enhancing its expressive power);

[0113] ( The Sigmoid activation function maps the output to a range of 0 to 1, thus obtaining the channel weight coefficients.

[0114] Preferably, the Attention Gate module controls skip connections through additive attention: spatial attention weight graph. The fusion output is the fusion feature of the focused patch region. , This is for element-wise multiplication.

[0115] in, This is a spatial attention weight map, with dimensions similar to the original encoder features. Consistent, with values ​​between 0 and 1;

[0116] These are the raw skip features at the encoding end, originating from a certain layer in the U-Net encoding stage;

[0117] The guiding features at the decoding end include the channel-enhanced feature F. se Specifically, the enhanced channel feature F se As the initial input features for the decoding stage, they are first upsampled (e.g., by transposed convolution) to recover some spatial dimensions. Then, a 1×1 or 3×3 convolution is used for feature transformation to match the number of channels and semantic information of the current decoding level, resulting in the decoding features of the initial level of the decoding stage. For subsequent levels in the decoding stage, the guiding feature g is generated from the decoding features of the previous level through upsampling and a 1×1 or 3×3 convolution, and always carries the channel-enhanced feature F. se The channel enhancement information; the final output is the decoding feature of the current level obtained through the above steps, which is the guiding feature g required by the skip attention gate.

[0118] To act on the original encoded features The linear transformation weight matrix aims to transform the original encoded features. Mapping to guiding features Same feature dimensions;

[0119] The linear transformation weight matrix acting on the guiding feature g;

[0120] ( ) is the activation function, which introduces non-linearity and filters out negative values;

[0121] This is a bias term used to adjust the output of the linear transformation;

[0122] The final linear projection weight matrix maps the fused features into a prototype of single-channel attention weights;

[0123] ( () is the Sigmoid activation function, which compresses the values ​​of the weight prototype to the range of 0 to 1.

[0124] Preferably, the segmentation network employs a multi-task supervision mechanism, including main segmentation loss and boundary auxiliary loss.

[0125] The comprehensive loss is defined as: ,in (Weights are determined through cross-validation to balance segmentation accuracy and boundary integrity). For Dice's loss, For binary classification, cross-entropy loss, This is the boundary loss.

[0126] Boundary supervision signals are generated by the morphological gradient of the mask and are used to improve the clarity and consistency of patch boundaries.

[0127] Specifically, the inputs to the joint loss function (training phase) are: normalized score map, ground truth label mask, and mask morphological gradient (boundary supervision signal).

[0128] The source of the ground truth mask: a gold standard mask manually annotated by clinicians, which is part of the training dataset. Doctors manually delineate the true boundaries of patches on CEUS images as a reference benchmark for model learning.

[0129] The morphological gradient of the mask (boundary supervision signal) comes from derived data obtained by performing morphological gradient operations on the ground truth labeled mask. The morphological gradient highlights the edge contours of the mask, aiming to make the model focus more on the boundary segmentation accuracy of patches during training and reduce edge blurring.

[0130] Operation: Calculate Dice, BCE, and boundary loss, and then perform weighted fusion;

[0131] Output: Total loss (used to optimize network parameters).

[0132] During the training phase, forward inference yields the patch mask output by the network, and the total loss is calculated by combining the real label mask and the mask morphological gradient.

[0133] This total loss is passed to the optimizer (such as Adam or SGD). The optimizer backpropagates and updates the parameters of all modules in MS-Att U-Net based on the magnitude of the loss and the direction of the gradient—including the weights and biases of multi-scale convolution, ASPP, SE channel attention, skip attention gate, etc.

[0134] After the parameters are updated, proceed to the next round of forward inference and repeat the above process until the loss converges and the model performance meets the target.

[0135] Training phase: The segmentation network outputs a probability mask (pixel value range 0~1), representing the probability that each pixel belongs to a patch. The loss is then calculated using the ground truth labels, and parameters are optimized through backpropagation.

[0136] Inference phase: A threshold (usually 0.5) is set for the probability mask. Pixels with a probability ≥ 0.5 are identified as patches (assigned a value of 1, foreground), and pixels with a probability < 0.5 are identified as background (assigned a value of 0), thus obtaining the binary mask.

[0137] The size of this binary mask is exactly the same as the input CEUS keyframe image, and it can accurately correspond to the patch positions in the original image.

[0138] Step S4, the temporal feature tracking and fusion module, based on the preprocessed CEUS stable frame sequence and the image of each keyframe in the preprocessed CEUS stable frame sequence. The patch binary mask obtained by the segmentation network High-level feature map Obtain the intermediate frames between keyframes. Image The corrected mask .

[0139] In this context, within the CEUS (Contrast-Enhanced Ultrasound) sequence, keyframes (such as frames 1, 11, and 21) typically correspond to significant phases of contrast agent perfusion, while adjacent frames (such as frames 2–10 and 12–20) still contain rich local dynamic information. To ensure temporal continuity and computational efficiency, this invention proposes a keyframe-guided temporal mask propagation and feature tracking method, achieving semantic consistency propagation and spatiotemporal feature fusion from keyframes to intermediate frames.

[0140] Preferably, step S4 includes:

[0141] Step S41, Problem Definition and Objective:

[0142] Suppose the preprocessed CEUS stable frame sequence is as follows: The total number of frames is [number], and the set of keyframes is: The image of each keyframe A patch binary mask is obtained through the segmentation network. High-level feature map Among them, high-level feature maps That is, global and local fusion features The goal is to find the intermediate frame between keyframes. , Generate the corresponding prediction mask. With features This enables cross-time tracking from keyframes to intermediate frames.

[0143] Step S42, Keyframe Inter-frame Optical Flow Estimation and Mask Propagation:

[0144] Step S421, in the images of adjacent keyframes and First, the displacement field of dense pixels is estimated using the optical flow network RAFT. : ;

[0145] Step S422, based on dense pixel displacement field Intermediate frames between adjacent keyframes Image Interpolation yields the image from the keyframe to the intermediate frame. displacement field : ,in, ;

[0146] Step S423, based on displacement field mask the keyframe High-level feature map Mapping to intermediate frames To obtain the image of the intermediate frame Forward mask The image of the middle frame propagation characteristics ,in, The optical flow warping operator for bilinear interpolation.

[0147] This step is equivalent to semantic mask propagation, which "guides intermediate frames from keyframes", enabling intermediate frames to have preliminary semantic segmentation estimates.

[0148] Step S43, bidirectional tracking and feature fusion (keyframe ↔ non-keyframe):

[0149] To reduce one-way propagation error, this invention introduces a two-way time-series propagation strategy:

[0150] 1) Forward propagation: from the previous keyframe Propagated to intermediate frames Intermediate frames Forward mask ;

[0151] 2) Backpropagation: from the next keyframe Propagate back to intermediate frame To obtain the image of the intermediate frame Reverse mask ;

[0152] The final fusion result is obtained by weighted summation based on confidence level to obtain the intermediate frames. initial fusion mask : Among them, confidence weight and Adaptive calculation based on optical flow consistency and image similarity:

[0153] , Similarly.

[0154] This mechanism can effectively reduce the accumulated bias of one-sided propagation, so that the intermediate frames between keyframes are spatially consistent with the keyframes on both sides.

[0155] Step S44, Mask Refiner:

[0156] Initial mask after fusion Errors may still occur at the boundaries, therefore a lightweight U-Net-type mask correction module was designed to obtain the corrected mask. , The input includes intermediate frames. Image Initial mask With propagation characteristics .

[0157] The lightweight U-Net-type mask correction module adopts a 3-layer convolutional encoder + deconvolutional decoder structure, and outputs a refined mask, which is the corrected mask. .

[0158] The training loss function for the lightweight U-Net-type mask correction module includes cross-entropy and boundary consistency loss: ,in, For real label masking, For gradient operators, (Experiments have verified that the mask segmentation accuracy and boundary smoothness can be balanced).

[0159] Specifically, the input to the lightweight U-Net-type mask correction module is: intermediate frames. Image Initial mask Characteristics of transmission And the real label mask Gt (training phase);

[0160] Operation: Using the lightweight U-Net module, input "intermediate frames" + Initial Mask +Propagation characteristics Output the corrected mask. Use cross-entropy + boundary consistency loss ( Supervised training of a lightweight U-Net-type mask correction module to optimize mask boundary accuracy;

[0161] Output: Intermediate frames Image The corrected mask .

[0162] Step S5, Timing Consistency and Rollback Strategy:

[0163] To prevent the accumulation of propagation errors, based on intermediate frames Image The corrected mask The previous intermediate frame -1 image The corrected mask Compute timing consistency constraints : .

[0164] Temporal consistency constraints during training of the temporal feature tracking and fusion module The cross-entropy and boundary consistency loss from step 4 will be combined. Together with (mask correction loss), they form the overall training loss function, which is used for backpropagation to optimize the parameters of the entire temporal feature tracking and fusion module (including the weights of components such as optical flow propagation, bidirectional fusion, and mask correction).

[0165] Its function is to enable the model to "learn to maintain the consistency of inter-frame masks" during training, so that it can naturally output more temporally coherent masks during the inference stage and reduce the probability of error accumulation.

[0166] During the reasoning stage, It will not be calculated, only by confidence level. and Indicators are used to determine whether a rollback has been triggered.

[0167] Responsible for single-frame accuracy: making the mask itself more accurate for each intermediate frame;

[0168] Responsible for inter-frame stability: making the mask transitions between adjacent frames more seamless and reducing the accumulation of timing errors.

[0169] This is to enable the refined mask in subsequent frames. More coherent. During the training phase, Will make peace Together, they constitute the overall loss and participate in backpropagation to update the parameters of the entire temporal tracking module (including the weights of optical flow estimation, bidirectional fusion, and mask correction modules). This allows the model to "learn" the continuity rules of inter-frame masks. For example, the mask of the current frame should be as consistent as possible with the mask propagated from the previous frame. In this way, during subsequent training iterations or inference, the newly generated refined mask Mt will not have the problem of drastic inter-frame jumps, and the temporal coherence will be significantly improved.

[0170] Based on intermediate frames Image and the previous intermediate frame -1 image Calculate and define the confidence index : . For the previous frame and the previous intermediate frame -1 image Perform optical flow-guided deformation operations.

[0171] when ( Based on statistical analysis of 300 sets of clinical data, a balance was achieved between the false trigger rate and the false negative rate. ( When (as above), the system triggers a keyframe fallback: directly runs the full segmentation network (MS-Att U-Net) on that frame to reset the tracking error.

[0172] when or If this happens, a rollback is triggered, the patch region segmentation module is run, and the output patch binary mask image of the intermediate frame is used as the updated and corrected mask. Otherwise, keep the original modified mask. .

[0173] Step S6, Risk grading inference module, based on patch binary masks with consistent keyframe image size. Corrected mask for non-keyframes The risk level is predicted for all frames, including key frames and non-key frames. Then, based on a preset decision mechanism, such as the highest risk level among all frames, the final classification of the corresponding patch is obtained.

[0174] To comprehensively assess the risk of patch damage in CEUS images from a spatiotemporal perspective, this invention designs a CNN-Transformer hybrid structure (CT-Risk) to achieve the fusion of spatial feature extraction and temporal dynamic modeling (the CNN backbone can be either ResNet50 or EfficientNet-B2, both of which have demonstrated excellent feature extraction capabilities in the CEUS dataset through experiments).

[0175] Specifically, the risk grading inference module can be based on a patch binary mask with consistent keyframe image size. Corrected mask for non-keyframes The system predicts the neovascularization level for all frames, including keyframes and non-keyframes, and then obtains the final classification of neovascularization within the plaque based on a pre-defined decision mechanism.

[0176] Step S61, Spatial Feature Extraction (CNN Backbone):

[0177] For each frame of the image and patch binary mask Or a modified mask (Can be concatenated into a dual-channel input) Perform convolutional encoding, extract frame-level embeddings, and obtain frame-level feature vectors. : ,in, This is a frame-level feature vector.

[0178] Here, CNN backbone networks such as ResNet50 / EfficientNet-B2 can be used to perform convolutional encoding on the input to obtain frame-level feature vectors. Each frame corresponds to a feature of dimension d, thus obtaining;

[0179] Step S62, Temporal Feature Modeling (Transformer Encoder):

[0180] Frame-level feature sequences A Transformer encoder is used to model the input temporal features, resulting in a feature sequence that fuses spatiotemporal information. ,in, To enable learnable positional encoding, the Transformer layer uses a multi-head self-attention mechanism.

[0181] Optional Structure: The Transformer encoder for temporal feature modeling can be replaced with a bidirectional LSTM or a TCN (Temporal Convolutional Network). The bidirectional LSTM is suitable for capturing short temporal dependencies (fast inference speed), while the TCN is suitable for capturing long temporal dependencies (higher accuracy) through dilated convolutions. The choice can be made based on hardware computing power and accuracy requirements.

[0182] Step S63, Classification head and loss function:

[0183] Feature sequences based on fusion of spatiotemporal information The time-series representation is obtained through global pooling. : ;

[0184] Based on time-series representation Ultimately, the risk level is output through two layers of sensors. : , ( ) is the activation function. This is the weight matrix of the first-layer perceptron, used for temporal representation. Perform a linear transformation; This is the weight matrix of the second-layer perceptron, used to perform a quadratic linear transformation on the activated features;

[0185] loss function A combination of cross-entropy and focus loss is used: Among them, the weighting coefficient (Experimental verification shows that this method can effectively balance the problem of a low proportion of high-risk samples.) For cross-entropy loss, The loss is the focus.

[0186] The risk grading inference module can capture the dynamic perfusion pattern of CEUS patches over time, enabling accurate grading and prediction of damage risk.

[0187] Risk level prediction results across all frames in the entire CEUS video sequence The risk level with the highest value is directly selected as the final risk level for the entire sequence.

[0188] Here, the neovascularization grade prediction results for all frames can be iterated. Based on a pre-defined decision-making mechanism, the final classification of neovascularization within the plaque is obtained.

[0189] Optional decision-making mechanisms: alternatives to the "highest risk" decision-making principle, including:

[0190] 1) Weighted average: The weight of key frames is set to 0.7, and the weight of non-key frames is set to 0.3 (determined through clinical expert consensus to highlight the perfusion characteristic value of key frames). The weighted sum of the risk levels of all frames is then rounded down.

[0191] 2) Confidence fusion: Weighted fusion is performed based on the confidence level (maximum probability value output by Softmax) of the risk prediction for each frame. Higher confidence levels result in higher weights. The formula is as follows: ,in For the first Frame prediction confidence.

[0192] The present invention provides a CEUS (Contrast-Enhanced Ultrasound) image analysis method based on keyframe segmentation and temporal tracking fusion. This method, based on a comprehensive CEUS video risk grading framework that integrates keyframe selection, segmentation, and temporal fusion, incorporates video preprocessing (including Z-Score brightness normalization and ORB feature motion stabilization), keyframe extraction (fixed interval / adaptive strategy), plaque segmentation (MS-Att U-Net), temporal feature fusion (bidirectional optical flow propagation + confidence weighting), and multi-frame risk decision-making (highest risk / weighted average / confidence fusion). This forms a complete risk grading process that can be used for automated grading of carotid artery plaque rupture risk.

[0193] Specifically, this invention provides a video analysis method for carotid plaque contrast-enhanced ultrasound (CEUS) based on keyframe segmentation and temporal tracking fusion. Based on the overall framework of CEUS video neovascularization classification, which integrates keyframe screening, segmentation and temporal fusion, it integrates video preprocessing, keyframe extraction, plaque segmentation, temporal feature fusion and multi-frame classification decision-making to form a complete intraplaque neovascularization classification process, which can be used for automated classification of intraplaque neovascularization and plaque stability assessment.

[0194] This invention is based on a keyframe extraction mechanism that selects the first frame of the video and every 10 subsequent frames (all frames are selected if the number of frames is less than 10, and the interval is determined by grid search optimization), and an optional adaptive keyframe selection strategy based on inter-frame entropy (threshold 0.15) and motion (threshold 5 pixels). This type of mechanism is simple and can effectively cover the video temporal sequence, reduce redundant frames, and retain key temporal information.

[0195] This invention improves the patch segmentation structure of U-Net (MS-Att U-Net), specifically including multi-scale convolutional branches (3×3 / 5×5 / 7×7 convolutions) and dilation rate. The ASPP module, SE channel attention and jump attention gates, and ( The joint loss function is used to achieve accurate segmentation of patch regions.

[0196] The keyframe-guided temporal mask propagation and feature tracking method of the present invention includes adjacent keyframe RAFT optical flow estimation, bidirectional propagation (forward / backward), and confidence-weighted fusion. (Calculation method), lightweight mask correction module (3-layer convolutional encoder + deconvolutional decoder), and based on or The rollback strategy ensures the accuracy and stability of timing tracking.

[0197] The CNN-Transformer hybrid risk classification structure (CT-Risk) of this invention, along with the optional bidirectional LSTM / TCN temporal modeling module and multi-frame risk decision mechanism (highest risk / weighted average / confidence fusion), enables accurate and stable classification of patch damage risk.

[0198] This invention provides an efficient, stable, and interpretable method for predicting plaque rupture risk in CEUS sequences (supporting DICOM / AVI format, 15-60fps frame rate, and capable of locating high-risk frames and generating heatmaps), offering a powerful tool for clinical carotid plaque rupture risk assessment, and can be extended to CEUS image analysis of organs such as the liver and kidneys.

[0199] The keyframe selection strategy of this invention can be replaced by sampling at fixed intervals (10 frames after the first frame, and all frames are selected when the number of frames is less than 10). Alternatively, keyframes can be selected adaptively using inter-frame entropy (calculating the information entropy difference between adjacent frames, and selecting a keyframe when the difference is greater than a threshold of 0.15) or motion (calculating inter-frame motion using optical flow, and selecting a keyframe when the motion is greater than a threshold of 5 pixels) to more flexibly select representative frames according to the characteristics of different videos.

[0200] The temporal encoder of this invention is replaceable: the Transformer in the temporal feature modeling module can be replaced with a bidirectional LSTM (which uses a bidirectional recurrent structure to capture short temporal dependencies, has a fast inference speed, and is suitable for low-computing-power devices), TCN (temporal convolutional network, which captures long temporal dependencies through dilated convolutions, has higher accuracy, and is suitable for high-computing-power scenarios), etc. Different temporal encoders have their own characteristics in terms of the ability to capture temporal features and computational efficiency, and can be selected according to actual needs.

[0201] The risk decision-making mechanism of this invention is variable: the highest risk principle can be replaced by a weighted average (key frames are assigned a weight of 0.7, and non-key frames are assigned a weight of 0.3, determined through clinical expert consensus) or confidence fusion (combining the confidence level of risk prediction for each frame, with higher confidence levels having higher weights, as shown in the formula). ( ), to adapt to different clinical decision-making preferences.

[0202] The feature fusion method of the present invention can be adjusted: temporal features and spatial features can be fused using a multimodal attention mechanism. By learning the weights of temporal features and spatial features through the attention mechanism (such as assigning higher attention weights to spatial features during the peak stage of plaque perfusion), more effective multimodal feature fusion can be achieved, further improving the accuracy of risk classification.

[0203] According to another aspect of the present invention, a computer-readable storage medium is also provided, having stored thereon computer-executable instructions, wherein when executed by a processor, the computer-executable instructions cause the processor to perform the method of any of the above embodiments.

[0204] According to another aspect of the present invention, a calculator device is also provided, comprising:

[0205] Processor; and

[0206] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any of the above embodiments.

[0207] The present invention has the following beneficial effects:

[0208] 1) Significantly improved efficiency: Keyframe filtering (selecting the first frame and every 10 subsequent frames, selecting all if there are fewer than 10 frames) reduces the number of input frames by about 90%, which greatly reduces the number of frames the model needs to calculate when processing videos, shortening the inference time to about 1 / 5 of the original, and enabling faster provision of risk grading results for clinicians; it also supports mainstream CEUS video formats such as DICOM and AVI, and is compatible with a frame rate range of 15-60fps, with strong compatibility.

[0209] 2) Enhanced temporal sensitivity: The temporal feature modeling module (optional bidirectional LSTM, Transformer, TCN) captures the features of the entire blood perfusion process. The temporal features extracted from the keyframe sequence can reflect the dynamic changes of plaque perfusion from start to finish. Compared with analyzing only a single frame or short segment, it can capture dynamic information related to plaque stability more comprehensively and accurately, and improve the accuracy of risk assessment.

[0210] 3) Stable classification results: By using a decision-making mechanism that predicts the risk level of all frames and takes the highest risk (or weighted average, confidence fusion), the random fluctuations of the prediction results of a single frame or some frames are avoided from affecting the final classification, making the classification results more stable and reliable.

[0211] 4) Strong clinical interpretability: Since risk level prediction is performed for each frame, high-risk key frames or non-key frames can be located. The model is presented to doctors by labeling frame indexes, generating perfusion feature heatmaps (highlighting high-risk areas), and outputting frame-level risk scores (0-100 points). This helps doctors understand the specific image frames and perfusion features on which the model is based, and improves clinical trust in the model.

[0212] 5) Good versatility: The core of the method is keyframe extraction, temporal fusion and multi-frame risk decision-making, which can be extended to CEUS image analysis of organs such as liver and kidney for risk classification of related lesions in these organs.

[0213] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0214] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0215] Obviously, those skilled in the art can make various modifications and variations to the invention without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A CEUS image analysis method based on keyframe segmentation and temporal tracking fusion, characterized in that, include: The original CEUS video is preprocessed to obtain a preprocessed frame sequence; Extract keyframes from the preprocessed frame sequence to obtain a keyframe sequence; The patch region segmentation module inputs keyframe images from the keyframe sequence into the trained segmentation network to segment patch regions and obtain a patch binary mask. ; The temporal feature tracking and fusion module is based on the preprocessed frame sequence and the image of each keyframe. The patch binary mask obtained by the segmentation network High-level feature map Obtain the intermediate frames between keyframes. Image The corrected mask ; Binary mask based on patchy dimensions of keyframe images Corrected mask for non-keyframes Risk level prediction is performed on all frame images, including keyframes and non-keyframes, and then the final classification of the corresponding patches is obtained based on a preset decision mechanism.

2. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 1, characterized in that, The original CEUS video is preprocessed to obtain a preprocessed frame sequence, including: Convert the original CEUS video (non-standard format) to DICOM or AVI format to obtain a resolvable frame sequence; Z-score normalization is performed on the parsable frame sequence to obtain a frame sequence with uniform brightness; A motion stabilization algorithm based on ORB feature point matching is used to eliminate inter-frame motion artifacts in a frame sequence with uniform brightness, resulting in a stable frame sequence with motion artifacts eliminated, which is then used as the preprocessed frame sequence.

3. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 1, characterized in that, Keyframes are extracted from the preprocessed frame sequence to obtain a keyframe sequence, including: The first frame of the preprocessed frame sequence is used as the initial keyframe, and then a frame is selected as the keyframe every 10 frames. Alternatively, calculate the information entropy difference between adjacent frames in the preprocessed frame sequence, and select key frames from adjacent frames when the difference is greater than a threshold. Alternatively, the inter-frame motion can be calculated using optical flow, and the corresponding frame can be selected as a keyframe when the inter-frame motion exceeds a threshold.

4. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 1, characterized in that, The segmentation network includes: The multi-scale convolution and feature aggregation module is used to take keyframe images from the keyframe sequence as input features. Based on input features The input feature X is processed in parallel using 3×3, 5×5, and 7×7 convolutional branches, and the features obtained from the parallel processing are concatenated to obtain multi-scale fused features. ; A dilated spatial pyramid pooling module is used to process multi-scale fused features with dilated convolutions of dilation rates of 1, 6, 12, and 18. By fusing features from different receptive fields, global and local fused features are obtained. ; Channel attention and skip attention gate modules, based on global and local fusion features This yields a patchy binary mask with the same size as the keyframe image. .

5. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 4, characterized in that, Channel attention and skip attention gate modules are used for fusion of global and local features. This yields a patchy binary mask with the same size as the keyframe image. ,include: Channel attention and skip attention gate modules, based on global and local fusion features Calculate channel weights ; Channel weights Features of global and local fusion The channel-wise multiplication yields the channel-enhanced feature F. se ; The raw encoder features of each layer in the encoding stage are obtained from the keyframe images. Based on the original encoding end features and the channel-enhanced feature F, which is part of the decoding end feature. se By using additive attention to control the feature fusion of skip connections, a spatial attention weight map is obtained. ; Based on spatial attention weight graph and original encoding end features The fusion characteristics of the focused patch region were obtained. ; Focusing on the fusion characteristics of patch regions The decoding features are concatenated with the guiding features of the decoding process to update the decoding features, resulting in the final feature map F of the decoding stage. dec ; The final feature map F in the decoding stage dec A 1×1 convolution is performed to compress the number of channels to 1, generating a single-channel score map. Sigmoid activation is applied to the values ​​at each position in the single-channel score map, mapping the values ​​to the 0~1 range to obtain a normalized score map. A segmentation threshold is set, and the normalized score map is evaluated pixel-by-pixel: if the score is greater than or equal to the segmentation threshold, it is marked as 1; if the score is less than the segmentation threshold, it is marked as 0, thus obtaining a patchy binary mask with the same size as the keyframe image. .

6. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 1, characterized in that, The overall loss of the segmentation network is: ,in, , For Dice's loss, For binary classification, cross-entropy loss, This is the boundary loss.

7. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 1, characterized in that, The temporal feature tracking and fusion module is based on the preprocessed frame sequence and the image of each keyframe. The patch binary mask obtained by the segmentation network High-level feature map Obtain the intermediate frames between keyframes. Image The corrected mask ,include: Suppose the preprocessed frame sequence is as follows: Given the total number of frames, the set of keyframes is: The image of each keyframe A patch binary mask is obtained through the segmentation network. High-level feature map Among them, high-level feature maps That is, global and local fusion features ; Images in adjacent keyframes and First, the displacement field of dense pixels is estimated using the optical flow network RAFT. : ; Based on dense pixel displacement field Intermediate frames between adjacent keyframes Image Interpolation yields the image from the keyframe to the intermediate frame. displacement field : ,in, ; Based on displacement field mask the keyframe High-level feature map Mapping to intermediate frames To obtain the image of the intermediate frame Forward mask The image of the middle frame propagation characteristics ,in, The optical flow warping operator for bilinear interpolation; From the next keyframe Propagate back to intermediate frame To obtain the image of the intermediate frame Reverse mask ; The intermediate frame is obtained by weighted summation based on confidence level. initial fusion mask : Among them, confidence weight and Adaptive calculation based on optical flow consistency and image similarity: , Similarly; The corrected mask is obtained using a lightweight U-Net mask correction module. , The input includes intermediate frames. Image Initial mask With propagation characteristics ; Based on intermediate frames Image and the previous intermediate frame -1 image Calculate and define the confidence index : ; when or This triggers a rollback, changing the patch binary mask image. As the updated and revised mask Otherwise, keep the original modified mask. .

8. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 7, characterized in that, The training loss function for the lightweight U-Net-like mask correction module includes cross-entropy and boundary consistency loss. : ,in, For real label masking, For gradient operators, .

9. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 8, characterized in that, During the training of the temporal feature tracking and fusion module, temporal consistency constraints are applied. Cross-entropy and boundary consistency loss Together, they form the overall training loss function, which is used for backpropagation to optimize the parameters of the entire temporal feature tracking and fusion module; wherein, based on intermediate frames Image The corrected mask The previous intermediate frame -1 image The corrected mask Compute timing consistency constraints : .

10. The CEUS image analysis method based on keyframe segmentation and temporal tracking fusion as described in claim 1, characterized in that, Binary mask based on patchy dimensions of keyframe images Corrected mask for non-keyframes Risk level prediction is performed on all frame images, including keyframes and non-keyframes. Then, based on a preset decision mechanism, the final classification of the corresponding patches is obtained, including: For each frame of the image and patch binary mask Or a modified mask Perform convolutional encoding to extract frame-level embeddings and obtain frame-level feature vectors. : ,in, It is a frame-level feature vector; Frame-level feature sequences A Transformer encoder is used to model the input temporal features, resulting in a feature sequence that fuses spatiotemporal information. ,in, To enable learnable positional encoding, the Transformer layer uses a multi-head self-attention mechanism; Feature sequences based on fusion of spatiotemporal information The time-series representation is obtained through global pooling. : ; Based on time-series representation Ultimately, the risk level is output through two layers of sensors. : , ( ) is the activation function. This is the weight matrix of the first-layer perceptron, used for temporal representation. Perform a linear transformation; This is the weight matrix of the second-layer perceptron, used to perform a quadratic linear transformation on the activated features; loss function A combination of cross-entropy and focus loss is used: ,in, Weighting coefficients , For cross-entropy loss, Loss of focus; Risk level prediction results for all frames Based on a pre-defined decision-making mechanism, the final classification of the corresponding plaque is obtained.

Images