A method, apparatus, and storage medium for processing ablation images.

Abstract

This invention relates to the field of image processing technology, specifically to a method, apparatus, and storage medium for processing ablation images. The invention applies the YOLOv5-SFT algorithm to the original video to locate the initial sequence of ablation images contained within the original video. Then, based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained within the initial sequence, the target sequence of ablation images is selected from the initial sequence. Finally, the EfficientTet-SK algorithm is applied to the target sequence of ablation images to obtain the evaluation result. Because the YOLOv5-SFT algorithm of this invention adds an SE attention mechanism and a Transformer module, it accurately locates the initial sequence of ablation images in the original video. Furthermore, the EfficientTet-SK algorithm adds an SK and a time-shifting module (TSM), thus obtaining accurate evaluation results for the ablation region.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method, apparatus, and storage medium for processing ablated images. Background Technology

[0002] Ablation therapy is a commonly used minimally invasive treatment for early-stage hepatocellular carcinoma (HCC). However, the recurrence rate after ablation is high, with approximately 50%-70% of HCC patients experiencing recurrence within 5 years. Therefore, timely postoperative efficacy assessment is extremely important. Contrast-enhanced ultrasound (CEUS) has become the preferred guiding method for postoperative monitoring of HCC due to its radiation-free nature, convenience, and ability to dynamically observe liver structures in real time, providing more sensitive visualization of HCC characteristics. Clinically, physicians typically first locate the ablation area (the area previously treated surgically) in CEUS images, and then further diagnose the tumor activity in the ablation area to assess the risk of recurrence and evaluate the efficacy of ablation therapy for HCC patients. However, currently, these two steps are usually performed by experienced physicians, which is time-consuming, labor-intensive, and highly dependent on the physician. That is, current technology relies solely on the physician's experience when locating the ablation area in acquired patient images, resulting in inaccurate ablation area localization and consequently, poor accuracy in subsequent assessments of the ablation area (reflecting the activity of cancer cells at the postoperative site).

[0003] In summary, the accuracy of evaluating ablation images in existing technologies is relatively low.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method, apparatus, and storage medium for processing ablation images, thereby resolving the issue of low accuracy in evaluating ablation images in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for processing ablation images, comprising:

[0008] The YOLOv5-SFT algorithm is applied to the original video to locate the initial sequence of ablation images contained in the original video. The YOLOv5-SFT algorithm is used to locate the ablation region in the original video. The YOLOv5-SFT algorithm includes an SE attention mechanism and a Transformer module. The initial sequence of ablation images is used to characterize the site where the patient was surgically treated.

[0009] Based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained in the initial sequence of ablation images, the target sequence of ablation images is selected from the initial sequence of ablation images;

[0010] The EfficientTet-SK algorithm is applied to the ablation image target sequence to obtain the evaluation result output by the EfficientTet-SK algorithm. The EfficientTet-SK algorithm is used to identify information about the ablation region in the ablation image target sequence. The EfficientTet-SK algorithm includes the SK attention mechanism and the time-shifting module (TSM).

[0011] In one implementation, the YOLOv5-SFT algorithm is applied to the original video to locate the initial sequence of ablation images contained in the original video. The YOLOv5-SFT algorithm is used to locate the ablation region in the original video. The YOLOv5-SFT algorithm includes an SE attention mechanism and a Transformer module. The initial sequence of ablation images is used to characterize the site where the patient was surgically treated, including:

[0012] The first YOLOv5-SFT algorithm is applied to the original video to obtain the ultrasound contrast image sequence output by the first YOLOv5-SFT algorithm;

[0013] The second Yolov5-SFT algorithm is applied to the ultrasound contrast imaging sequence to locate the initial sequence of ablation images contained in the ultrasound contrast imaging sequence.

[0014] In one implementation, both the first Yolov5-SFT algorithm and the second Yolov5-SFT algorithm include:

[0015] The channel weight adjustment module is used to adjust the weight of each channel in the original video or the ultrasound contrast image sequence.

[0016] The global feature extraction module has its input end electrically connected to the output end of the channel weight adjustment module, and is used to extract the global features of the image after adjusting the channel weights output by the channel weight adjustment module.

[0017] The motion compensation module, whose input is electrically connected to the output of the global feature extraction module, is used to remove interference to the image caused by the user's breathing movements, where the user is the user in the original video.

[0018] In one implementation, the channel weight adjustment module includes:

[0019] The first convolutional module Conv_F has the input terminal used to input the original video or the ultrasound contrast image sequence, and the output terminal is electrically connected to the input terminal of the SE attention mechanism;

[0020] The second convolutional module Conv_F has an input terminal for the original video or the ultrasound contrast image sequence.

[0021] The fully connected module has its input terminal electrically connected to the output terminals of both the first convolutional module Conv_F and the second convolutional module Conv_F.

[0022] The third convolutional module, Conv_F, has its input electrically connected to the output of the fully connected module, and its output electrically connected to the input of the global feature extraction module.

[0023] In one implementation, the first convolutional module Conv_F, the second convolutional module Conv_F, and the third convolutional module Conv_F all include:

[0024] Convolutional layer (Conv);

[0025] The pooling layer BN is connected to the convolutional layer Conv;

[0026] The activation function module is used to extract the semantic features contained in the output image of the pooling layer BN.

[0027] In one implementation, the step of selecting the ablation image target sequence from the initial ablation image sequence based on the frame length corresponding to the initial ablation image sequence and the similarity between adjacent frame images contained in the initial ablation image sequence includes:

[0028] When the frame length corresponding to the initial sequence of ablation images is greater than the frame length threshold, the similarity of adjacent frame images in the initial sequence of ablation images is determined.

[0029] When the similarity between adjacent frames is greater than a similarity threshold, one of the adjacent frames is deleted to obtain the ablation image target sequence.

[0030] Alternatively, when the frame length corresponding to the initial sequence of the ablation image is less than or equal to the frame length threshold, the initial sequence of the ablation image is upsampled to obtain the target sequence of the ablation image with a frame length equal to the frame length threshold.

[0031] In one implementation, the similarity calculation method for adjacent frame images includes:

[0032] Determine the structural similarity of adjacent frame images;

[0033] Determine the peak signal-to-noise ratio of adjacent frames;

[0034] The similarity between adjacent frames is determined based on the structural similarity and the peak signal-to-noise ratio.

[0035] In one implementation, the SK attention mechanism includes:

[0036] First convolution kernel;

[0037] Second convolution kernel;

[0038] A global average pooling pool is connected to the first convolutional kernel and the second convolutional kernel, respectively.

[0039] A fully connected layer is connected to the global average pool;

[0040] The first weighted layer is connected to the fully connected layer and the first convolutional kernel, respectively.

[0041] The second weighted layer is connected to the fully connected layer and the second convolutional kernel, respectively.

[0042] Secondly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and a processing program for ablation images stored in the memory and executable on the processor, wherein when the processor executes the processing program for ablation images, it implements the steps of the ablation image processing method described above.

[0043] Thirdly, embodiments of the present invention also provide a computer-readable storage medium storing a processing program for an ablation image, wherein when the processing program for the ablation image is executed by a processor, the steps of the ablation image processing method described above are implemented.

[0044] Beneficial effects: This invention first applies the YOLOv5-SFT algorithm to the original video (the video of the patient after ablation surgery) to locate the initial sequence of ablation images contained in the original video (that is, to locate the images containing the ablation region in the video and the specific position of the ablation region in each frame). Then, based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained in the initial sequence of ablation images, the target sequence of ablation images is selected from the initial sequence of ablation images. Finally, the EfficientTet-SK algorithm is applied to the target sequence of ablation images to obtain the evaluation result output by the EfficientTet-SK algorithm. Because the Yolov5-SFT algorithm of this invention adds the SE attention mechanism and the Transformer module, the SE attention mechanism can extract local features about the ablation region, and the Transformer module can extract global features about the ablation region. Therefore, by combining global and local information, the initial sequence of ablation images in the original video can be accurately located. Furthermore, the EfficientTet-SK algorithm of this invention adds the SK attention mechanism and the time-shifting module TSM. TSM realizes the fusion of the temporal features of the video and the spatial features of the image. The SK attention mechanism obtains different receptive fields (the range of data that the EfficientTet algorithm can process) to obtain the characteristics of the ablation region in the image, thereby obtaining an accurate evaluation result for the ablation region (the evaluation result is used to reflect the activity of cancer cells after the patient's ablation). Attached Figure Description

[0045] Figure 1 This is an overall flowchart of the present invention;

[0046] Figure 2 This is a diagram of the Yolbv5-SFT network structure in an embodiment of the present invention;

[0047] Figure 3 This is a structural diagram of the SPP in an embodiment of the present invention;

[0048] Figure 4 This is a diagram of the FPN structure in an embodiment of the present invention;

[0049] Figure 5 This is a diagram of the PAN structure in an embodiment of the present invention;

[0050] Figure 6 This is a flowchart illustrating the two-stage ablation region assessment method in this embodiment of the invention.

[0051] Figure 7 This is a structural diagram of the similarity comparison module in an embodiment of the present invention;

[0052] Figure 8This is a diagram of the EfficientTet-SK network structure in an embodiment of the present invention;

[0053] Figure 9 This is a structural diagram of the SK attention module in an embodiment of the present invention;

[0054] Figure 10 This is a diagram of the Transformer block structure in an embodiment of the present invention;

[0055] Figure 11 This is a structural diagram of the TransformerLayer in an embodiment of the present invention;

[0056] Figure 12 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0058] Studies have shown that ablation therapy is a commonly used minimally invasive treatment for early-stage hepatocellular carcinoma (HCC). However, the recurrence rate after ablation is high, with approximately 50%-70% of HCC patients experiencing recurrence within 5 years. Therefore, timely postoperative efficacy assessment is extremely important. Contrast-enhanced ultrasound (CEUS) has become the preferred guiding method for postoperative monitoring of HCC due to its radiation-free nature, convenience, and ability to dynamically observe liver structures in real time, providing more sensitive visualization of HCC characteristics. Clinically, doctors typically first locate the ablation area (the area previously treated surgically) in the CEUS image, and then further diagnose the tumor activity in the ablation area to determine the risk of recurrence, thus assessing the efficacy of ablation therapy for HCC patients. However, currently, these two steps are usually performed by experienced physicians, which is time-consuming, labor-intensive, and highly dependent on the physician. That is, current technology relies solely on the physician's experience when locating the ablation area in the acquired patient images, resulting in inaccurate ablation area localization and consequently, poor accuracy in subsequent assessments of the ablation area (reflecting the activity of cancer cells at the postoperative site).

[0059] To address the aforementioned technical problems, this invention provides a method, apparatus, and storage medium for processing ablation images, resolving the issue of low accuracy in evaluating ablation images in existing technologies. Specifically, the invention first applies the Yolov5-SFT algorithm to the original video to locate the initial sequence of ablation images contained within it. Then, based on the frame length corresponding to the initial sequence and the similarity between adjacent frames within the initial sequence, a target sequence of ablation images is selected from the initial sequence. Finally, the EfficientTet-SK algorithm is applied to the target sequence to obtain the evaluation result output by the EfficientTet-SK algorithm. This invention improves the accuracy of the evaluation results.

[0060] For example, when acquiring raw video of a patient's liver after ablation, the video may contain images where the ablation area is not present. Therefore, the YOLOv5-SFT algorithm is needed to accurately locate which images in the raw video contain the ablation area and at what location within those images (the sequence of images containing the ablation area is denoted as the initial ablation image sequence). The YOLOv5-SFT algorithm locates the initial ablation image sequence (for example, the initial ablation image sequence contains frames a, b, c, d, and e, with the capture times for a, b, c, d, and e set sequentially). Since a, b, c... Since the frame length 5 formed by a, b, c, and e is relatively large, it is necessary to calculate the similarity x1 between a and b, the similarity x2 between b and c, the similarity x3 between c and d, and the similarity x4 between d and e. If x1 is much larger than the threshold, then either a or b is deleted (assuming a is deleted), thus forming the ablation image target sequence b, c, d, and e. Finally, the EfficientTet-SK algorithm is applied to evaluate the activity value of the ablation region in the four frames of the ablation image target sequence b, c, d, and e. The larger the activity value, the greater the probability of cancer cell spread in the patient after the ablation surgery; the smaller the activity value, the lower the probability of cancer cell spread in the patient after the ablation surgery.

[0061] Exemplary methods

[0062] The ablation image processing method of this embodiment can be applied to a terminal device, which can be a terminal product with video acquisition function, such as an ultrasound imaging device. In this embodiment, as... Figure 1 As shown, the method for processing the ablation image specifically includes the following steps S100, S200, and S300:

[0063] S100, Apply the YOLOv5-SFT algorithm to the original video to locate the initial sequence of ablation images contained in the original video. The YOLOv5-SFT algorithm is used to locate the ablation region in the original video. The YOLOv5-SFT algorithm includes an SE attention mechanism and a Transformer module. The initial sequence of ablation images is used to characterize the site where the patient was surgically treated.

[0064] The YOLOv5-SFT algorithm is an algorithm that adds the SE attention mechanism and the Transformer module to YOLOv5. YOLOv5 is an algorithm used for object detection. In this embodiment, YOLOv5-SFT is used to detect ablation regions.

[0065] The Yolov5-SFT algorithm based on Yolov5 was chosen in this embodiment because the computing resources of the graphics processing unit (GPU) used to process the acquired videos of the patient's lesion after ablation are relatively limited. This makes some complex network methods unsuitable for clinical applications and unable to meet the requirements of real-time applications. Yolov5, on the other hand, has higher accuracy, faster running speed, and does not consume too much memory, making it more suitable for practical applications. In addition, because Yolov5 includes a feature pyramid network, a path aggregation network, and a spatial pyramid pooling module, it can optimize feature fusion at different scales. This makes Yolov5 better able to detect targets of different scales, which is beneficial for detecting ablation regions of different sizes.

[0066] The reason why the SE attention mechanism (SE is used to extract features of the RGB channels of an image, such as pixel values ​​of the RGB channels) is introduced in the YOLOv5 algorithm (object detection algorithm) in this embodiment is that the SE attention mechanism mainly increases the features of the detection area from the perspective of the channel domain, suppresses the background noise of ultrasound imaging, and improves the ability to extract image features.

[0067] In one implementation, the Yolov5-SFT algorithm framework is as follows: Figure 2 As shown, it includes a three-channel weight adjustment module C3-SEF, a global feature extraction module C3TR, and a motion compensation module (composed of...). Figure 2 The algorithm consists of Localization and Crop. Localization outputs the location of the ablation area on the input ultrasound contrast image based on the coordinate values ​​predicted by the network and draws the corresponding coordinate frame. Crop outputs the ablation area map cropped based on the predicted coordinate frame values.

[0068] Figure 2 The structure diagram of SPP (Spatial Pyramid Pooling Module) is as follows: Figure 3As shown, the SPP module consists of an input convolutional layer (CBL), three parallel max-pooling layers, and an output convolutional layer (CBL). The high-level features obtained from C3TR are input into the SPP module. First, they undergo convolution operations in the input convolutional layer, then local features are fused through the max-pooling layers. Padding is then used to ensure that the three parallel pooling layers produce feature maps of the same size. Subsequently, the three features and the input features are concatenated along the channel direction, combining local and global features. Finally, the concatenated features are convolved to obtain output features that fuse information at different scales, enriching the expressive power of the feature maps and enhancing the network's ability to learn semantic information about targets at different scales.

[0069] Figure 2 The structure diagram of FPN (Feature Pyramid Networks) in the text is as follows: Figure 4 As shown, the reason for setting up FPN is that deeper features contain more semantic features and less positional information, while shallower features contain more positional information and less semantic features. The FPN module mainly fuses features from different layers, passing deep semantic features into shallow layers to enhance semantic expression at multiple scales and achieve prediction on multi-scale features. The input of the FPN structure is the feature Ci from C3TR or C3-SEF in the current layer and the higher-level feature Fi+1. After upsampling, feature Fi+1 is added element-wise with feature Ci to obtain the feature Fi of the current layer. These features fuse feature information from different scales of the backbone network, ultimately yielding the refined features F1, F2, and F3.

[0070] Figure 2 The structure diagram of PAN (Path Aggregation Network) in the image is as follows: Figure 5 As shown, similar to the FPN module, PAN mainly fuses features from different layers. PAN passes shallow-layer positional features to deeper layers, enhancing localization capabilities across multiple scales and enabling prediction based on multi-scale features. The PAN structure input consists of high-level feature Fi+1 from FPN and low-level feature Pi. After downsampling, feature Pi is concatenated with feature Fi along the channel direction to obtain the current layer's feature Pi. Through the PAN module, more refined multi-scale features P1, P2, and P3, which fuse high-level semantic features and low-level spatial features, are finally obtained.

[0071] Figure 2 The HEAD in the diagram is the target detection head, which consists of three YOLO detectors with different scaling scales. Each detector consists of a 1×1 convolutional layer, a batch regularization layer, and an activation layer. The final output is the superposition of the outputs of the three detectors, which yields the network's predicted bounding box category, coordinates, and confidence score.

[0072] like Figure 2 As shown, each channel weight adjustment module C3-SEF includes a first convolutional module Conv_F, a second convolutional module Conv_F, and a third convolutional module Conv_F (these three convolutional modules have the same structure), an SE attention mechanism (SEBottleneck), and a connection layer cat. Each convolutional module Conv_F includes a convolutional layer Conv, a pooling layer BN, and an activation function module (used to extract semantic features from the image) connected in sequence.

[0073] In one embodiment, the activation function module employs the FReLU activation function to enhance the extraction of semantic features, improve the spatial sensitivity of the activation function, and significantly improve image vision.

[0074] like Figure 2 As shown, the global feature extraction module C3TR includes three convolutional kernels (Conv), one TransformerBlock, and one connection layer (cat).

[0075] In one embodiment, step S100 includes the following steps S101 and S102:

[0076] S101, apply the first Yolov5-SFT algorithm to the original video to obtain the ultrasound contrast image sequence output by the first Yolov5-SFT algorithm.

[0077] The Yolov5-SFT algorithm in this embodiment is an algorithm that adds the SE attention mechanism and the Transformer module to the original Yolov5 algorithm, i.e., the Yolov5-SFT network.

[0078] First, the original video is input into the channel weight adjustment module C3-SEF of the YOLOv5-SFT network. The final C3-SEF outputs a feature map with adjusted RGB weights. This feature map is then input into the global feature extraction module C3TR. C3TR extracts the global features of the feature map. Finally, motion compensation is applied to the global features, thus completing the extraction of the ultrasound contrast image sequence from the original video (each frame of the original video of the patient's liver contains a conventional B-mode ultrasound image and an ultrasound contrast image. Doctors usually only observe the ultrasound contrast image in the video, so the ultrasound image needs to be removed).

[0079] Because C3-SEF adds the SE attention mechanism and the FReLU activation function, the two work together to improve the efficiency of capturing spatial correlation in convolutional neural networks and reduce the duplication of gradient information during the optimization process.

[0080] like Figure 6 As shown, the original video Vo( Figure 2 V in CEUS The input is fed into the first and second convolutional modules Conv_F of the C3-SEF in the YOLOv5-SFT network. Simultaneously, the width w and height h of each frame in Vo, the number of frames the YOLOv5-SFT network can process at once bs, and the RGB channels c1 of each frame are also input (Vo, bs, w, h, and c1 are denoted as image features) to the first and second convolutional modules Conv_F and Conv_F. This divides the image features into two parts: one part is calculated using the Conv_F and SEBottleneck blocks, and the other part is calculated directly using the Conv-F blocks. The results of these two feature sets are concatenated using cat, and finally, the Conw_F block is used to obtain new features (i.e., the feature map after the first adjustment of the RGB weights, denoted as RGB features). Figure 1 Then add RGB features. Figure 1 The input is fed into the next C3-SEF, resulting in a feature map with further adjusted RGB weights, denoted as the RGB feature map. Figure 2 Finally, the RGB features Figure 2 The input is fed into the last C3-SEF, resulting in the final RGB feature map output by the last C3-SEF. For example... Figure 2 As shown, the final RGB feature map is input into the global feature extraction module C3TR, which performs global feature extraction on the final RGB feature map.

[0081] The specific process of global feature extraction by the global feature extraction module C3TR on the final RGB feature map is as follows:

[0082] The RGB feature map output by C3-SEF is divided into two parts: one part is processed by Conv blocks and Transformer blocks (i.e., ... Figure 2 The TransformerBlock in the code includes a multi-head self-attention module (MAB) for acquiring global information and combining it with local information to capture more features. This part is directly computed through the Conv block, concatenating the results of these two features, and finally using the Conv block to obtain new features. The introduced multi-head self-attention module can map to multiple subspaces, enabling simultaneous attention to multiple key regions and effectively acquiring global information. It combines global and local information to capture more features and incorporates coordinate encoding into the input features, better capturing the order and distance of the input element sequence, which is more beneficial for the coordinates of the predicted ablation region bounding box in the output sequence {Vceus}.

[0083] In one embodiment, the Transformer block is as follows: Figure 10 As shown, Figure 2 The features output by Conv in the Transformer block are fed into the Transformer layer, such as... Figure 11 As shown, the TransformerLayer processes the obtained features through three parallel Linear operations, and then uses the MultiheadAttention mechanism to enable the model to learn relevant information in different representation subspaces, thereby enhancing the model's expressive power. Next, the original features and the features obtained by multihead attention are added together through an addition operation, and finally added to the result of the two Linear operations to obtain the output.

[0084] After extracting the global features, motion compensation is performed on the global features. Figure 2 The Localization and Crop processes in this context are as follows:

[0085] Due to factors such as the movement of the liver caused by ultrasound imaging equipment and human respiratory movements, the observed target area may shift, resulting in lesions that may only be visible in certain frames of the video. Furthermore, the back-and-forth movement of the target nodule can affect real-time contrast imaging observation, and its motion artifacts can easily interfere with the doctor's judgment, affecting subsequent analysis and treatment.

[0086] Because contrast-enhanced ultrasound can clearly display internal blood vessels that move with organs, enhance blood flow imaging, and provide real-time dynamic observation, motion compensation can be achieved by detecting blood vessels in contrast-enhanced ultrasound to reduce the influence of factors such as respiration. When predicting the ablation area, frames with a predicted confidence value Ci below a threshold a or where no ablation area is predicted are discarded (in this embodiment, a is set to 0.3). By discarding single-frame images where the target ablation area is significantly deviated from the imaging plane, motion compensation is performed to reduce the influence of equipment and respiration, while simultaneously enhancing the usability of the obtained ablation localization sequence.

[0087] S102, apply the second Yolov5-SFT algorithm to the ultrasound contrast imaging sequence to locate the initial sequence of ablation images contained in the ultrasound contrast imaging sequence.

[0088] The ultrasound contrast imaging sequence Vceus is extracted from the original video in step S101, as follows: Figure 6 As shown, the Yolov5-SFT algorithm was then applied again to locate the initial ablation image sequence V1 from the ultrasound contrast imaging sequence. The Yolov5-SFT algorithm was used in this process. Figure 2 The framework within.

[0089] In this embodiment, S101 uses the YOLOv5-SFT algorithm to perform coarse detection on the original video to detect the ultrasound contrast imaging image sequence Vceus. S102 uses the YOLOv5-SFT algorithm to perform fine detection on Vceus to output the initial ablation image sequence V1. Since the coarse detection yields a sequence containing only the ultrasound contrast portion, and doctors typically observe the ablation area and surrounding liver parenchyma during diagnosis, fine detection using YOLOv5-SFT is then used to accurately locate the ablation area in sequence {Vceus} and output the ablation localization sequence {V1}.

[0090] S200, based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained in the initial sequence of ablation images, the target sequence of ablation images {V} is selected from the initial sequence of ablation images. c}

[0091] Because contrast-enhanced ultrasound (CEUS) uses dynamic sequence imaging, liver CEUS perfusion acquisition typically requires capturing different vascular phases for contrast enhancement, resulting in image acquisition times of several minutes and hundreds or even thousands of frames. Furthermore, the differences between adjacent frames in the same case video are small, leading to redundant information and high repetition. Therefore, this embodiment inputs the detected ablation sequence {V1} (i.e., the initial ablation image sequence V1) into... Figure 7 The similarity comparison module in the code is used to reduce repetition.

[0092] In one embodiment, step S200 is performed as follows:

[0093] like Figure 7 As shown, when the frame length N0 corresponding to the initial sequence {V1} of the ablation image is greater than the frame length threshold 2L, the similarity of adjacent frame images in the initial sequence of the ablation image is determined.

[0094] In one embodiment, L is a constant because current classification networks cannot handle video frame sets of different lengths. Therefore, all input videos need to be processed before being fed into the network to obtain videos with the same frame length. Considering that most hepatocellular carcinoma videos are between 200-800 frames, and the number of frames that video networks can currently handle is limited, after conducting multiple experiments with different frame lengths (L=120, 150, 90, etc.), it was found that L=120 yielded the best results. Therefore, this embodiment sets L=120. For other types of videos, factors such as the specific video frame length and the similarity between video frames need to be considered.

[0095] When the similarity between adjacent frames is greater than a similarity threshold, one of the adjacent frames is deleted to obtain the ablation image target sequence.

[0096] Alternatively, when the frame length corresponding to the initial sequence of the ablation image is less than or equal to the frame length threshold, the initial sequence of the ablation image is upsampled to obtain the target sequence of the ablation image with a frame length equal to the frame length threshold.

[0097] In this embodiment, the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) of adjacent frame images are used to determine the similarity of adjacent frame images:

[0098]

[0099] Mean, standard deviation, and covariance formed by SSIM(x,y):

[0100]

[0101]

[0102]

[0103] In the formula, μ x μ y These are the average values ​​of x and y, respectively, and σ x and σ y These are the standard deviations of x and y, respectively, and σ xy Let x and y be the covariance, I and K be two images of size m*n, and MSE be the average of the sums of the squares of the subtractions of pixel values ​​at the same position in the two images. MSE represents the average difference between the pixel values ​​at each position in the two images. I It represents the maximum value of the color of an image point, i.e., MAX. I It is 256.

[0104] In one embodiment, the mean, standard deviation, covariance, and PSNR are weighted and calculated. If the weighted result is greater than a threshold, one of the two adjacent frames is deleted.

[0105] When the frame length corresponding to the initial sequence of ablation images is less than or equal to the frame length threshold (case videos with frame length N0 ≤ 2L), it is upsampled to 2L through frame sampling (the so-called upsampling is as follows). Figure 7 As shown, image k-1 is copied to obtain two images (k-1). For case videos with frame length N0 > 2L, SSIM and PSNR are used to determine the similarity between two adjacent frames in the video. A similarity threshold is set for adjacent frames. When the similarity between adjacent frames is greater than this threshold, one of the two adjacent frames is randomly deleted. Downsampling based on the similarity characteristics between images can effectively delete frames with small differences while avoiding excessive downsampling. Through the similarity comparison module, a fixed-length ablation localization frame sequence {Vc} with large differences between adjacent frames is finally obtained.

[0106] In one embodiment, due to limitations in the image itself and imaging conditions, the image contrast is low. The localization sequence {Vc} obtained using the ACE (Adaptive Contrast Enhancement) algorithm undergoes contrast enhancement processing. By calculating the relative brightness relationship between the target point and surrounding pixels through difference, the final pixel value is corrected, making the enhancement effect on the weaker contrast areas of the image more pronounced. This allows for better observation of the features of the ablation area for further evaluation.

[0107] S300, apply the EfficientTet-SK algorithm to the ablation image target sequence to obtain the evaluation result output by the EfficientTet-SK algorithm. The EfficientTet-SK algorithm is used to identify information about the ablation region in the ablation image target sequence. The EfficientTet-SK algorithm includes the SK attention mechanism.

[0108] In one embodiment, the EfficientTet-SK algorithm is as follows: Figure 8 As shown, Figure 8 The FC in the output is the evaluation result.

[0109] In one embodiment, such as Figure 9 As shown, the SK attention mechanism includes a first convolutional kernel U1, a second convolutional kernel U2, a global average pooling Fgp, a fully connected layer Ffc, a first weighted layer a, and a second weighted layer b.

[0110] In this embodiment, the SK attention module uses convolutional kernels with different weights for different images to obtain information from different receptive fields. For example... Figure 9 As shown, first, the input vector X( Figure 8 The structure output by DepwiseConv is convolved with kernels of different sizes, and then two matrices a and b are output through Fgp and Ffc. Then, U1 and U2 are weighted using a and b respectively, and finally the output vector A is obtained by summing.

[0111] In one embodiment, such as Figure 8 As shown, EfficientTet-SK also includes a Time Shifting Module (TSM). TSM facilitates information exchange between adjacent frames by shifting a portion of the channel count along the time dimension. For each inserted time shifting module, the temporal receptive field is expanded by a factor of 2, enabling highly complex temporal modeling. Ultimately, temporal information is introduced into the 2D network, effectively fusing multi-layer temporal information without increasing computational or parameter count. Figure 8The TSM module uses a residual model. One branch performs a shift operation in the time dimension, exchanging information with nearby frames for each frame to perform temporal modeling. The other branch retains the original spatial information. This preserves the spatial feature learning ability of the original 2D model while adding temporal information.

[0112] The MBConv-SKT module in the EfficientTet-SK framework is the main module of the EfficientTet-SK network. For example... Figure 8 In the main branch of the MBConv-SKT module, 1x1 convolutions are first used for dimensionality increase, followed by 3x3 or 5x5 depthwise convolutions. Then, the SK module uses an attention module to adjust the feature matrix, followed by the TSM module and then 1x1 convolutions for dimensionality reduction. After passing through the Dropout layer, the obtained feature matrix is ​​added to the original feature matrix. Finally, a new feature matrix is ​​output.

[0113] The specific process of global feature extraction using the EfficientTet-SK framework is as follows:

[0114] EfficientTet-SK is used to process each frame of the video sequence (ablation image target sequence {Vc}) separately, and a predicted output is obtained for each frame. The outputs of all frames are then averaged to obtain the final predicted result. By using EfficientTet-SK to identify the ablation sequence {Vc}, spatial information of each image frame and temporal information between video frames can be combined while reducing model parameters, achieving higher recognition accuracy through information fusion from both spatial and temporal perspectives.

[0115] Because the SK attention module can adaptively adjust the size of the receptive field according to multiple different scales of the input information, the network can acquire information from different receptive fields and better capture the characteristics of the ablation region; the TSM module can extract temporal information by shifting channels in time, achieving temporal information fusion without adding computational load; EfficientNet balances the depth, width, and resolution of deep learning networks, resulting in higher accuracy and faster speed. Therefore, based on EfficientNet, the SK attention module and the TSM module are introduced into the EfficientNet network to obtain the main module MBConv-SKT, and EfficienTet-SK is obtained through continuous iteration of MBConv-SKT (e.g., ...). Figure 8 The ablation localization sequence {Vc} obtained above.

[0116] The following data, consisting of 230 patients (112 assessed as active and 118 as inactive), are used to verify the accuracy of this invention in assessing the activity of the ablation area:

[0117] Data was acquired using instruments such as the Aplio 500 scanner and the Airexplorer scanner (ultrasound). Patients were those diagnosed with hepatocellular carcinoma between July 2014 and December 2018 who underwent ablation therapy and had undergone CT / MRI at least one year after ultrasound contrast imaging evaluation. Ablation boxes were labeled by physicians at collaborating hospitals. Pathological results obtained from surgery or biopsy were used as efficacy evaluation labels. Videos primarily included the arterial and delayed phases. To prevent network overfitting, the training data was augmented using various transformations, including translation, rotation, flipping, brightness enhancement, chroma enhancement, contrast enhancement, sharpness enhancement, and pseudo-color enhancement. The degree of each transformation was controlled by a relative parameter (e.g., rotation angle, enhancement intensity, etc.), and each transformation was performed with a different probability. For fair comparison, all settings were consistent across all comparison methods.

[0118] In order to measure the proposed Figure 6 The performance of the two-stage diagnostic framework was evaluated. The first stage used precision, recall, F1 score, and IOU (set to 0.5) to calculate the average AP (mAP@.5) for each category to assess the accuracy of ablation region localization. Figure 6 The second phase of the study uses accuracy, sensitivity, recall, specificity, precision, balanced accuracy (BAC), and F1 score to evaluate the predictive results of postoperative ablation assessment.

[0119] Adaptive Moment Estimation (Adam) was chosen as the optimizer for iterative model optimization. The learning rate was set to 0.001. The model was trained from scratch instead of using a pre-trained model. The training epochs were set to 200. Furthermore, the algorithm, which utilizes cross-entropy loss as the model's loss function, was implemented on the PyTorch platform using two NVIDIA TITAN X GPUs.

[0120] In summary, this invention first applies the YOLOv5-SFT algorithm to the original video (the video of the patient after ablation surgery) to locate the initial sequence of ablation images contained in the original video (that is, to locate the images containing the ablation region in the video and the specific position of the ablation region in each frame). Then, based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained in the initial sequence of ablation images, the target sequence of ablation images is selected from the initial sequence of ablation images. Finally, the EfficientTet-SK algorithm is applied to the target sequence of ablation images to obtain the evaluation result output by the EfficientTet-SK algorithm. Because the Yolov5-SFT algorithm of this invention adds the SE attention mechanism and the Transformer module, the SE attention mechanism can extract local features about the ablation region, and the Transformer module can extract global features about the ablation region. Therefore, by combining global and local information, the initial sequence of ablation images in the original video can be accurately located. Furthermore, the EfficientTet-SK algorithm of this invention adds the SK attention mechanism and the time-shifting module TSM. TSM realizes the fusion of the temporal features of the video and the spatial features of the image. The SK attention mechanism obtains different receptive fields (the range of data that the EfficientTet algorithm can process) to obtain the characteristics of the ablation region in the image, thereby obtaining an accurate evaluation result for the ablation region (the evaluation result is used to reflect the activity of cancer cells after the patient's ablation).

[0121] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 12 As shown, the terminal device includes a processor, memory, network interface, display screen, and temperature sensor connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for processing ablation images. The display screen can be a liquid crystal display (LCD) or an e-ink display. The temperature sensor is pre-installed inside the terminal device to detect the operating temperature of the internal components.

[0122] Those skilled in the art will understand that Figure 12The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0123] In one embodiment, a terminal device is provided, comprising a memory, a processor, and a processing program for ablation images stored in the memory and executable on the processor. When the processor executes the processing program for ablation images, it implements the following operation instructions:

[0124] The Yolov5-SFT algorithm is applied to the original video to locate the initial sequence of ablation images contained in the original video. The Yolov5-SFT algorithm is an algorithm that adds an SE attention mechanism and a Transformer module between the convolutional layer and the fully connected layer of the original Yolov5 (object detection algorithm). The initial sequence of ablation images is used to characterize the site where the patient was surgically treated.

[0125] Based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained in the initial sequence of ablation images, the target sequence of ablation images is selected from the initial sequence of ablation images;

[0126] The EfficientTet-SK algorithm is applied to the ablation image target sequence to obtain the evaluation result output by the EfficientTet-SK algorithm. The EfficientTet-SK algorithm is a target recognition algorithm with the addition of the time-shifting module TSM and the SK attention mechanism.

[0127] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing ablation images, characterized in that, include: The YOLOv5-SFT algorithm is applied to the original video to locate the initial sequence of ablation images contained in the original video. The YOLOv5-SFT algorithm is used to locate the ablation region in the original video. The YOLOv5-SFT algorithm includes an SE attention mechanism and a Transformer module. The initial sequence of ablation images is used to characterize the site where the patient was surgically treated. The YOLOv5-SFT algorithm includes three channel adjustment modules, a global feature extraction module, and a motion supplementation module. Based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained in the initial sequence of ablation images, the target sequence of ablation images is selected from the initial sequence of ablation images; The EfficientTet-SK algorithm is applied to the ablation image target sequence to obtain the evaluation result output by the EfficientTet-SK algorithm. The EfficientTet-SK algorithm is used to identify information about the ablation region in the ablation image target sequence. The EfficientTet-SK algorithm includes an SK attention mechanism and a time-shifting module (TSM). The SK attention module and the TSM module are introduced into the EfficientNet network to obtain the main module MBConv-SKT, and EfficientTet-SK is obtained through continuous iteration of MBConv-SKT.

2. The method for processing ablated images as described in claim 1, characterized in that, The YOLOv5-SFT algorithm is applied to the original video to locate the initial sequence of ablation images contained in the original video. The YOLOv5-SFT algorithm is used to locate the ablation region in the original video. The YOLOv5-SFT algorithm includes an SE attention mechanism and a Transformer module. The initial sequence of ablation images is used to characterize the site of surgical treatment of the patient, including: The first YOLOv5-SFT algorithm is applied to the original video to obtain the ultrasound contrast image sequence output by the first YOLOv5-SFT algorithm; The second Yolov5-SFT algorithm is applied to the ultrasound contrast imaging sequence to locate the initial sequence of ablation images contained in the ultrasound contrast imaging sequence.

3. The method for processing ablation images as described in claim 2, characterized in that, Both the first Yolov5-SFT algorithm and the second Yolov5-SFT algorithm include: The channel weight adjustment module is used to adjust the weight of each channel in the original video or the ultrasound contrast image sequence. The global feature extraction module has its input end electrically connected to the output end of the channel weight adjustment module, and is used to extract the global features of the image after adjusting the channel weights output by the channel weight adjustment module. The motion compensation module, whose input is electrically connected to the output of the global feature extraction module, is used to remove interference to the image caused by the user's breathing movements, where the user is the user in the original video.

4. The method for processing ablation images as described in claim 3, characterized in that, The channel weight adjustment module includes: The first convolutional module Conv_F has the input terminal used to input the original video or the ultrasound contrast image sequence, and the output terminal is electrically connected to the input terminal of the SE attention mechanism; The second convolutional module Conv_F has an input terminal for the original video or the ultrasound contrast image sequence. The fully connected module has its input terminal electrically connected to the output terminals of both the first convolutional module Conv_F and the second convolutional module Conv_F. The third convolutional module, Conv_F, has its input electrically connected to the output of the fully connected module, and its output electrically connected to the input of the global feature extraction module.

5. The method for processing ablation images as described in claim 4, characterized in that, The first convolutional module Conv_F, the second convolutional module Conv_F, and the third convolutional module Conv_F each include: Convolutional layer (Conv); The pooling layer BN is connected to the convolutional layer Conv; The activation function module is used to extract the semantic features contained in the output image of the pooling layer BN.

6. The method for processing ablation images as described in any one of claims 1-5, characterized in that, The step of selecting the target sequence of ablation images from the initial sequence of ablation images based on the frame length corresponding to the initial sequence of ablation images and the similarity between adjacent frames contained in the initial sequence of ablation images includes: When the frame length corresponding to the initial sequence of ablation images is greater than the frame length threshold, the similarity of adjacent frame images in the initial sequence of ablation images is determined. When the similarity between adjacent frames is greater than a similarity threshold, one of the adjacent frames is deleted to obtain the ablation image target sequence. Alternatively, when the frame length corresponding to the initial sequence of the ablation image is less than or equal to the frame length threshold, the initial sequence of the ablation image is upsampled to obtain the target sequence of the ablation image with a frame length equal to the frame length threshold.

7. The method for processing ablation images as described in claim 6, characterized in that, The method for calculating the similarity between adjacent frame images includes: Determine the structural similarity of adjacent frame images; Determine the peak signal-to-noise ratio of adjacent frames; The similarity between adjacent frames is determined based on the structural similarity and the peak signal-to-noise ratio.

8. The method for processing ablation images as described in any one of claims 1-5, characterized in that, The SK attention mechanism includes: First convolution kernel; Second convolution kernel; A global average pooling pool is connected to the first convolutional kernel and the second convolutional kernel, respectively. A fully connected layer is connected to the global average pool; The first weighted layer is connected to the fully connected layer and the first convolutional kernel, respectively. The second weighted layer is connected to the fully connected layer and the second convolutional kernel, respectively.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a processing program for ablation images stored in the memory and executable on the processor. When the processor executes the processing program for ablation images, it implements the steps of the ablation image processing method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a processing program for an ablation image, which, when executed by a processor, implements the steps of the ablation image processing method as described in any one of claims 1-8.

Images