Handheld ultrasonic video probe tracking method based on mount model

Abstract

This invention discloses a handheld ultrasound video probe tracking method based on the MOUNT model, mainly addressing the poor probe tracking performance in existing technologies due to the lack of consideration for the thin probe shape, uneven motion between adjacent frames, and low probe visibility. The implementation scheme is as follows: acquiring an ultrasound video dataset of the liver tumor ablation process; constructing a handheld ultrasound video probe tracking network MOUNT; iteratively training the handheld ultrasound video probe tracking network MOUNT; testing the trained MOUNT network on a test set to obtain a set of predicted candidate results; performing post-processing operations to filter and optimize the predicted candidate results set to obtain the probe entry point and tip coordinates, thus completing probe tracking. This invention focuses on the unique characteristics of the handheld ultrasound video probe tracking task, effectively improving the accuracy of handheld ultrasound video probe tracking, and can be used for real-time probe tracking in liver tumor ablation surgery.

Description

Technical Field

[0001] This invention belongs to the field of medical video image processing technology, and specifically relates to a handheld ultrasound video probe tracking method, which can be used for real-time tracking of probes during liver tumor ablation surgery, making it easier for doctors to perform liver tumor ablation treatment. Background Technology

[0002] According to the International Agency for Research on Cancer (IARC) of the World Health Organization, primary liver cancer is the sixth most common cancer worldwide and the third leading cause of cancer death. Furthermore, the liver is a common site for metastatic cancers in other parts of the body, such as the colon, lungs, breast, pancreas, and esophagus. Percutaneous liver tumor ablation, such as radiofrequency and microwave ablation, has been used to treat early-stage tumors ≤3cm in size for patients who are not suitable for surgical resection or liver transplantation.

[0003] Ultrasound provides real-time images and, compared to computed tomography (CT) scanners, is not only less expensive but also widely used to guide ablation device placement. During handheld ultrasound probe guidance, the physician operates the 2D ultrasound probe with one hand and inserts the ablation probe with the other, ensuring the probe remains within the plane of the real-time ultrasound image to visualize the probe and ablation target within the patient's body. For liver tumor ablation, the ablation needle must be inserted into the tumor, typically at the tumor's centroid, to ensure complete tumor ablation. After insertion, the probe generates heat in a specific area around the needle tip, creating a thermal ablation zone that induces irreversible cell death. In practice, because tumors are located deep within the body, often adjacent to critical organs and structures such as the gallbladder, bile ducts, and portal vein, precise probe placement is crucial, requiring the operator to visualize the needle's trajectory and direction. Understanding the overlap between the ablation zone and the tumor, and whether the tumor area will be completely covered, is critical when the probe is within the tumor.

[0004] However, although the identification of the needle tip and / or needle axis has been discussed in various techniques, percutaneous liver tumor ablation guided by a handheld ultrasound probe still faces unique challenges. The limitations of in-plane imaging, poor ultrasound image quality, other diseases in liver tumor patients, and ultrasound artifacts all contribute to the low visibility of the probe, increasing the difficulty of probe identification.

[0005] In recent years, deep learning-based methods have been applied to tasks such as probe tip or axis segmentation due to their powerful representation capabilities on large datasets. These works attempt to use deep learning models to detect probes in two-dimensional ultrasound videos, i.e., sequences of two-dimensional video image frames. Studies have shown that ultrasound videos can provide richer information than standalone 2D ultrasound images. Furthermore, these studies also demonstrate that, within a certain image sequence, rationally utilizing the context and temporal features of adjacent frames is a solution to improve probe visibility and mitigate the problem of missing needle information caused by the relative motion between handheld ultrasound probes and tissues.

[0006] Utilizing deep learning-based methods, Mwikirize et al. proposed a fast region-based convolutional neural network framework, Faster R-CNN, at the NIPS conference for probe classification. This framework builds upon previous R-CNN research by adding a Region Proposal Network (RPN) after the convolutional feature extractor for the first time. This RPN generates candidate bounding boxes at almost no cost by sharing convolutional features across the entire image, enabling real-time probe target detection. This method demonstrates the feasibility of deep learning techniques in probe localization tasks, but it does not delve into how to improve the accuracy of probe localization.

[0007] Gillies and Rodgers presented an improved Unet-based model at CVPR, segmenting linear probes in various clinical procedures, including liver tumor ablation, kidney biopsy and ablation, and prostate and gynecological brachytherapy. They introduced a needle-tip tracking module based on a Siamese neural network to perform cross-correlation operations on target probe features and all regions in the search frame to find the most similar region; subsequently, a bounding box was adjusted using an RPN module to achieve fast and accurate target detection. While this model achieves robust tracking and precise needle tip localization, its performance is based on evaluations of ex vivo biological tissues, making it difficult to apply these models to clinical medicine, especially considering the need to account for patient respiration and surgeon probe movement. Furthermore, research on probe localization in percutaneous liver tumor ablation is limited; existing studies have only shown probe insertion depths of approximately 7 cm. Therefore, the problem of poor probe appearance image quality due to various factors in ultrasound imaging during clinical liver ablation remains unresolved.

[0008] In summary, existing methods generally focus on better extracting the overall features of the image, but pay less attention to the unique characteristics of ultrasound video probe tracking tasks, such as the thin shape of the probe tip caused by the patient's breathing and probe movement, uneven motion between adjacent frames, and low probe visibility, resulting in relatively poor probe tracking performance. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of the prior art by proposing a handheld ultrasound video probe tracking method based on the MOUNT model. This method focuses on the unique characteristics of the probe tip's slender shape, uneven motion between adjacent frames, and low probe visibility, thereby improving the performance of handheld ultrasound video probe tracking.

[0010] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0011] (1) Obtain and define the ultrasound video dataset D of the liver tumor ablation process, and divide the defined dataset into training set D1 and test set D2 in a 7:3 ratio;

[0012] (2) Constructing the handheld ultrasound video probe tracking network MOUNT:

[0013] (2a) The existing Mask-RCNN network is used as the overall framework of the handheld ultrasound video probe tracking model. Based on this framework, the non-uniform motion perception UMP module and the adjacent frame aggregation AFA module are established, and the region candidate network RPN in the original Mask-RCNN network is replaced with the region candidate network TD-RPN with needle tip specific detection.

[0014] (2b) The non-uniform motion sensing UMP module, the region candidate network TD-RPN with needle tip specific detection, and the adjacent frame aggregation AFA module are connected in series to form the handheld ultrasound video probe tracking network MOUNT, and the weights W and biases b of the MOUNT network are initialized.

[0015] (3) Iteratively train the handheld ultrasound video probe tracking network MOUNT:

[0016] (3a) Input the frame combination of each ultrasound video in the ultrasound video training set D1 into the handheld ultrasound video probe tracking network MOUNT in the temporal order to obtain the output of the network MOUNT.

[0017] (3b) Construct the loss function L based on the output of the MOUNT network:

[0018] (3c) Update the parameters of the MOUNT network using stochastic gradient descent;

[0019] (3d) Repeat steps (3a) to (3c) to obtain the trained handheld ultrasound video probe tracking network MOUNT;

[0020] (4) Input the frames of each ultrasound video in the test set D2 into the trained handheld ultrasound video probe tracking network MOUNT in temporal order to obtain the prediction result set P. e ;

[0021] (5) For the prediction result set P e After performing post-processing operations for screening and optimization, the final probe inlet point coordinates p are obtained. ent Tip coordinates p tip and the angle of the needle axis direction Complete the tracking of the handheld ultrasound video probe.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] 1. The present invention, by constructing the handheld ultrasound video probe tracking network MOUNT, takes into account the problem of missing needle information caused by the relative motion between the handheld ultrasound probe and the tissue, and focuses on the unique characteristics of uneven motion of the handheld ultrasound probe, thin probe target, and low probe visibility, thus effectively improving the accuracy of probe tracking in handheld ultrasound video.

[0024] 2. This invention constructs a non-uniform motion perception (UMP) module and an adjacent frame aggregation (AFA) module in the MOUNT network. By perceiving the relative motion between key frames and reference frames, it can alleviate the problem of non-uniform motion of handheld ultrasound probes and aggregate the features between reference frames and key frames, effectively improving the model's tracking performance of the probe.

[0025] 3. In this invention, the region candidate network TD-RPN with specific detection for probe tip is used in the MOUNT network instead of the region candidate network RPN in the original Mask-RCNN, so candidate results for probe tips can be generated, which can effectively assist in the localization and detection of probes. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0027] Figure 2 This is a schematic diagram of the structure of the handheld ultrasound video probe tracking network MOUNT constructed in this invention;

[0028] Figure 3 This is a schematic diagram of the non-uniform motion UMP module structure constructed in this invention;

[0029] Figure 4 This is a schematic diagram of the TD-RPN structure with needle tip displacement prediction constructed in this invention;

[0030] Figure 5 This is a schematic diagram of the adjacent frame aggregation AFA module structure constructed in this invention. Detailed Implementation

[0031] The specific embodiments and effects of the present invention will be further explained and described below with reference to the accompanying drawings.

[0032] Reference Figure 1 The implementation steps of this invention are as follows:

[0033] Step 1: Obtain ultrasound video dataset of liver tumor ablation process.

[0034] A dataset D of ultrasound videos from liver tumor ablation procedures was obtained from the hospital, comprising 62 ultrasound videos of liver tumor ablation procedures from 13 patients with different types of liver tumors. Each ultrasound video in dataset D was defined. in M represents the i-th video image frame of size W×H in the ultrasound video. i ∈{0,1} W×H This represents the probe mask label corresponding to the i-th frame in the video, where |V| is the length of the ultrasound video V.

[0035] The defined dataset is divided into training set D1 and test set D2 in a 7:3 ratio.

[0036] Step 2: Construct the handheld ultrasound video probe tracking network MOUNT.

[0037] Reference Figure 2 The implementation of this step includes the following:

[0038] (2.1) The existing Mask-RCNN network is used as the overall framework of the handheld ultrasound video probe tracking model MOUNT. The Mask-RCNN is a two-stage framework. The first stage is used to scan the image and generate proposal boxes, and the second stage is used to classify the proposal boxes and generate bounding boxes and masks.

[0039] (3.2) Based on the Mask-RCNN framework, a non-uniform motion-aware UMP module and an adjacent frame aggregation AFA module are established, and the region candidate network RPN in the original Mask-RCNN network is replaced with a region candidate network TD-RPN with pinpoint-specific detection:

[0040] (2.2.1) Establish an optical flow generator G flow A feature extractor F f A non-uniform motion sensing UMP module is formed by sequentially connecting a residual block R and a residual block R, such as... Figure 3 As shown, it is used to align reference frames and keyframes with uneven motion:

[0041] The optical flow generator G flow We adopt the existing pre-trained ARFlow model, which consists of a feature pyramid extractor, an optical flow estimation mapping layer, a cost volume layer, an optical flow prediction layer, and a context network connected in sequence.

[0042] The feature extractor Ff It adopts the existing ResNet50 network, which consists of an input layer, a convolutional stage, and an output layer connected in sequence.

[0043] The residual block R is composed of a first convolutional layer, a first batch of normalized layers (BN), a ReLU activation function, a second convolutional layer, a second batch of normalized layers (BN), a summation layer, and a ReLU activation function layer connected in sequence.

[0044] (2.2.2) Establish a feature extractor F f A region candidate network (RPN) connection forms a region candidate network (TD-RPN) with tip-specific detection, as shown in the reference. Figure 4 It is used to achieve specific detection of the needle tip region;

[0045] (2.2.3) Establish an adjacent frame aggregation AFA module consisting of an addition operator and a multiplication operator, such as... Figure 5 As shown;

[0046] (2.3) The non-uniform motion sensing UMP module, the region candidate network TD-RPN with needle tip specific detection, and the adjacent frame aggregation AFA module are connected in series to form the handheld ultrasound video probe tracking network MOUNT, and the weights W and biases b of the MOUNT network are initialized.

[0047] Step 3: Input the frames of each ultrasound video in the ultrasound video training set D1 into the MOUNT network in temporal order to obtain the output of the MOUNT network.

[0048] (3.1) Obtain the ultrasound videos in the training set D1. For each ultrasound video V, let the ultrasound video frame collected at the current time be the keyframe I. k And set a width of W before the keyframe. r The frame reference window sets the selected frame in the window as the reference frame I. r ;

[0049] (3.2) According to reference frame I r and keyframe I k By determining the temporal sequence of each ultrasound video, a combination of frames can be obtained. Where F r It is a set of indices for reference frames;

[0050] (3.3) Combining frames Input to the network MOUNT and output to the network:

[0051] (3.3.1) After passing through feature extractor F f Extracting features f from keyframes k and the features of the reference frame {f r};

[0052] (3.3.2) This keyframe I k Reference frame set {I r} and the feature set {f} of the reference frame r The UMP module is used to obtain a set of reference frame features aligned with the keyframes. Its implementation is as follows:

[0053] Each reference frame I r and keyframe I k The pre-trained optical flow generator G flow The optical flow O from each reference frame to the key frame is obtained. r→k ;

[0054] Feature Extractor F f From the generated optical flow O r→k Extracting optical flow features f r→k f r→k Feature map f of the corresponding reference frame r The feature f is obtained through the concat operation. r' ;

[0055] f r' After obtaining from each reference frame I via residual block R r To keyframe I k M motion mode r→k and M r→k Add to the corresponding reference frame feature map f r In this process, a set of reference frame features aligned with the keyframe is obtained.

[0056] (3.3.3) The keyframe feature map f k and the set of aligned reference frame feature maps Several candidate boxes b are obtained through the TD-RPN network. RPN The candidate box corresponds to the diagonal σ and the feature vector f, and a predicted displacement is generated for each needle tip prior. The implementation is as follows:

[0057] (3.3.3.1) In feature map f k and Perform a sliding window operation, establishing a polar coordinate system A with the center of the sliding window as the origin; generate N for each sliding window based on the pre-defined polar angle set and polar radius set. t There are N needle tip priors, and the position of each prior is represented by its polar coordinates (μ, θ) in polar coordinate system A, where N t = (|Θ|-1)×(|P|-1)+1, where Θ = {θ} represents the set of polar angles, and P = {μ} represents the set of polar radii;

[0058] (3.3.3.2) Based on the polar coordinates (μ, θ) of the needle tip prior, a displacement (d) relative to the needle tip target is generated for each needle tip prior. x ,d y );

[0059] (3.3.3.3) This feature map f k and The position offsets of the preset prior boxes are obtained through the regression branch in the Region Candidate Network (RPN). These offsets are then used to readjust the positions of the preset prior boxes, resulting in several candidate boxes b. RPN The diagonal σ corresponding to the candidate box, and the true displacement (d) generated for each needle tip prior. x ,d y Predicted displacement like Figure 4 As shown;

[0060] (3.3.3.4) Move candidate box b RPN After passing through the RoI pooling layer, it is downsampled into a fixed-length feature vector f;

[0061] (3.3.4) The candidate box b RPN The corresponding feature vector f and the bounding box diagonal σ are processed by the AFA module in two stages: filtering and aggregation, to achieve feature aggregation of the reference frame and keyframe, and obtain the aggregated feature vector.

[0062] Reference Figure 5 The specific implementation of this step is as follows:

[0063] (3.3.4.1) Filter the reference candidates, that is, filter them according to the reference candidates. Other reference candidates Average similarity of feature vectors between Average overlap Difference in angle along the diagonal direction These three indicators are used to sort all reference candidates in ascending, ascending, and descending order, respectively. The top 10% of reference candidates in each sorting are filtered out, resulting in a reference candidate set P. r and the key candidate set P k :

[0064]

[0065] Where n'1 and n'2 represent the number of reference candidates and the number of key candidates, respectively;

[0066] (3.3.4.2) Perform feature aggregation, that is, aggregate the reference candidates according to the following formula. eigenvectors Aggregation to key candidates eigenvectors In the process, the aggregated feature vector is obtained.

[0067]

[0068] in, Indicates reference candidate Aggregated into key candidates Normalized aggregate weights;

[0069] Indicates the current reference candidate Aggregated into key candidates Aggregate weights, and Representing the reference candidates With key candidates Standardized feature similarity and overlap (IoU) between them; and Representing the reference candidates The standardized diagonal length and the standardized diagonal continuity;

[0070] Indicates other reference candidates Aggregated into key candidates Aggregate weights, and Representing the reference candidates With key candidates Standardized feature similarity and overlap (IoU) between them; and Representing the reference candidates The standardized diagonal length and the standardized diagonal continuity;

[0071] and In this context, α, β, and γ are learnable hyperparameters that satisfy α + β + γ = 1;

[0072] Represents the standardized feature similarity, where They represent the current reference candidates respectively. Other reference candidates With key candidates Feature similarity between them;

[0073] Represents the standardized overlap, where They represent the current reference candidates respectively. Other reference candidates With key candidates The degree of overlap (IoU) between them;

[0074] Represents the standardized diagonal length, where They represent the current reference candidates respectively. Other reference candidates Connect the lengths of the diagonals;

[0075] Represents the continuity of the normalized diagonal, where They represent the current reference candidates respectively. Other reference candidates Diagonal continuity;

[0076] (3.3.5) Aggregate the feature vectors The output of the MOUNT network is obtained after passing through a regressor and a classifier.

[0077] Step 4: Construct the loss function of the handheld ultrasound video probe tracking network MOUNT based on the output of the MOUNT network.

[0078] (4.1) Based on the candidate box classification probabilities predicted by the regional candidate networks TD-RPN and MOUNT, calculate the binary classification cross-entropy loss of the TD-RPN and MOUNT networks. and

[0079]

[0080] in This represents the probability that the i-th candidate box output by the TD-RPN network contains the probe. This represents the true class label corresponding to the i-th candidate box output by the TD-RPN network. This represents the probability that the i-th candidate box output by the MOUNT network contains the probe. This represents the true category label corresponding to the i-th candidate box output by the MOUNT network. The true category label has two classes: 0 and 1. If the candidate box contains a probe, the category label is 1; otherwise, the category label is 0.

[0081] (4.2) Calculate the regression loss of the TD-RPN network based on the candidate box offset vectors predicted by the regional candidate network TD-RPN and the MOUNT network. Regression loss of the MOUNT network

[0082]

[0083] in, The L1 loss function represents the smoothing of the mean absolute error.

[0084] is a vector representing the positional offset of the predicted candidate box relative to the prior box after regression of the i-th prior box in the TD-RPN network.

[0085] Is with Vectors of the same dimension represent the actual offsets of the true bounding boxes relative to the prior boxes.

[0086] This represents the number of candidate bounding boxes participating in the regression loss calculation of the TD-RPN network.

[0087] δ R These are pre-defined weight parameters used to balance the classification loss of the TD-RPN network. and regression loss

[0088] The L1 loss function represents the smoothing of the mean absolute error.

[0089] is a vector representing the positional offset of the predicted candidate box relative to the prior box after regression of the i-th prior box in the MOUNT network.

[0090] Is with Vectors of the same dimension represent the actual offsets of the true bounding boxes relative to the prior boxes.

[0091] This represents the number of candidate bounding boxes participating in the regression loss calculation of the MOUNT network.

[0092] δ M These are pre-defined weight parameters used to balance the classification loss of the MOUNT network. and regression loss

[0093] (4.3) Needle tip displacement predicted by TD-RPN network Calculate the needle tip displacement loss L tip :

[0094]

[0095] Where μ represents the polar radius corresponding to the prior of the needle tip, (d x ,d y ) represents the actual a priori displacement of the needle tip;

[0096] (4.4) Construct and calculate the mask loss L based on the classification branch output of the MOUNT model. mask :

[0097]

[0098] Where, N mask It is the number of candidate bounding boxes, y i It is the label of the i-th pixel in the real probe mask. If the pixel belongs to the probe, then y i =1, otherwise 0. It is the probability that the i-th pixel in the predicted mask is marked as a probe;

[0099] (4.5) Based on the results of steps (4.1) to (4.4), the loss function L of the MOUNT network is obtained as follows:

[0100]

[0101] Step 5: Update the parameters of the handheld ultrasound video probe network MOUNT using the stochastic gradient descent method.

[0102] Calculate the network loss function L based on the network output in step (3), and update the parameters of the MOUNT network using stochastic gradient descent:

[0103] (5.1) Calculate the gradient G of the loss function L with respect to the initial weights W and bias b. W and G b :

[0104]

[0105] (5.2) Using gradient G W and G b Update the MOUNT network parameters to obtain the updated weights W. new and bias b new :

[0106] W new =W-η·G W b new =b-η·G b

[0107] Where η represents the learning rate, which is used to control the step size of parameter updates;

[0108] Step 6: Repeat steps (3) to (5) until the training loss L of the MOUNT network converges, and obtain the trained handheld ultrasound video probe tracking network MOUNT.

[0109] Step 7: Test using the trained handheld ultrasound video probe tracking network MOUNT.

[0110] The frames of each ultrasound video in the ultrasound video test set D2 are combined and input into the trained ultrasound video probe tracking network MOUNT in temporal order to obtain the candidate result set P of the final test set. e =[(b1,m1),…(b i ,m i ...,(b n ,m n )], where b i ,m i These represent the i-th candidate box and the probe mask, respectively.

[0111] Step 8: For the predicted candidate result set P e Post-processing operations for filtering and optimization.

[0112] (8.1) The candidate result set [(b1,m1),…(b1,m1),…(b1,m1),…(b1,m1, ... i ,m i ...,(b n ,m n As input to the post-processing step, set the IoU threshold T for the post-processing process. IoU =0.5, Diagonal Direction Angle Difference Threshold T angle =10°;

[0113] (8.2) Calculate the average overlap (AvgIoU) between each candidate box and all other candidate boxes:

[0114]

[0115] in, Represents the i-th and candidate boxes b i and the j-th candidate box b j The overlap s between them, AvgIoU i represents the average overlap (IoU) between the i-th candidate box and all other candidate boxes, and n represents the number of all candidate boxes;

[0116] (8.3) Based on the predicted probe mask m i Extract the set of coordinates of the mask pixels. in, Indicates the mask m of the i-th candidate probe. i The coordinates of the j-th pixel in the image;

[0117] (8.4) Fit the line containing the probe using the least squares method, and set the equation of the line corresponding to the probe mask as follows:

[0118]

[0119] Where a i and z i Let m represent the mask of the i-th candidate probe, respectively. i The slope and intercept of the line in question;

[0120] (8.4.1) Based on the coordinates of each pixel and the model-predicted coordinates, calculate the fitting error ∈ of the line containing the probe mask:

[0121]

[0122] in Indicates the candidate probe mask m i The ordinate of the j-th pixel obtained by the model prediction;

[0123] (8.4.2) Set the fitting target as the sum of squares S of all coordinate errors:

[0124]

[0125] Where n represents the i-th candidate probe mask m i The number of pixels, S(a i ,z i () represents a straight line The sum of squared errors;

[0126] (8.4.3) Based on the sum of squared errors S, respectively, for slope a i and intercept z i Taking the partial derivative, we obtain the following formula:

[0127]

[0128] Solving the system of equations (8.4.4) and (8.4.3) simultaneously, we can obtain the mask m for each probe. i The corresponding slope a i and intercept z i ;

[0129] (8.5) Based on the solved probe mask m i The slope a of the line i The direction angle corresponding to the slope can be obtained. Calculate the orientation angle of each candidate result other candidate result orientation angles The average difference between them i :

[0130]

[0131] Where n is the number of candidate boxes;

[0132] (8.6) Based on the IoU threshold T IoU Difference threshold T between diagonal direction angle and angle Determine candidate results [(b1,m1),…(b i ,m i ...,(b n ,m n Is it reserved?

[0133] If candidate result (b) i ,m i The average overlap of AvgIoU i <T IoU Or its average difference in direction angles, diff i >T angle Then the candidate result (b) is removed. i ,m i );

[0134] If candidate result (b) i ,m i Average overlap (AvgIoU) i >T IoU And its average difference in direction angles, diff i <T angle If so, retain the candidate result and mark it as... The selected and retained candidate result set is obtained as follows:

[0135]

[0136] Where n res Indicates the number of candidate results retained;

[0137] (8.7) Based on the candidate result set P after filtering and retention res The final probe positioning frame b is obtained. u and probe mask m u :

[0138] Get all candidate bounding boxes The union of these sets yields the final probe positioning frame b. u ;

[0139] Get all candidate probe masks The intersection of these points yields the final probe mask m. u ;

[0140] According to probe mask m u The direction angle of the fitted line of the corresponding probe mask is obtained. and intercept z u ;

[0141] (8.8) Mask the probe m uThe fitted line and the positioning bounding box b u The two intersection points are used as the coordinates of the probe's entry point. and needle tip coordinates Complete probe tracking for handheld ultrasound video.

[0142] The effects of this invention can be further illustrated by the following simulation.

[0143] 1. Simulation conditions:

[0144] The hardware devices used in the simulation of this invention include an Intel Xeon(R) Gold 5120 CPU @ 2.20GHz × 8 (CPU) and an NVIDIA Tesla V100 PCIe 32GB (GPU).

[0145] The MOUNT handheld ultrasound video probe tracking network was built and trained based on MMTracking and PyTorch 1.10.0 framework.

[0146] The probe tracking performance evaluation metrics used in the simulation experiments include the tip distance error d. t The accuracy h and axial direction error of the needle tip prediction results d The specific calculation formula is as follows:

[0147]

[0148]

[0149] in Indicates the needle tip prediction result coordinates Indicates the needle tip label n t The coordinates, where N represents the number of ultrasound video frames. ∠a represents the angle between the needle axis and the positive x-axis direction in the image frame. This indicates the angle between the predicted needle axis and the positive x-axis.

[0150] 2. Simulation Content and Results

[0151] The present invention and five existing target detection methods were used for handheld ultrasonic video probe tracking, and their respective tracking performance evaluation indicators were calculated. The results are shown in Table 1.

[0152] Table 1. Calculation results of probe tracking metrics using different methods.

[0153]

[0154] As shown in Table 1, the handheld ultrasonic video tracking network MOUNT used in this invention has a smaller axial direction error E compared to the other five existing methods.d Error d between needle tip and needle tip t The median, mean, and variance all decreased significantly at the threshold T. d When the needle tip is 3mm and 5mm respectively, the hit rate h of the prediction result is significantly improved, indicating that the present invention can more effectively track handheld ultrasound video probes.

[0155] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0156] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A handheld ultrasonic video probe tracking method based on the MOUNT model, characterized in that, Includes the following steps: (1) Obtain and define the ultrasound video dataset D of the liver tumor ablation process, and divide the defined dataset into training sets in a 7:3 ratio. and test set ; (2) Constructing the handheld ultrasound video probe tracking network MOUNT: (2a) The existing Mask-RCNN network is used as the overall framework of the handheld ultrasound video probe tracking model. Based on this framework, the non-uniform motion perception UMP module and the adjacent frame aggregation AFA module are established, and the region candidate network RPN in the original Mask-RCNN network is replaced with the region candidate network TD-RPN with needle tip specific detection. (2b) The non-uniform motion sensing UMP module, the region candidate network TD-RPN with needle tip specific detection, and the adjacent frame aggregation AFA module are connected in series to form the handheld ultrasound video probe tracking network MOUNT, and the weights W and biases b of the MOUNT network are initialized. (3) Iterative training of the handheld ultrasound video probe tracking network MOUNT: (3a) The ultrasound video training set The frames of each ultrasound video are combined and input into the handheld ultrasound video probe tracking network MOUNT in temporal order to obtain the output of the network MOUNT; (3b) Construct the loss function L based on the output of the MOUNT network: (3c) Update the parameters of the MOUNT network using stochastic gradient descent; (3d) Repeat steps (3a) to (3c) to obtain the trained handheld ultrasound video probe tracking network MOUNT; (4) Test set The frames of each ultrasound video are combined and input sequentially into the trained handheld ultrasound video probe tracking network MOUNT to obtain a set of prediction results. ; (5) For the set of prediction results Post-processing operations involving filtering and optimization are performed to obtain the final probe inlet point coordinates. Tip coordinates and the angle of the needle axis direction This completes the tracking of the handheld ultrasound video probe.

2. The method according to claim 1, characterized in that, The liver tumor ablation process obtained in step (1) is super The audio-visual dataset D is defined by defining each ultrasound video in dataset D based on ablation ultrasound videos of liver tumors from several patients with different types of liver tumors. for: ， in Indicating the first in the ultrasound video The size is H's video image frame, Indicates the first in the video The probe mask annotation corresponding to the frame, It is the length of the video frame of the ultrasound video V.

3. The method according to claim 1, characterized in that, Step (2a) establishes the non-uniform motion perception (UMP) module and the adjacent frame aggregation (AFA) module within the overall framework of the handheld ultrasound video probe tracking model. The structures are as follows: The non-uniform motion sensing UMP module consists of an optical flow generator. A feature extractor and a residual block Composition, used to align reference frames and keyframes with uneven motion; The optical flow generator The existing pre-trained ARFlow model is adopted, which consists of a feature pyramid extractor, an optical flow estimation mapping layer, a cost volume layer, an optical flow prediction layer, and a context network connected in sequence. This feature extractor The existing ResNet50 network is used, which consists of an input layer, a convolutional stage, and an output layer connected in sequence. The residual block It consists of a first convolutional layer, a first batch of normalized layers (BN), a ReLU activation function, a second convolutional layer, a second batch of normalized layers (BN), a summation layer, and a ReLU activation function layer connected in sequence; The adjacent frame aggregation (AFA) module consists of an addition operator and a multiplication operator, and is used to perform weighted aggregation of reference frame features and key frame features.

4. The method according to claim 1, characterized in that, The region candidate network TD-RPN in step (2a) consists of a feature extractor. It consists of a region candidate network (RPN) connected sequentially to achieve specific detection of the needle tip region.

5. The method according to claim 1, characterized in that, Step (3a) trains the ultrasound video set Each frame combination of ultrasound video is input into the handheld ultrasound video probe tracking network MOUNT in temporal order to obtain the output of the MOUNT network, as follows: (3a1) Obtain the training set In the ultrasound video, for each ultrasound video V, let the ultrasound video frame collected at the current time be the keyframe. And set a width of before the keyframe. The frame reference window sets the selected frame in the window as the reference frame. ; (3a2) Based on the reference frame and keyframes By determining the temporal sequence of each ultrasound video, a combination of frames can be obtained. ,in It is a set of indices for reference frames; (3a3) Combine frames The input is fed into the MOUNT network, where it passes through a feature extractor. Features extracted from keyframes and features of the reference frame This keyframe Reference frame set { and the feature set of the reference frame The UMP module outputs a set of reference frame features aligned with the keyframe. }, the keyframe feature map and the set of aligned reference frame feature maps { The data is fed into the TD-RPN network, and several candidate boxes are output. The diagonal of the candidate box And the feature vector f, simultaneously generating a predicted displacement for each needle tip prior. The candidate box , corresponding eigenvectors and the diagonal of the bounding box The feature aggregation between the reference frame and the key frame is achieved through two stages: filtering and aggregation in the AFA module. The aggregated feature vector is then used to generate the feature vector. After passing through a regressor and a classifier, the output of the MOUNT network is obtained.

6. The method according to claim 1, characterized in that, In step (3b), the loss function L is constructed based on the output of the MOUNT network, as follows: ； in and These represent the binary cross-entropy loss of the TD-RPN network and the MOUNT network, respectively. and These represent the regression losses of the TD-RPN network and the MOUNT network, respectively. This represents the segmentation mask loss of the classification branch of the MOUNT network; Indicates the tip displacement loss of the TD-RPN network: ； In the formula This indicates the polar radius corresponding to the prior of the needle tip. This represents the true prior displacement of the needle tip. This represents the needle tip displacement predicted by the TD-RPN network.

7. The method according to claim 1, characterized in that, In step (3c), the opponent is subjected to stochastic gradient descent. The MOUNT ultrasound video probe tracking network was iteratively trained to achieve the following: By combining the training loss L of the handheld ultrasound video probe tracking network MOUNT, stochastic gradient descent is used to iteratively train the MOUNT network: (3c1) Calculate the gradient of the model loss function L with respect to the initial weights W and biases b in step (3b). and : ， ； (3c2) Using gradient and Update MOUNT network parameters: ， ； in, This represents the learning rate, which controls the step size for parameter updates. and These represent the updated model weights and biases, respectively.

8. The method according to claim 1, characterized in that, In step (5), the set of prediction results is... Screening The optimized post-processing operation is implemented as follows: (8a) The set of candidate results output by the handheld ultrasound video probe tracking network MOUNT As input to the post-processing step, set the IoU threshold for the overlap during post-processing. and the threshold for diagonal direction angle difference ; (8b) Calculate the average overlap (AvgIoU) of each candidate box with all other candidate boxes: ； in, Represents the i-th candidate box and the j-th candidate box The degree of overlap between them represents the average overlap (IoU) between the i-th candidate box and all other candidate boxes, and n represents the number of all candidate boxes; (8c) Based on the predicted probe mask Extract the set of coordinates of the mask pixels. ,in, Represents the mask of the i-th candidate probe. The coordinates of the j-th pixel in the image; (8d) Fit the line containing the probe using the least squares method, and set the equation of the line corresponding to the probe mask as follows: ； in and These represent the masks of the i-th candidate probe. The slope and intercept of the line in question; (8e) Based on the solution The direction angle corresponding to the slope can be obtained. Calculate the orientation angle of each candidate result. other candidate result orientation angles Average difference between : ； Where n is the number of candidate boxes; (8f) Based on the IoU threshold Difference threshold between diagonal direction angles Determine candidate results Whether to retain: If candidate Average overlap Or the average difference in its direction angle If so, then the candidate result will be removed. ; Conversely, if the candidate Average overlap And its average direction angle difference If so, retain the candidate result and mark it as ( ); Finally, the selected and retained candidate result set is obtained. ,in Indicates the number of candidate results retained; (8g) Get all candidate bounding boxes The union of these sets yields the final probe positioning frame. Get all candidate probe masks The intersection of these points yields the final probe mask. And the direction angle of the corresponding probe mask fitting line. and intercept ; (8h) Mask the probe The fitted line and the positioning bounding box The two intersection points are used as the coordinates of the probe's entry point. and needle tip coordinates .

9. The method according to claim 5, characterized in that, The keyframe described in step (3a3) Reference frame set { and the feature set of the reference frame The UMP module outputs a set of reference frame features aligned with the keyframe. The implementation is as follows: Each reference frame and keyframes Pre-trained optical flow generator Obtain the optical flow from each reference frame to the key frame. ; Feature extractor From the generated optical flow Extracting optical flow features , Feature map of corresponding reference frame Features are obtained through the concat operation. ; The residual block R is obtained from each reference frame. To keyframe sports mode and will Add to the corresponding reference frame feature map In the process, a set of reference frame features aligned with the keyframe is obtained. } 10. The method according to claim 5, characterized in that, The keyframe feature map described in step (3a3) and the set of aligned reference frame feature maps { The TD-RPN network outputs several candidate boxes. The diagonal of the candidate box And the feature vector f, simultaneously generating predicted displacements for each needle tip prior. The implementation is as follows: exist and{ A sliding window operation is performed on the [database name], establishing a polar coordinate system A with the center of the sliding window as the origin; a [database name] is generated for each sliding window based on a pre-defined set of polar angles and polar radii. Each needle tip prior is located using its polar coordinates in polar coordinate system A. To indicate, among which , Represents the set of polar angles. Represents the set of polar radii; According to the polar coordinates of the needle tip prior For each needle tip prior, a true displacement relative to the needle tip target is generated. ; This feature map and{ The position offset of the preset prior box is obtained through the regression branch in the Region Candidate Network (RPN). ,use Several candidate boxes are obtained by readjusting the positions of the preset prior boxes. diagonal of the candidate box Simultaneously, the a priori true displacement of each needle tip is obtained. Corresponding predicted displacement ; Candidate boxes After passing through the RoI pooling layer, the data is downsampled into a fixed-length feature vector. .

11. The method according to claim 5, characterized in that, The candidate box described in step (3a3) , corresponding eigenvectors and the diagonal of the bounding box The AFA module performs two stages—filtering and aggregation—to aggregate features from reference frames and keyframes, outputting the aggregated feature vector. The implementation is as follows: Filter the reference candidates, that is, based on the reference candidates Other reference candidates Average similarity of feature vectors between Average overlap Difference in angle along the diagonal direction These three metrics are used to sort all reference candidates in ascending, ascending, and descending order, respectively. The top 10% of reference candidates in each sorting are then filtered out, resulting in the reference candidate set. and key candidate set : ； ； in and These represent the number of reference candidates and the number of key candidates, respectively. Perform feature aggregation, that is, aggregate the reference candidates according to the following formula. eigenvectors Aggregation to key candidates eigenvectors In the process, the aggregated feature vector is obtained. : ； in, Indicates reference candidate Aggregated into key candidates Normalized aggregate weights; Indicates the current reference candidate Aggregated into key candidates Aggregate weights, Representing the reference candidates With key candidates Standardized feature similarity and overlap (IoU) between them; Representing the reference candidates Standardized diagonal length and standardized diagonal continuity; Indicates other reference candidates Aggregated into key candidates Aggregate weights, Indicates reference candidate With key candidates Standardized feature similarity and overlap (IoU) between them; and Representing the reference candidates Standardized diagonal length and standardized diagonal continuity; and In , For learnable hyperparameters, satisfying ; Represents the standardized feature similarity, where They represent the current reference candidates respectively. Other reference candidates With key candidates Feature similarity between them; Represents the standardized overlap, where They represent the current reference candidates respectively. Other reference candidates With key candidates The degree of overlap (IoU) between them; Indicates the standardized diagonal length. They represent the current reference candidates respectively. Other reference candidates The length of the diagonal; Indicates the continuity of the standardized diagonal. They represent the current reference candidates respectively. Other reference candidates The diagonal continuity.

12. The method according to claim 8, characterized in that, probe mask in step (8e) The corresponding straight line slope in and intercept The solution process includes the following: (8e1) Set the probe mask The straight-line fitting error is the difference between the ordinate of each pixel and the ordinate predicted by the fitted straight line. : ； in Represents the mask of the i-th candidate probe. The coordinates of the j-th pixel in the image; The ordinate represents the predicted value of the fitted line; (8e2) Set the fitting objective as the sum of squares S of all coordinate errors: ； Where n represents the i-th candidate probe mask. The number of pixels, Represents a straight line The sum of squared coordinate errors; (8e3) Based on the sum of squared errors S, respectively, the slope and intercept Taking the partial derivative, we obtain the following formula: ； ； Solving the system of equations (8e4) and (8e3) simultaneously yields the mask for each probe. Corresponding slope and intercept .

Images