A method for recognizing a puncture needle pose in a three-dimensional ultrasound image

Abstract

This invention relates to the field of image processing technology, specifically to a method for recognizing the pose of a puncture needle in a three-dimensional ultrasound image. The method includes the following steps: using a preoperative MRI image as a reference image and an intraoperative ultrasound image as a floating image, image registration is performed to obtain a registration deformation field and an ultrasound registration image; a puncture needle segmentation model guided by the registration deformation field is used to segment the puncture needle on the ultrasound registration image to obtain the puncture needle segmentation result. This invention first registers the preoperative MRI and intraoperative ultrasound images to construct a spatially consistent multimodal image foundation. Then, based on this, a segmentation model incorporating a long short-term memory self-attention mechanism is designed to achieve collaborative perception of local details and global structure. Furthermore, an attention weighting mechanism guided by the registration deformation field is introduced, enabling the network to adaptively focus on key regions based on physical deformation information, achieving high-precision and efficient segmentation of the puncture needle in the ultrasound image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for recognizing the position of a puncture needle in a three-dimensional ultrasound image. Background Technology

[0002] Three-dimensional medical image segmentation has always been a key technical challenge in precision medicine. In prostate biopsy navigation, preoperative MRI images can provide a clear view of the prostate's anatomical structure, but they cannot display the instrument position in real time during the procedure. Intraoperative ultrasound images, while capable of real-time imaging, suffer from poor image quality, severe noise interference, and significant spatial differences and anatomical deformations between preoperative MRI and intraoperative ultrasound images due to patient positioning changes and organ deformation. Therefore, current methods relying solely on intraoperative ultrasound images for needle segmentation are limited by the inherent limitations of ultrasound image quality and the lack of accurate prior knowledge of anatomical structures.

[0003] Moreover, while existing CNN-based puncture needle segmentation methods (such as U-Net and NestedUNet) perform well in local feature extraction, they are difficult to effectively model long-range dependencies, which limits their understanding of the overall shape and spatial position of the puncture needle, making it difficult to achieve high-precision segmentation of the puncture needle in ultrasound images. Summary of the Invention

[0004] The purpose of this invention is to provide a method for recognizing the pose of a puncture needle in a three-dimensional ultrasound image, in order to solve the technical problem that the existing technology is difficult to effectively model long-range dependencies, which limits the understanding of the overall shape and spatial position of the puncture needle, and thus makes it difficult to achieve high-precision segmentation of the puncture needle in the ultrasound image.

[0005] To solve the above-mentioned technical problems, the present invention specifically provides the following technical solution: A method for recognizing the pose of a puncture needle in a three-dimensional ultrasound image includes the following steps: Obtain MRI images of the prostate before the biopsy and ultrasound images during the procedure; Using preoperative MRI images as reference images and intraoperative ultrasound images as floating images, image registration was performed to obtain the registration deformation field and ultrasound registration images. The puncture needle segmentation model guided by the registration deformation field was used to segment the puncture needle on the ultrasound registration image, and the puncture needle segmentation results were obtained.

[0006] As a preferred embodiment of the present invention, the method for acquiring the registration deformation field and the ultrasonic registration image includes: Preoperative MRI images and intraoperative ultrasound images are input into a multimodal registration model used to generate the registration deformation field, and the multimodal registration model outputs the registration deformation field. The intraoperative ultrasound images are spatially transformed based on the registration deformation field to obtain the ultrasound registration images.

[0007] As a preferred embodiment of the present invention, the method for obtaining the puncture needle segmentation result includes: The ultrasound-registered image is input into the puncture needle segmentation model, and the puncture needle segmentation model outputs the puncture needle segmentation result.

[0008] As a preferred embodiment of the present invention, the method for constructing the puncture needle segmentation model includes: Acquire ultrasound registration images with segmentation labels, and perform dataset augmentation and pixel value normalization to obtain the training set; On the training set, a deep convolutional network is trained to obtain a puncture needle segmentation model that takes ultrasound registration images as input and puncture needle segmentation results as output. The loss function of the puncture needle segmentation model From cross-entropy L CE It is composed of a weighted sum of the Dice coefficients; in, ; ; In the formula, Let Dice coefficient be the value of category j. , and Let represent the number of true positives, false negatives, and false positives for category j, respectively, and N represent the number of segmentation categories. If the true category of ultrasound registration image sample i is j, then... ,otherwise , M represents the predicted probability that ultrasound registration image sample i belongs to category j, where M represents the total number of samples. This is a hyperparameter.

[0009] As a preferred embodiment of the present invention, the network structure of the deep convolutional network includes a feature extraction module, a deformation field guidance module, a convolutional layer branch, a short-term self-attention branch, a long-term self-attention module, and a reconstruction module. The feature extraction module extracts features from the ultrasound registration image through multi-layer convolution to obtain a multi-channel feature map; The convolutional layer branch contains four repeated convolutional layers. The four repeated convolutional layers continuously extract different levels of features from shallow to deep in the multi-channel feature map, providing the first stage of memory features and the current features from the first to the third stages for the short-term self-attention branch. The deformation field guidance module updates the memory features of the second and third stages in the short-term self-attention branch based on the registration deformation field. The short-term self-attention branch includes a three-stage short-term self-attention module. In the first to third stages, important features from different levels of features are extracted and retained through self-attention calculations on memory features and current features. The long-term self-attention module performs self-attention calculation on the merged features of the important features of the three stages in the short-term self-attention branch, and extracts and retains high-level semantic features; The reconstruction module reconstructs the high-level semantic features to obtain the puncture needle segmentation result.

[0010] As a preferred embodiment of the present invention, the feature output by the first convolutional layer in the convolutional layer branch is used as the memory feature of the first-stage short-term self-attention module, and the feature output by the second convolutional layer in the convolutional layer branch is used as the current feature of the first-stage short-term self-attention module. The features output by the third convolutional layer in the convolutional layer branch are used as the current features of the second-stage short-term self-attention module, and the features output by the first-stage short-term self-attention module are used as the memory features of the second-stage short-term self-attention module. The features output by the fourth convolutional layer in the convolutional layer branch are used as the current features of the third-stage short-term self-attention module, and the features output by the second-stage short-term self-attention module are the memory features of the third-stage short-term self-attention module.

[0011] As a preferred embodiment of the present invention, the method for the deformation field guiding module to update the memory features of the second and third stages includes: Calculate the amplitude of the registration deformation field Register the deformation field direction and the gradient magnitude of the registered deformation field. ,in, , , These are the registration deformation fields. Components in the x, y, z directions It is the transpose operator. For the Euclidean norm, It is the Frobenius norm. It is a very small constant; based on , as well as Forming coding features ,in, This involves concatenating operations along the channel dimension. It is a 3D convolution; Encoding features Multi-scale feature fusion is performed sequentially. And the attention weights are calculated to obtain the spatial attention map. ,in, For upsampling operations, , They are respectively Features downsampled by 2 or 4 times It is the sigmoid activation function; Spatial attention map Memory characteristics of the second and third stages By weighting, the updated memory features of the second and third stages are obtained. ,in, For Hadamard products.

[0012] As a preferred embodiment of the present invention, the self-attention operation expression of the short-term self-attention module in the three stages is as follows: ; Among them, when , ; ; when , ; , ; In the formula, These are the two input items for the short-term self-attention module in the l-th stage. This is the output of the short-term self-attention module in the l-th stage. For the middle term, For multi-head attention computation, As The Query value in for The key and value in the data. For layer normalization operations, This is for feedforward neural network operations.

[0013] As a preferred embodiment of the present invention, the self-attention operation expression of the long-term self-attention module is as follows: ; in, ; In the formula, These are the two inputs to the long-term self-attention module. For the output of the long-term self-attention module, For the middle term, For multi-head attention computation, As The Query value in for The key and value in the data. For layer normalization operations, This is for feedforward neural network operations.

[0014] As a preferred embodiment of the present invention, the evaluation metrics of the puncture needle segmentation model include the Dice coefficient, computation speed (FPS), and model complexity (FLOPs).

[0015] Compared with the prior art, the present invention has the following advantages: This invention first registers preoperative MRI and intraoperative ultrasound images to construct a spatially consistent multimodal image foundation. Then, based on this, a segmentation model integrating long short-term memory self-attention mechanism is designed to achieve collaborative perception of local details and global structure. Furthermore, an attention weighting mechanism guided by registration deformation field is introduced, enabling the network to adaptively focus on key areas based on physical deformation information, thus achieving high-precision and efficient segmentation of the puncture needle in ultrasound images. Attached Figure Description

[0016] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0017] Figure 1 This is a flowchart of a method for segmenting puncture needles in ultrasound images provided in an embodiment of the present invention; Figure 2 This is a segmentation model of a puncture needle provided in an embodiment of the present invention; Figure 3 This invention provides a segmentation model for a puncture needle lacking registration deformation field guidance. Figure 4 The network structure of the long short-term attention module provided in the embodiments of the present invention; Figure 5 The feature extraction module, convolutional layer branch, and reconstruction module provided in this embodiment of the invention; Figure 6 This is a 2D comparison image of the puncture needle segmentation results of each model provided in the embodiments of the present invention; Figure 7 This is a 3D comparison image of the puncture needle segmentation results of each model provided in the embodiments of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, the present invention provides a method for recognizing the pose of a puncture needle in a three-dimensional ultrasound image, comprising the following steps: Obtain MRI images of the prostate before the biopsy and ultrasound images during the procedure; Using preoperative MRI images as reference images and intraoperative ultrasound images as floating images, image registration was performed to obtain the registration deformation field and ultrasound registration images. The puncture needle segmentation model guided by the registration deformation field was used to segment the puncture needle on the ultrasound registration image, and the puncture needle segmentation results were obtained.

[0020] Methods for acquiring registration deformation fields and ultrasonic registration images include: Preoperative MRI images and intraoperative ultrasound images are input into a multimodal registration model used to generate the registration deformation field, and the multimodal registration model outputs the registration deformation field. The intraoperative ultrasound images are spatially transformed based on the registration deformation field to obtain the ultrasound registration images.

[0021] To mitigate the limitation of puncture needle segmentation accuracy caused by the lack of accurate anatomical priors in ultrasound images, this invention employs registration of intraoperative ultrasound images with preoperative MRI images. The registered ultrasound images and MRI images share the same anatomical background, fusing structural information from MRI with real-time ultrasound information. This facilitates the differentiation of the puncture needle from surrounding tissues, improves segmentation robustness, and allows the tissue deformation field estimated during registration to guide subsequent puncture needle segmentation models to focus on areas of significant deformation, enhancing adaptability to needle bending and positional changes.

[0022] This invention combines long and short-term self-attention mechanisms in a puncture needle segmentation model. The short-term self-attention mechanism fuses features from the current layer with those from the previous layer at each stage, enabling dynamic interaction and retention of important information between layers. It also integrates output features from all stages, modeling global dependencies across layers and enhancing the understanding of the overall morphology of the puncture needle. Therefore, the short-term self-attention module captures local details, while the long-term self-attention module models the global structure, creating a complementary effect that improves the accuracy of puncture needle segmentation.

[0023] In this invention, the registration deformation field reflects the displacement and degree of deformation of the tissue during the registration process. Its amplitude, direction and gradient information can be used as spatial priors. Then, a spatial attention map is generated by three-dimensional convolution and multi-scale fusion. The memory features in the short-term self-attention module are weighted to guide the network to focus on the deformed areas. This realizes the addition of the physical information of the registration deformation field to the long-term and short-term self-attention mechanisms to guide the short-term attention mechanism. This gives the attention mechanism clear anatomical significance, rather than simply relying on the appearance of the image, and ensures that the prediction of the puncture needle segmentation at the needle tip and needle body boundary is more accurate.

[0024] Methods for obtaining the results of puncture needle segmentation include: The ultrasound-registered image is input into the puncture needle segmentation model, and the puncture needle segmentation model outputs the puncture needle segmentation result.

[0025] Methods for constructing puncture needle segmentation models include: Acquire ultrasound registration images with segmentation labels, and perform dataset augmentation and pixel value normalization to obtain the training set; On the training set, a deep convolutional network is trained to obtain a puncture needle segmentation model that takes ultrasound registration images as input and puncture needle segmentation results as output. Loss function of puncture needle segmentation model From cross-entropy L CE It is composed of a weighted sum of the Dice coefficients; in, ; ; In the formula, Let Dice coefficient be the value of category j. , and Let represent the number of true positives, false negatives, and false positives for category j, respectively, and N represent the number of segmentation categories. If the true category of ultrasound registration image sample i is j, then... ,otherwise , M represents the predicted probability that ultrasound registration image sample i belongs to category j, where M represents the total number of samples. This is a hyperparameter.

[0026] The Dice coefficient is a commonly used segmentation loss, defined as the ratio of the intersection of the predicted result and the true value of the predicted region to its sum. While Dice can intuitively reflect the similarity between two regions, it is difficult to determine the segmentation accuracy of each pixel. Therefore, limiting pixel-level errors through cross-entropy can help improve model accuracy. The gradient of cross-entropy increases with increasing error, thus its convergence speed is fast, avoiding the gradient vanishing problem caused by activation function saturation. Therefore, the loss function LCE+Dice in this invention is a weighted sum of cross-entropy LCE and the Dice coefficient, where the weights are determined by hyperparameters. Confirmed. Since a Dice value closer to 1 indicates better performance.

[0027] The network structure of a deep convolutional network includes a feature extraction module, a deformation field guidance module, convolutional layer branches, short-term self-attention branches, long-term self-attention modules, and a reconstruction module; like Figure 5 The feature extraction module extracts features from the ultrasound registration image through multi-layer convolution to obtain a multi-channel feature map; The convolutional layer branch contains four repeated convolutional layers. These four repeated convolutional layers continuously extract different levels of features from shallow to deep in the multi-channel feature map, providing the first-stage memory features and the current features from the first to the third stages for the short-term self-attention branch. The deformation field guidance module guides the updating of memory features in the second and third stages of the short-term self-attention branch based on the registration deformation field; The short-term self-attention branch includes a three-stage short-term self-attention module. In the first to third stages, important features at different levels are extracted and retained through self-attention calculations on memory features and current features. The long-term self-attention module performs self-attention calculation on the merged features of the three stages in the short-term self-attention branch, extracting and retaining high-level semantic features; The reconstruction module reconstructs the high-level semantic features to obtain the puncture needle segmentation results.

[0028] The features output by the first convolutional layer in the convolutional layer branch are used as the memory features of the first-stage short-term self-attention module, and the features output by the second convolutional layer in the convolutional layer branch are used as the current features of the first-stage short-term self-attention module. The features output by the third convolutional layer in the convolutional layer branch are used as the current features of the second-stage short-term self-attention module, and the features output by the first-stage short-term self-attention module are used as the memory features of the second-stage short-term self-attention module. The features output by the fourth convolutional layer in the convolutional layer branch are used as the current features of the third-stage short-term self-attention module, and the features output by the second-stage short-term self-attention module are the memory features of the third-stage short-term self-attention module.

[0029] The backbone of the deep convolutional network is a convolutional structure composed of multiple convolutional layers connected in series. First, a feature extraction module extracts image features through multiple convolutions, converting the medical image into a multi-channel feature map and reducing its resolution to one-quarter of the original size. Then, features are continuously extracted through repeated convolutional layers (Conv Layers), maintaining the resolution of the feature map throughout the process. Another parallel short-term memory self-attention branch requires two inputs at each stage, called the memory feature and the current feature, which are the outputs of the corresponding convolutional layer for each stage. The first short-term memory self-attention module selects the output of the first convolutional layer as the memory feature, and each subsequent stage uses the computation result of the previous stage as the memory feature, updating it by registering the deformation field to obtain new memory features, thus focusing on regions with significant deformation.

[0030] In each stage of short-term self-attention, the output of the short-term self-attention module is merged with the current feature, and the resolution of the remembered feature is further reduced to one-eighth of its original size to reduce computation and speed up the inference process. Simultaneously, the number of channels in the self-attention branch is doubled to retain sufficient feature information. Downsampling and channel number transformation of the remembered feature are accomplished through convolution. Channel expansion and compression of interactive features are achieved through 1×1×1 convolutions, such as... Figure 2 and Figure 3 As shown by the red dashed arrows in the diagram. The outputs of each short-term memory self-attention module are merged along the channel dimension and used as the input to the long-term memory self-attention module, which then generates the segmentation result through the reconstruction module.

[0031] Methods for updating the memory features of the second and third stages guided by the deformation field include: Calculate the amplitude of the registration deformation field Register the deformation field direction and the gradient magnitude of the registered deformation field. ,in, , , These are the registration deformation fields. Components in the x, y, z directions It is the transpose operator. For the Euclidean norm, It is the Frobenius norm. It is a very small constant; based on , as well as Forming coding features ,in, This involves concatenating operations along the channel dimension. It is a 3D convolution; Encoding features Multi-scale feature fusion is performed sequentially. And the attention weights are calculated to obtain the spatial attention map. ,in, For upsampling operations, , They are respectively Features downsampled by 2 or 4 times It is the sigmoid activation function; Spatial attention map Memory characteristics of the second and third stages By weighting, the updated memory features of the second and third stages are obtained. ,in, For Hadamard products.

[0032] The short-term and long-term memory self-attention modules of this invention have the same structure as the decoder in a Transformer network, such as... Figure 4 As shown, the calculation process can be represented as follows: ; Here, Inter represents the intermediate variable of the module, FFN and Norm represent the feedforward network and LayerNorm function, respectively, and residual structures are used in both stages. During self-attention computation, Input1 is treated as the Query, and Input2 as the Key and Value. The two inputs of the long-term memory self-attention module are identical, and the output resolution is the same as that of Input2. The correlation between deep and shallow features is extracted and combined through a unique short-term memory self-attention module to continuously preserve important features during branch propagation. The self-attention mechanism allows this combination to focus on the entire image rather than local structures, which is beneficial for capturing various high-level semantic features. The long-term memory self-attention module achieves the fusion of different stages, and the comprehensive learning of features at different levels can make the segmentation results more accurate. Moreover, residual connections and layer normalization ensure training stability and information flow.

[0033] The above calculation process is mapped to the short-term and long-term self-attention processes, and the specific operations are as follows: The self-attention operation expression for the three-stage short-term self-attention module is as follows: ; Among them, when , ; ; when , ; , ; In the formula, These are the two input items for the short-term self-attention module in the l-th stage. This is the output of the short-term self-attention module in the l-th stage. For the middle term, For multi-head attention computation, As The Query value in for The key and value in the data. For layer normalization operations, This is for feedforward neural network operations.

[0034] The self-attention operation expression for the long-term self-attention module is: ; in, ; In the formula, These are the two inputs to the long-term self-attention module. For the output of the long-term self-attention module, For the middle term, For multi-head attention computation, As The Query value in for The key and value in the data. For layer normalization operations, This is for feedforward neural network operations.

[0035] Evaluation metrics for puncture needle segmentation models include Dice coefficient, computational speed (FPS), and model complexity (FLOPs).

[0036] This invention utilizes a short-term self-attention module to progressively fuse shallow and deep features across multiple stages, preserving detailed information while leveraging high-level semantics. A long-term self-attention module integrates features from all stages, achieving cross-level global context modeling. Short-term memory allows the network to transfer and update important features between stages, avoiding information loss, while long-term memory captures key features throughout the network, enhancing the model's understanding of complex structures. This further enables CNN branches to extract local features, and the self-attention mechanism captures long-range dependencies, establishing a complementary effect between global and local features. Ultimately, through short-term and long-term attention mechanisms, the model establishes dependencies at multiple scales, improving segmentation accuracy. Furthermore, by using the registration deformation field as prior knowledge, attention is guided to focus on areas of significant deformation, enhancing the ability to locate the puncture needle.

[0037] Compared with the traditional Transformer network structure, the puncture needle segmentation model of this invention reduces computational complexity through staged self-attention calculation, resulting in low computational cost, fast inference speed, and suitability for real-time applications.

[0038] In this invention, during network training, random weights are used to initialize the model, and the initial learning rate is set to 0.0001. The model optimizer is Adam, with weight decay of 0.0003, smoothing constants of 0.9 and 0.999, and the learning rate is automatically optimized using the CosineAnnealingLR function. Hyperparameters in the loss function... The initialization parameter was set to 0.5, the maximum number of epochs was set to 100, and the batch size was 16. We compared the proposed segmentation algorithm with U-Net, Att-Unet, NestedUNet, TransBTS, and UNETR segmentation models, training it multiple times with different initialization parameters and training sample order, and selecting the model parameters with the best accuracy as the final evaluation.

[0039] Six beagle dogs undergoing prostate biopsies had their prostates analyzed using a 3D probe, resulting in 10 video sequences containing a total of 443 3D ultrasound images. Before training, the 443 images underwent preprocessing. The process included: first, image cropping with the assistance of a veterinarian and cropping tools to ensure the bounding box was large enough to encompass the entire needle; then, the cropped images were resized to a uniform 96×96×96 pixels, and a senior urologist with ten years of prostate biopsy experience marked the needle body on each section containing the needle in the 443 cropped images. 400 images were randomly selected from the 443 raw datasets for training, with the remaining 43 used for testing. During training, online enhancement operations such as rotation, scaling, translation, and flipping were performed on the obtained gold standard images for needle segmentation and localization, as well as the raw images.

[0040] The Dice metric was used to measure the segmentation accuracy of deep learning models for puncture needles. Considering the impact of computational speed and resource consumption on application deployment, the computational speed (FPS) and model complexity (FLOPs) of different models were also analyzed. The comparison results are shown in Table 1 below.

[0041] Table 1. Puncture needle segmentation and comparison of different deep learning models Wherein, invention 1 refers to the puncture needle segmentation model proposed in this invention, such as Figure 3 As shown, the present invention 2 refers to the lack of a registration deformation field-guided short-term attention mechanism compared to the puncture needle segmentation model of the present invention, such as... Figure 2As shown, U-Net refers to needle segmentation using the U-Net network structure; Att-Unet refers to needle segmentation by adding an attention mechanism to the U-Net network structure; NestedUNet refers to the U-Net++ network structure formed by reconstructing the encoder-decoder path of U-Net with dense skip connections; TransBTS refers to a medical image segmentation framework for needle segmentation that combines a 3D convolutional neural network (CNN) with a Transformer model; and UNETR refers to a 3D medical image segmentation model for needle segmentation based on the Transformer architecture.

[0042] As can be seen from the comparison results in Table 1, the puncture needle segmentation model proposed in this invention achieves the highest puncture needle segmentation accuracy (Dice 0.736) compared to other segmentation models without the addition of registration deformation field guidance. After incorporating registration deformation field attention guidance, it further achieves an accuracy of 0.782. This demonstrates that the addition of the long short-term attention mechanism effectively improves puncture needle segmentation accuracy, and this improvement is further enhanced with the addition of the registration deformation field guidance strategy. Moreover, the puncture needle segmentation model in this invention possesses a computational speed that meets the requirements of real-time applications, along with significantly lower computational and parameter requirements. The above comparisons demonstrate the superiority of the proposed method in terms of both segmentation accuracy and efficiency.

[0043] Figure 6 and Figure 7 The visualization results of our method and the comparison method are presented in 2D and 3D modes respectively. Figure 6 The methods of invention 1 and invention 2 are shown to be more accurate in detecting both ends of the puncture needle. Figure 6 In the image, a, b, c, and d represent different image samples, where blue, green, and cyan represent false negative, false positive, and true positive regions, respectively. Figure 7 This further demonstrates that the methods of invention 1 and invention 2 provide prediction results for puncture needle morphology that are closer to the gold standard. Figure 7 In the figure, ag represents the gold standard, and the 3D segmentation results of this invention 1, this invention 2, U-Net, Att-Unet, NestedUNet, TransBTS, and UNETR, respectively.

[0044] This invention first registers preoperative MRI and intraoperative ultrasound images to construct a spatially consistent multimodal image foundation. Then, based on this, a segmentation model integrating long short-term memory self-attention mechanism is designed to achieve collaborative perception of local details and global structure. Furthermore, an attention weighting mechanism guided by registration deformation field is introduced, enabling the network to adaptively focus on key areas based on physical deformation information, thus achieving high-precision and efficient segmentation of the puncture needle in ultrasound images.

[0045] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A method for recognizing the pose of a puncture needle in a three-dimensional ultrasound image, characterized in that, Includes the following steps: Obtain MRI images of the prostate before the biopsy and ultrasound images during the procedure; Using preoperative MRI images as reference images and intraoperative ultrasound images as floating images, image registration was performed to obtain the registration deformation field and ultrasound registration images. The puncture needle segmentation model guided by the registration deformation field was used to segment the puncture needle on the ultrasound registration image, and the puncture needle segmentation results were obtained.

2. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 1, characterized in that: The methods for acquiring the registration deformation field and the ultrasonic registration image include: Preoperative MRI images and intraoperative ultrasound images are input into a multimodal registration model used to generate the registration deformation field, and the multimodal registration model outputs the registration deformation field. The intraoperative ultrasound images are spatially transformed based on the registration deformation field to obtain the ultrasound registration images.

3. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 2, characterized in that: The method for obtaining the puncture needle segmentation result includes: The ultrasound-registered image is input into the puncture needle segmentation model, and the puncture needle segmentation model outputs the puncture needle segmentation result.

4. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 3, characterized in that: The method for constructing the puncture needle segmentation model includes: Acquire ultrasound registration images with segmentation labels, and perform dataset augmentation and pixel value normalization to obtain the training set; On the training set, a deep convolutional network is trained to obtain a puncture needle segmentation model that takes ultrasound registration images as input and puncture needle segmentation results as output. The loss function of the puncture needle segmentation model From cross-entropy L CE It is composed of a weighted sum of the Dice coefficients; in, ; ； In the formula, Let Dice coefficient be the value of category j. , and Let represent the number of true positives, false negatives, and false positives for category j, respectively, and N represent the number of segmentation categories. If the true category of ultrasound registration image sample i is j, then... ,otherwise , M represents the predicted probability that ultrasound registration image sample i belongs to category j, where M represents the total number of samples. This is a hyperparameter.

5. The method for recognizing the puncture needle position in a three-dimensional ultrasound image according to claim 4, characterized in that: The network structure of the deep convolutional network includes a feature extraction module, a deformation field guidance module, convolutional layer branches, a short-term self-attention branch, a long-term self-attention module, and a reconstruction module. The feature extraction module extracts features from the ultrasound registration image through multi-layer convolution to obtain a multi-channel feature map; The convolutional layer branch contains four repeated convolutional layers. The four repeated convolutional layers continuously extract different levels of features from shallow to deep in the multi-channel feature map, providing the first stage of memory features and the current features from the first to the third stages for the short-term self-attention branch. The deformation field guidance module updates the memory features of the second and third stages in the short-term self-attention branch based on the registration deformation field. The short-term self-attention branch includes a three-stage short-term self-attention module. In the first to third stages, important features from different levels of features are extracted and retained through self-attention calculations on memory features and current features. The long-term self-attention module performs self-attention calculation on the merged features of the important features in the three stages of the short-term self-attention branch, and extracts and retains high-level semantic features; The reconstruction module reconstructs the high-level semantic features to obtain the puncture needle segmentation result.

6. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 5, characterized in that: The features output by the first convolutional layer in the convolutional layer branch are used as the memory features of the first-stage short-term self-attention module, and the features output by the second convolutional layer in the convolutional layer branch are used as the current features of the first-stage short-term self-attention module. The features output by the third convolutional layer in the convolutional layer branch are used as the current features of the second-stage short-term self-attention module, and the features output by the first-stage short-term self-attention module are used as the memory features of the second-stage short-term self-attention module. The features output by the fourth convolutional layer in the convolutional layer branch are used as the current features of the third-stage short-term self-attention module, and the features output by the second-stage short-term self-attention module are the memory features of the third-stage short-term self-attention module.

7. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 6, characterized in that: The method for updating the memory features of the second and third stages by the deformation field-guided module includes: Calculate the amplitude of the registration deformation field Register the deformation field direction and the gradient magnitude of the registered deformation field. ,in, , , These are the registration deformation fields. Components in the x, y, z directions It is the transpose operator. For the Euclidean norm, It is the Frobenius norm. It is a very small constant; based on , as well as Forming coding features ,in, This involves concatenating operations along the channel dimension. It is a 3D convolution; Encoding features Multi-scale feature fusion is performed sequentially. And the attention weights are calculated to obtain the spatial attention map. ,in, For upsampling operations, , They are respectively Features downsampled by 2 or 4 times It is the sigmoid activation function; Spatial attention map Memory characteristics of the second and third stages By weighting, the updated memory features of the second and third stages are obtained. ,in, For Hadamard products.

8. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 7, characterized in that: The self-attention operation expression for the short-term self-attention module in the three stages is as follows: ； Among them, when , ; ; when , ; , ; In the formula, These are the two input items for the short-term self-attention module in the l-th stage. This is the output of the short-term self-attention module in the l-th stage. For the middle term, For multi-head attention computation, As The Query value in for The key and value in the data. For layer normalization operations, This is for feedforward neural network operations.

9. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 8, characterized in that: The self-attention operation expression for the long-term self-attention module is: ； in, ; In the formula, These are the two inputs to the long-term self-attention module. For the output of the long-term self-attention module, For the middle term, For multi-head attention computation, As The Query value in for The key and value in the data. For layer normalization operations, This is for feedforward neural network operations.

10. The method for recognizing the position of a puncture needle in a three-dimensional ultrasound image according to claim 9, characterized in that: The evaluation metrics for the puncture needle segmentation model include the Dice coefficient, computation speed (FPS), and model complexity (FLOPs).

Images