Puncture planning method and system based on prostate multi-modal imaging and storage medium

Abstract

The present application relates to the technical field of medical imaging processing, in particular to a puncture planning method and system based on prostate multi-modal imaging and a storage medium, comprising: planning a first path with a prostate cancer region as a target point in prostate multi-modal imaging; acquiring an ultrasound image of a puncture needle in a running process according to the first path in real time; detecting and segmenting the puncture needle in the ultrasound image and determining positioning information of a needle tip of the puncture needle; registering the first path to the ultrasound image in the prostate multi-modal imaging and determining correction information of the puncture needle running process by using the first path and the positioning information of the needle tip of the puncture needle. The present application plans a puncture path of prostate cancer on prostate multi-modal imaging, realizes providing guiding information for puncture, and accurately positions a puncture needle shape and positioning information in a puncture guiding process through ultrasound guiding, so as to accurately guide or correct information in a puncture process by using a planned path.

Description

Technical Field

[0001] This invention relates to the field of medical imaging processing technology, specifically to a puncture planning method, system, and storage medium based on prostate multimodal imaging. Background Technology

[0002] Current medical lesion identification largely relies on imaging methods, such as magnetic resonance imaging (MRI), ultrasound, and photoacoustic imaging, to identify lesions. Taking prostate lesion identification as an example, MRI, ultrasound, and photoacoustic imaging of the prostate area are acquired and combined for multimodal identification of prostate lesions, improving the accuracy of lesion identification. Correspondingly, these images need to be registered and fused to accurately display prostate cancer. Simultaneously, the multimodal imaging information of prostate cancer provides guidance for prostate biopsy procedures.

[0003] Therefore, needle guidance is crucial during prostate lesion biopsy, as it directly affects the biopsy outcome.

[0004] Currently, it is relatively difficult to directly guide the puncture needle using multimodal imaging of prostate lesions without a field of view, as the information is mixed and it is difficult to directly grasp the effective information related to puncture needle guidance. Summary of the Invention

[0005] The purpose of this invention is to provide a puncture planning method, system, and storage medium based on prostate multimodal imaging, in order to solve the technical problem that it is relatively difficult to directly guide the puncture needle using prostate lesion multimodal imaging in the absence of a field of view, and that the information is mixed, making it difficult to directly grasp the effective information related to puncture needle guidance.

[0006] To solve the above-mentioned technical problems, the present invention specifically provides the following technical solution:

[0007] A puncture planning method based on prostate multimodal imaging includes the following steps:

[0008] Acquire microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate region;

[0009] The medical image processing software 3D Slicer was used to register and fuse microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging to obtain prostate multimodal imaging.

[0010] In prostate multimodal imaging, prostate cancer regions were marked, and a first path was planned with prostate cancer regions as the target.

[0011] Real-time acquisition of ultrasound images of the puncture needle traveling along the first path;

[0012] The puncture needle is detected and segmented in the ultrasound image, and the positioning information of the puncture needle tip is determined.

[0013] The first path is registered in the prostate multimodal imaging to the ultrasound image, and the positioning information of the puncture needle tip is corrected in real time in the registered ultrasound image according to the first path, and displayed in the ultrasound image.

[0014] As a preferred embodiment of the present invention, the method for marking prostate cancer regions in prostate multimodal imaging includes:

[0015] The U-Net network was used to segment the prostate cancer region and the surrounding tissue region in prostate multimodal imaging.

[0016] As a preferred embodiment of the present invention, the method for planning the first path includes:

[0017] The multiple optimization objectives for determining the first path include:

[0018] Maximize the distance between the needle tip and the surrounding tissue area , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of each point on the contour of the j-th surrounding tissue region, n be the total number of points on the first path, and k be the total number of surrounding tissue regions. This is the Euclidean distance expression, where max is the maximization identifier;

[0019] Maximizing the overlap between the needle tip and the prostate cancer area , In the formula, The curve is obtained by fitting the needle tip to the i-th to n-th path points in the first path. The straight line obtained by fitting the maximum diameter of the prostate cancer region. Similarity operator;

[0020] Minimize the travel distance of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. The coordinates of the (i+1)th path point of the puncture needle tip in the first path are given, and min is the minimizer identifier.

[0021] Smooth puncture path of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of the (i+1)th path point of the puncture needle tip in the first path. The coordinates of the (i-1)th path point of the puncture needle tip in the first path;

[0022] Using the prostate region and transrectal region as the solution space, a multi-objective optimization algorithm is employed to solve the problem. , , and The first path is obtained by optimization and then smoothed.

[0023] As a preferred embodiment of the present invention, the method for detecting and segmenting the puncture needle in the ultrasound image and determining the positioning information of the puncture needle tip includes:

[0024] The YOLO3 algorithm is used to pre-position the puncture needle in the ultrasound image to obtain the puncture needle bounding box;

[0025] An improved U-Net network was used to segment the puncture needle in the ultrasound image within the puncture needle bounding box, resulting in a segmented binary image of the puncture needle.

[0026] Linear regression was used to locate the needle tip in the segmented binary image of the puncture needle, and the position coordinates of the needle tip were obtained.

[0027] As a preferred embodiment of the present invention, the method for determining the puncture needle boundary frame includes:

[0028] The ultrasound image is input into the YOLO3 algorithm to obtain the puncture needle bounding box;

[0029] When all sides of the puncture needle boundary frame are smaller than the preset size, a square frame of the preset size is formed by expanding outward from the center point of the puncture needle boundary frame, which serves as the corrected puncture needle boundary frame. At the same time, the ultrasound image in the square frame is the ultrasound image in the puncture needle boundary frame.

[0030] If at least one side of the puncture needle bounding box is larger than the preset size, a square frame is formed with the center point of the puncture needle bounding box as the center point and the longest side as the side length. The square frame and the ultrasound image within the square frame are simultaneously scaled to the preset size to serve as the corrected puncture needle bounding box and the ultrasound image within the puncture needle bounding box.

[0031] As a preferred embodiment of the present invention, the improved U-Net network consists of 50 convolutional layers, including 9 stages, namely 4 shrinking paths, 1 center path, and 4 expanding paths;

[0032] At the beginning of each stage, a 1×1 convolutional layer is added to perform pooling operations on the input feature map channels;

[0033] The contraction path follows a 1×1 convolutional layer with two residual blocks connected by jumps. The latter residual blocks are subjected to a 2×2 max pooling operation with a stride of 2. Each residual block consists of two 3×3 convolutional kernels, and a rectified linear unit (ReLU) is connected after each convolutional kernel.

[0034] The expansion path first upsamples the input feature map, then performs a 2×2 convolution to halve the number of feature channels, and then connects the output feature map to the corresponding feature map of the contraction path.

[0035] As a preferred embodiment of the present invention, the method for locating the needle tip in a segmented binary image of a puncture needle using linear regression includes:

[0036] Linear fitting was performed on the puncture needle to obtain the puncture needle fitting curve. ;

[0037] Obtain the pixel coordinates of the puncture needle in the segmented binary image. , Let m be the pixel coordinates of the l-th point on the puncture needle, and m be the total number of pixels on the puncture needle. Using the least squares method, a curve is formed on the fitted curve of the puncture needle. , , and Solution function , ;

[0038] Will Convert to matrix form to obtain ;

[0039] in, , , ;

[0040] Solving using matrix calculus Minimum point obtained ,according to Seeking and the result Substitute this into the fitting curve of the puncture needle;

[0041] Criterion functions for searching the needle tip position on the fitted curve of the puncture needle include:

[0042] Criteria for decrease in strength at the tip of a puncture needle ;

[0043] Gradient descent criterion of puncture needle tip ;

[0044] Puncture needle size guidelines ;

[0045] In the formula, These are the coordinates of the needle tip's position. The coordinates of the starting point of the puncture needle on the fitted curve of the puncture needle are given. The coordinates of the endpoint of the puncture needle on the fitted curve are given. The coordinates of the point located behind the needle tip on the curve fitted to the puncture needle, pointing towards the start of the curve. Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle The gradient magnitude at point L is the length of the puncture needle, max is the maximization identifier, and min is the minimization identifier. This is the Euclidean distance calculation formula;

[0046] The criterion function is solved by fitting the curve of the puncture needle, and the position coordinates of the puncture needle tip are obtained.

[0047] As a preferred embodiment of the present invention, the puncture needle fitting curve is located on arrive The local lines between the lines are superimposed onto the ultrasound image within the puncture needle boundary to obtain an ultrasound image enhanced by the puncture needle. This ultrasound image enhanced by the puncture needle is then registered and fused with prostate multimodal imaging.

[0048] As a preferred embodiment of the present invention, the present invention provides a puncture planning system based on prostate multimodal imaging, applied to a puncture planning method based on prostate multimodal imaging, the system comprising:

[0049] The first data processing unit is used to acquire microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging of the same prostate site. The medical image processing software 3D Slicer is used to register and fuse the microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging to obtain prostate multimodal imaging.

[0050] The path planning unit is used to mark the prostate cancer region in prostate multimodal imaging and plan the first path with the prostate cancer region as the target point.

[0051] An ultrasound guidance unit is used to acquire ultrasound images in real time as the puncture needle travels along the first path.

[0052] The second data processing unit is used to detect and segment the puncture needle in the ultrasound image and determine the positioning information of the puncture needle tip.

[0053] The registration correction unit is used to register the first path from the prostate multimodal imaging to the ultrasound image, and to perform real-time correction on the registered ultrasound image according to the positioning information of the puncture needle tip based on the first path, and display it in the ultrasound image.

[0054] As a preferred embodiment of the present invention, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement a puncture planning method based on prostate multimodal imaging.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] This invention first uses prostate multimodal imaging to accurately display the details of the prostate cancer area, and then plans the puncture path for prostate cancer on the prostate multimodal imaging, thereby providing guidance information for the puncture and improving the accuracy of the puncture path planning. During the puncture process, ultrasound guidance is used to accurately locate the shape and positioning information of the puncture needle, thereby mastering the relative position between the puncture needle tip and the guiding path, so as to use the planned path to provide precise guidance or correction information for the puncture process. Attached Figure Description

[0057] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0058] Figure 1 This is a flowchart of the guided puncture method provided in an embodiment of the present invention;

[0059] Figure 2 This is a block diagram of a guided puncture system provided in an embodiment of the present invention;

[0060] Figure 3This is a flowchart of the puncture needle detection, segmentation, and needle tip positioning provided in an embodiment of the present invention;

[0061] Figure 4 The diagram shows the results of needle detection, segmentation, and needle tip positioning provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] like Figure 1 As shown, this invention provides a puncture planning method based on prostate multimodal imaging, comprising the following steps:

[0064] Acquire microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate region;

[0065] The medical image processing software 3D Slicer was used to register and fuse microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging to obtain prostate multimodal imaging.

[0066] In prostate multimodal imaging, marking the prostate cancer region and planning the first path with the prostate cancer region as the target point is essentially a planning path to guide the puncture needle.

[0067] Real-time ultrasound images are acquired as the puncture needle travels along the first path. These ultrasound images are typically acquired using a transrectal ultrasound probe.

[0068] The puncture needle is detected and segmented in the ultrasound image, and the positioning information of the puncture needle tip is determined.

[0069] The first path is registered in the prostate multimodal imaging to the ultrasound image, and the correction information for the puncture needle travel process is determined in the registered ultrasound image using the positioning information of the first path and the puncture needle tip.

[0070] In this embodiment, the following data is mainly acquired:

[0071] The first data source is multimodal imaging data of prostate tissue. Object analysis is performed to obtain the condition of the target tissue, and the puncture path of the puncture needle is planned based on the condition of the tissue.

[0072] The second data is the travel data of the puncture needle. Essentially, this involves the analysis and processing of image data during the travel process to understand the deviation between the needle tip and the planned path. This allows for the design of a guidance method with correction capabilities for the puncture needle. This method can be based on a constructed human body model or a model of the entire prostate area, or other tissue models (which also use surrounding tissues as obstacles to plan the puncture path from the needle insertion point to the target point).

[0073] In addition, this method can also be applied to mechanical operation path planning in other situations where there is no field of vision.

[0074] In this invention, a preoperative analysis was first performed to obtain the individual structure of the puncture subject, and then the optimal guiding path for transrectal puncture of prostate cancer was determined based on the individual structure of the puncture subject.

[0075] After determining the optimal guidance path adapted to the individual structure, this invention uses it to guide the puncture needle during the puncture procedure. In addition, it combines intraoperative ultrasound guidance to obtain intraoperative ultrasound images. The puncture needle is segmented in the ultrasound images, and the needle tip position is located. This allows it to determine whether the puncture needle has deviated from the planned first path. Based on the deviation shown in the image, it provides travel correction information for the deviated puncture needle.

[0076] This invention acquires multimodal imaging of prostate cancer during pre-procedure analysis, namely, fused and registered magnetic resonance imaging (MRI), photoacoustic imaging, and ultrasound imaging. This allows for precise visualization of details within the prostate cancer region. Path planning based on multimodal imaging yields high-quality path planning results. This invention also segments the prostate cancer region based on multimodal imaging to understand individual circumstances, as detailed below:

[0077] Methods for marking prostate cancer regions using prostate multimodal imaging include:

[0078] The U-Net network was used to segment the prostate cancer region and surrounding tissue regions (urethra, ejaculatory duct, bladder, etc.) in prostate multimodal imaging.

[0079] After completing the organizational structure segmentation, this invention performs path planning based on the segmentation results, as follows:

[0080] The planning methods for the first path include:

[0081] The multiple optimization objectives for determining the first path include:

[0082] Maximize the distance between the needle tip and the surrounding tissue area , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of each point on the contour of the j-th surrounding tissue region, n be the total number of points on the first path, and k be the total number of surrounding tissue regions. This is the Euclidean distance expression, where max is the maximization identifier;

[0083] Target It characterizes the safety optimization goal, so that the puncture needle avoids the surrounding tissue area as much as possible during the puncture process, thereby ensuring safety.

[0084] Maximizing the overlap between the needle tip and the prostate cancer area , In the formula, The curve is obtained by fitting the needle tip to the i-th to n-th path points in the first path. The straight line obtained by fitting the maximum diameter of the prostate cancer region. Similarity operator;

[0085] Target This characterizes the sampling effect optimization target, so that the puncture needle overlaps with the prostate cancer area as much as possible during the puncture process, thereby ensuring the maximum acquisition of prostate cancer tissue.

[0086] Minimize the travel distance of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. The coordinates of the (i+1)th path point of the puncture needle tip in the first path are given, and min is the minimizer identifier.

[0087] Target It represents the goal of distance optimization, making the puncture path as short as possible and the puncture process as fast as possible to ensure puncture efficiency.

[0088] Smooth puncture path of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of the (i+1)th path point of the puncture needle tip in the first path. The coordinates of the (i-1)th path point of the puncture needle tip in the first path;

[0089] Target The target for optimizing path smoothness is characterized, so that the puncture process can reach the prostate cancer area as smoothly as possible, avoiding unnecessary damage and ensuring path continuity and safety.

[0090] Using the prostate region and transrectal region as the solution space, a multi-objective optimization algorithm is employed to solve the problem. , , and The first path is obtained by optimization and then smoothed.

[0091] This invention is achieved through , , and Planning the puncture path can yield a safe, smooth, and efficient path.

[0092] This invention utilizes ultrasound guidance during the puncture process, enabling the segmentation of the puncture needle within the ultrasound image, acquisition of the needle's shape, and localization of the needle tip position, as detailed below:

[0093] like Figure 3 and Figure 4 As shown, the method for detecting and segmenting the puncture needle in the ultrasound image and determining the positioning information of the puncture needle tip includes:

[0094] The YOLO3 algorithm is used to pre-position the puncture needle in the ultrasound image to obtain the puncture needle bounding box;

[0095] An improved U-Net network was used to segment the puncture needle in the ultrasound image within the puncture needle bounding box, resulting in a segmented binary image of the puncture needle.

[0096] Linear regression was used to locate the needle tip in the segmented binary image of the puncture needle, and the position coordinates of the needle tip were obtained.

[0097] Methods for determining the bounding box of the puncture needle include:

[0098] The ultrasound image is input into the YOLO3 algorithm to obtain the puncture needle bounding box;

[0099] The size and proportion of the bounding box of the region of interest determined by YOLO depend on the size of the pin and the insertion depth. Because the segmentation framework used requires the image to be of a preset size, such as 256×256 pixels, the probability of obtaining a bounding box of this size in practice is low. To solve this problem, this invention proposes a correction technique, the implementation of which is as follows:

[0100] When all sides of the puncture needle boundary frame are smaller than the preset size, a square frame of the preset size is formed by expanding outward from the center point of the puncture needle boundary frame, which serves as the corrected puncture needle boundary frame. At the same time, the ultrasound image in the square frame is the ultrasound image in the puncture needle boundary frame.

[0101] If at least one side of the puncture needle bounding box is larger than the preset size, a square frame is formed with the center point of the puncture needle bounding box as the center point and the longest side as the side length. The square frame and the ultrasound image within the square frame are simultaneously scaled to the preset size to serve as the corrected puncture needle bounding box and the ultrasound image within the puncture needle bounding box.

[0102] In other words, in this invention, the small frame processing uses the center point of the original bounding box as the anchor point and directly expands outward to generate a 256x256 pixel frame, with the original small bounding box located in the center of the new frame.

[0103] The large bounding box processing uses the center point of the original bounding box as the anchor point. First, a minimum square that can completely contain the original ROI (with a side length equal to the long side of the original box) is constructed. Then, this square area is scaled to 256x256 pixels.

[0104] This correction ensures that a bounding box image of the preset size is obtained regardless of the original bounding box size, without the loss of needle pixels.

[0105] The improved U-Net network consists of 50 convolutional layers, including 9 stages: 4 shrinking paths, 1 center path, and 4 expanding paths.

[0106] The entire network consists of 50 convolutional layers, resulting in a significant speed reduction compared to the general U-Net model. This speed reduction is due to the increased computational cost. To address this issue, a 1×1 convolutional layer is added at the beginning of each stage to perform pooling operations on the input feature map channels;

[0107] The contraction path follows a 1×1 convolutional layer with two residual blocks connected by jumps. The latter residual blocks are subjected to a 2×2 max pooling operation with a stride of 2. Each residual block consists of two 3×3 convolutional kernels, and a rectified linear unit (ReLU) is connected after each convolutional kernel.

[0108] The expansion path first upsamples the input feature map, then performs a 2×2 convolution to halve the number of feature channels, and then connects the output feature map to the corresponding feature map of the contraction path.

[0109] This invention utilizes the least squares method to linearly fit the segmentation results of the puncture needle. After obtaining the fitted curve of the puncture needle, the tip position of the puncture needle can be located, and the information can be enhanced on the puncture needle segmentation results to highlight the morphology of the puncture needle, as detailed below:

[0110] Methods for locating the needle tip in a segmented binary image of a puncture needle using linear regression include:

[0111] Linear fitting was performed on the puncture needle to obtain the puncture needle fitting curve. ;

[0112] Obtain the pixel coordinates of the puncture needle in the segmented binary image. , Let m be the pixel coordinates of the l-th point on the puncture needle, and m be the total number of pixels on the puncture needle. Using the least squares method, a curve is formed on the fitted curve of the puncture needle. , , and Solution function , ;

[0113] Will Convert to matrix form to obtain ;

[0114] in, , , ;

[0115] Solving using matrix calculus Minimum point obtained ,according to Seeking and the result Substitute this into the fitting curve of the puncture needle;

[0116] The present invention determines the fitting curve of the puncture needle through the above calculation process, and the needle tip position can be found by following the fitting curve, as follows:

[0117] Criterion functions for searching the needle tip position on the fitted curve of the puncture needle include:

[0118] Criteria for decrease in strength at the tip of a puncture needle ;

[0119] A significant decrease in grayscale intensity occurs at the tip of the puncture needle compared to the needle body. Furthermore, as the needle tip extends forward along the fitted curve, a similar significant decrease in grayscale intensity occurs due to the disappearance of the needle tip. Therefore, this invention establishes an intensity reduction criterion to search for solutions in the needle tip region. The coordinates of the endpoint of the puncture needle on the fitted curve are equivalent to the point on the puncture needle in the image that is closest to the needle tip. Therefore, the needle tip will be... Appeared nearby Maximize, expect to find the same The point where the strength drops significantly is the needle tip. To maximize the search, the intensity of the needle tip should decrease significantly between the found point and the point further extended along the fitted curve of the puncture needle. This further verifies the accuracy of the needle tip search. In other words, there should be a difference in grayscale intensity between the found needle tip and its preceding and following positions. Maximizing this implies that there should be a significant difference in grayscale between the positions in front of and behind the needle tip, constraining the first two terms. and This avoids misjudging the position of the needle tip, which could lead to inaccurate positioning results.

[0120] Gradient descent criterion of puncture needle tip ;

[0121] Similarly, a significant decrease in image gradient occurs at the tip of the puncture needle compared to the needle body. Furthermore, as the needle tip extends forward along the fitted curve, a similar significant decrease in image gradient occurs due to the disappearance of the needle tip. Therefore, this invention establishes a gradient descent criterion to search for solutions in the needle tip region. The coordinates of the endpoint of the puncture needle on the fitted curve are equivalent to the point on the puncture needle in the image that is closest to the needle tip. Therefore, the needle tip will be... Appeared nearby Maximize, expect to find the same There is a point where the gradient decreases significantly, which is the location of the needle tip. To maximize the desired gradient, the gradient between the found needle tip and the point further extended along the fitted curve of the puncture needle should decrease significantly. This further verifies the accuracy of the needle tip search; that is, there should be a difference in image gradient between the found needle tip and its preceding and following positions. Maximizing this indicates that there is a significant gradient difference between the positions in front of and behind the needle tip, constraining the first two terms. and This avoids misjudging the position of the needle tip, or in other words, random errors.

[0122] in, , where horizontal gradient Vertical gradient It is calculated using the Sobel operator.

[0123] Puncture needle size guidelines ;

[0124] This invention also utilizes the inherent length L of the puncture needle to set a size criterion. The coordinates of the starting point of the puncture needle on the fitted curve are equivalent to the needle tail. Therefore, the distance between the needle tip and the needle tail should be equal to the inherent length of the puncture needle. For size criteria.

[0125] This invention utilizes the above three criteria to optimize the solution on the fitted curve, thereby obtaining the position coordinates of the needle tip and realizing needle tip positioning.

[0126] In the formula, These are the coordinates of the needle tip's position. The coordinates of the starting point of the puncture needle on the fitted curve of the puncture needle are given. The coordinates of the endpoint of the puncture needle on the fitted curve are given. The coordinates of the point located behind the needle tip on the curve fitted to the puncture needle, pointing towards the start of the curve. Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle The gradient magnitude at point L is the length of the puncture needle, max is the maximization identifier, and min is the minimization identifier. This is the Euclidean distance calculation formula.

[0127] The criterion function is solved by fitting the curve of the puncture needle, and the position coordinates of the puncture needle tip are obtained.

[0128] The curve fitted by the puncture needle is located on arrive The local lines between the lines are superimposed onto the ultrasound image within the puncture needle boundary to obtain an ultrasound image enhanced by the puncture needle. This ultrasound image enhanced by the puncture needle is then registered and fused with prostate multimodal imaging.

[0129] Methods for registering a first path from medical images to ultrasound images include:

[0130] Using multimodal imaging as the reference image and ultrasound image as the floating image, the multimodal imaging and ultrasound image are registered and fused to obtain a registered ultrasound image containing the first path.

[0131] When the position coordinates of the puncture needle tip are on the first path, the next path point is used as guiding information to guide the puncture needle tip to continue puncturing.

[0132] When the position coordinates of the puncture needle tip deviate from the first path, the path point that is closest to the puncture needle tip in the first path is used as correction information to correct the offset part in the puncture process, so that it is restored to the first path and continues to move.

[0133] like Figure 2 As shown, this invention provides a prostate multimodal imaging-based puncture planning system, applied to a prostate multimodal imaging-based puncture planning method. The system includes:

[0134] The first data processing unit is used to acquire microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging of the same prostate site. The medical image processing software 3D Slicer is used to register and fuse the microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging to obtain prostate multimodal imaging.

[0135] The path planning unit is used to mark the prostate cancer region in prostate multimodal imaging and plan the first path with the prostate cancer region as the target point.

[0136] An ultrasound guidance unit is used to acquire ultrasound images in real time as the puncture needle travels along the first path.

[0137] The second data processing unit is used to detect and segment the puncture needle in the ultrasound image and determine the positioning information of the puncture needle tip.

[0138] The registration correction unit is used to register the first path from the prostate multimodal imaging to the ultrasound image, and to determine the correction information of the puncture needle travel process in the registered ultrasound image using the positioning information of the first path and the puncture needle tip.

[0139] The present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement a puncture planning method based on prostate multimodal imaging.

[0140] This invention first uses prostate multimodal imaging to accurately display the details of the prostate cancer area, and then plans the puncture path for prostate cancer on the prostate multimodal imaging, thereby providing guidance information for the puncture and improving the accuracy of the puncture path planning. During the puncture guidance process, ultrasound guidance is used to accurately locate the shape and positioning information of the puncture needle, thereby mastering the relative position between the puncture needle tip and the guiding path, so as to use the planned path to provide precise guidance or correction information for the puncture process.

[0141] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.

Claims

1. A puncture planning system based on prostate multimodal imaging, characterized in that, The system includes: The first data processing unit is used to acquire microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging of the same prostate site. The medical image processing software 3D Slicer is used to register and fuse the microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging to obtain prostate multimodal imaging. The path planning unit is used to mark the prostate cancer region in prostate multimodal imaging and plan the first path with the prostate cancer region as the target point. An ultrasound guidance unit is used to acquire ultrasound images in real time as the puncture needle travels along the first path. The second data processing unit is used to detect and segment the puncture needle in the ultrasound image and determine the positioning information of the puncture needle tip. The registration correction unit is used to register the first path from the prostate multimodal imaging to the ultrasound image, and to determine the correction information of the puncture needle travel process in the registered ultrasound image using the positioning information of the first path and the puncture needle tip. The planning method for the first path includes: The multiple optimization objectives for determining the first path include: Maximize the distance between the needle tip and the surrounding tissue area , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of each point on the contour of the j-th surrounding tissue region, n be the total number of points on the first path, and k be the total number of surrounding tissue regions. This is the Euclidean distance expression, where max is the maximization identifier; Maximizing the overlap between the needle tip and the prostate cancer area , In the formula, The curve is obtained by fitting the needle tip to the i-th to n-th path points in the first path. The straight line obtained by fitting the maximum diameter of the prostate cancer region. Similarity operator; Minimize the travel distance of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. The coordinates of the (i+1)th path point of the puncture needle tip in the first path are given, and min is the minimizer identifier. Smooth puncture path of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of the (i+1)th path point of the puncture needle tip in the first path. The coordinates of the (i-1)th path point of the puncture needle tip in the first path; Using the prostate region and transrectal region as the solution space, a multi-objective optimization algorithm is employed to solve the problem. , , and The first path is obtained by optimization and then smoothed.

2. The puncture planning system based on prostate multimodal imaging according to claim 1, characterized in that: Methods for marking prostate cancer regions using prostate multimodal imaging include: The U-Net network was used to segment the prostate cancer region and surrounding tissue region in prostate multimodal imaging.

3. The puncture planning system based on prostate multimodal imaging according to claim 1, characterized in that: The method for detecting and segmenting the puncture needle in the ultrasound image and determining the positioning information of the puncture needle tip includes: The YOLO3 algorithm is used to pre-position the puncture needle in the ultrasound image to obtain the puncture needle bounding box; An improved U-Net network was used to segment the puncture needle in the ultrasound image within the puncture needle bounding box, resulting in a segmented binary image of the puncture needle. Linear regression was used to locate the needle tip in the segmented binary image of the puncture needle, and the position coordinates of the needle tip were obtained.

4. The puncture planning system based on prostate multimodal imaging according to claim 3, characterized in that: The method for determining the bounding box of the puncture needle includes: The ultrasound image is input into the YOLO3 algorithm to obtain the bounding box of the puncture needle; When all sides of the puncture needle boundary frame are smaller than the preset size, a square frame of the preset size is formed by expanding outward from the center point of the puncture needle boundary frame, which serves as the corrected puncture needle boundary frame. At the same time, the ultrasound image in the square frame is the ultrasound image in the puncture needle boundary frame. If at least one side of the puncture needle bounding box is larger than the preset size, a square frame is formed with the center point of the puncture needle bounding box as the center point and the longest side as the side length. The square frame and the ultrasound image within the square frame are simultaneously scaled to the preset size to serve as the corrected puncture needle bounding box and the ultrasound image within the puncture needle bounding box.

5. A puncture planning system based on prostate multimodal imaging according to claim 4, characterized in that: The improved U-Net network consists of 50 convolutional layers, comprising 9 stages, including 4 shrinking paths, 1 centering path, and 4 expanding paths. At the beginning of each stage, a 1×1 convolutional layer is added to perform pooling operations on the input feature map channels; The contraction path follows a 1×1 convolutional layer with two residual blocks connected by jumps. The subsequent residual blocks are subjected to a 2×2 max pooling operation with a stride of 2. Each residual block consists of two 3×3 convolutional kernels, and a rectified linear unit (ReLU) is connected after each convolutional kernel. The extended path first upsamples the input feature map, then performs a 2×2 convolution to halve the number of feature channels, and connects the output feature map to the corresponding feature map of the contracted path.

6. The puncture planning system based on prostate multimodal imaging according to claim 4, characterized in that: Methods for locating the needle tip in a segmented binary image of a puncture needle using linear regression include: Linear fitting was performed on the puncture needle to obtain the puncture needle fitting curve. ; Obtain the pixel coordinates of the puncture needle in the segmented binary image. , Let m be the pixel coordinates of the l-th point on the puncture needle, and m be the total number of pixels on the puncture needle. Using the least squares method, a curve is formed on the fitted curve of the puncture needle. , , and Solution function , ; Will Convert to matrix form to obtain ; in, , , ; Solving using matrix calculus Minimum point obtained ,according to Seeking and the result Substitute this into the fitting curve of the puncture needle; Criterion functions for searching the needle tip position based on the fitted curve of the puncture needle include: Criteria for decrease in strength at the tip of a puncture needle ; Gradient descent criterion of puncture needle tip ; Puncture needle size guidelines ; In the formula, These are the coordinates of the needle tip's position. The coordinates of the starting point of the puncture needle on the fitted curve of the puncture needle are given. The coordinates of the endpoint of the puncture needle on the fitted curve are given. The coordinates of the point located behind the needle tip on the curve fitted to the puncture needle, pointing towards the start of the curve. Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle The gradient magnitude at point L is the length of the puncture needle, max is the maximization identifier, and min is the minimization identifier. This is the Euclidean distance calculation formula; The criterion function is solved by fitting the curve of the puncture needle, and the position coordinates of the puncture needle tip are obtained.

7. A puncture planning system based on prostate multimodal imaging according to claim 6, characterized in that: The curve fitted by the puncture needle is located on arrive The local lines between the lines are superimposed onto the ultrasound image within the puncture needle boundary to obtain an ultrasound image enhanced by the puncture needle. This ultrasound image enhanced by the puncture needle is then registered and fused with prostate multimodal imaging.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, it implements a puncture planning method based on prostate multimodal imaging, including the following steps: Acquire microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate region; The medical image processing software 3D Slicer was used to register and fuse microvascular photoacoustic imaging, ultrasound elastography and magnetic resonance imaging to obtain prostate multimodal imaging. In prostate multimodal imaging, prostate cancer regions were marked, and a first path was planned with prostate cancer regions as the target. Real-time acquisition of ultrasound images of the puncture needle traveling along the first path; The puncture needle is detected and segmented in the ultrasound image, and the positioning information of the puncture needle tip is determined. The first path is registered from the prostate multimodal imaging to the ultrasound image, and the correction information for the puncture needle travel process is determined in the registered ultrasound image using the positioning information of the first path and the puncture needle tip. The planning method for the first path includes: The multiple optimization objectives for determining the first path include: Maximize the distance between the needle tip and the surrounding tissue area , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of each point on the contour of the j-th surrounding tissue region, n be the total number of points on the first path, and k be the total number of surrounding tissue regions. This is the Euclidean distance expression, where max is the maximization identifier; Maximizing the overlap between the needle tip and the prostate cancer area , In the formula, The curve is obtained by fitting the needle tip to the i-th to n-th path points in the first path. The straight line obtained by fitting the maximum diameter of the prostate cancer region. Similarity operator; Minimize the travel distance of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. The coordinates of the (i+1)th path point of the puncture needle tip in the first path are given, and min is the minimizer identifier. Smooth puncture path of the puncture needle tip , In the formula, Let be the coordinates of the i-th point on the first path of the puncture needle tip. Let be the coordinates of the (i+1)th path point of the puncture needle tip in the first path. The coordinates of the (i-1)th path point of the puncture needle tip in the first path; Using the prostate region and transrectal region as the solution space, a multi-objective optimization algorithm is employed to solve the problem. , , and The first path is obtained by optimization and then smoothed.

9. A computer-readable storage medium according to claim 8, characterized in that: The method for detecting and segmenting the puncture needle in the ultrasound image and determining the positioning information of the puncture needle tip includes: The YOLO3 algorithm is used to pre-position the puncture needle in the ultrasound image to obtain the puncture needle bounding box; An improved U-Net network was used to segment the puncture needle in the ultrasound image within the puncture needle bounding box, resulting in a segmented binary image of the puncture needle. Linear regression was used to locate the needle tip in the segmented binary image of the puncture needle, and the position coordinates of the needle tip were obtained. The method for determining the bounding box of the puncture needle includes: The ultrasound image is input into the YOLO3 algorithm to obtain the bounding box of the puncture needle; When all sides of the puncture needle boundary frame are smaller than the preset size, a square frame of the preset size is formed by expanding outward from the center point of the puncture needle boundary frame, which serves as the corrected puncture needle boundary frame. At the same time, the ultrasound image in the square frame is the ultrasound image in the puncture needle boundary frame. If at least one side of the puncture needle bounding box is larger than the preset size, a square frame is formed with the center point of the puncture needle bounding box as the center point and the longest side as the side length. The square frame and the ultrasound image in the square frame are simultaneously scaled to the preset size to serve as the corrected puncture needle bounding box and the ultrasound image in the puncture needle bounding box. The improved U-Net network consists of 50 convolutional layers, comprising 9 stages, including 4 shrinking paths, 1 centering path, and 4 expanding paths. At the beginning of each stage, a 1×1 convolutional layer is added to perform pooling operations on the input feature map channels; The contraction path follows a 1×1 convolutional layer with two residual blocks connected by jumps. The subsequent residual blocks are subjected to a 2×2 max pooling operation with a stride of 2. Each residual block consists of two 3×3 convolutional kernels, and a rectified linear unit (ReLU) is connected after each convolutional kernel. The extended path first upsamples the input feature map, then performs a 2×2 convolution to halve the number of feature channels, and connects the output feature map to the corresponding feature map of the contracted path.

10. A computer-readable storage medium according to claim 9, characterized in that: Methods for locating the needle tip in a segmented binary image of a puncture needle using linear regression include: Linear fitting was performed on the puncture needle to obtain the puncture needle fitting curve. ; Obtain the pixel coordinates of the puncture needle in the segmented binary image. , Let m be the pixel coordinates of the l-th point on the puncture needle, and m be the total number of pixels on the puncture needle. Using the least squares method, a curve is formed on the fitted curve of the puncture needle. , , and Solution function , ; Will Convert to matrix form to obtain ; in, , , ; Solving using matrix calculus Minimum point obtained ,according to Seeking and the result Substitute this into the fitting curve of the puncture needle; Criterion functions for searching the needle tip position based on the fitted curve of the puncture needle include: Criteria for decrease in strength at the tip of a puncture needle ; Gradient descent criterion of puncture needle tip ; Puncture needle size guidelines ; In the formula, These are the coordinates of the needle tip's position. The coordinates of the starting point of the puncture needle on the fitted curve of the puncture needle are given. The coordinates of the endpoint of the puncture needle on the fitted curve are given. The coordinates of the point located behind the needle tip on the curve fitted to the puncture needle, pointing towards the start of the curve. Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gray intensity at that location Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle gradient magnitude at that point Segmenting the binary image of the puncture needle The gradient magnitude at point L is the length of the puncture needle, max is the maximization identifier, and min is the minimization identifier. This is the Euclidean distance calculation formula; The criterion function is solved by fitting the curve of the puncture needle to obtain the position coordinates of the puncture needle tip; The curve fitted by the puncture needle is located on arrive The local lines between the lines are superimposed onto the ultrasound image within the puncture needle boundary to obtain an ultrasound image enhanced by the puncture needle. This ultrasound image enhanced by the puncture needle is then registered and fused with prostate multimodal imaging.

Images