Puncture planning method and system based on prostate multi-modal imaging and storage medium

Abstract

The invention relates to the technical field of medical imaging processing, in particular to a puncture planning method and system based on prostate multi-modal imaging and a storage medium, and the method comprises the steps: planning a first path taking a prostate cancer area as a target spot in prostate multi-modal imaging; an ultrasonic image of the puncture needle in the advancing process according to the first path is obtained in real time; detecting and segmenting the puncture needle in the ultrasonic image, and determining the positioning information of the tip of the puncture needle; the first path is registered into an ultrasonic image in the prostate multi-mode imaging, and correction information of the advancing process of the puncture needle is determined according to the positioning information of the first path and the needle point of the puncture needle. The prostate cancer puncture path is planned on the basis of prostate multi-mode imaging, guide information is provided for puncture, and the form and positioning information of the puncture needle are accurately positioned through ultrasonic guide in the puncture guide process, so that the planned path is utilized to provide accurate guide or correction information for the puncture process.

Description

Technical Field

[0001] The present invention relates to the technical field of medical imaging processing, and in particular to a puncture planning method, system and storage medium based on prostate multimodal imaging. Background Art

[0002] Currently, the identification of medical lesions relies heavily on imaging methods, specifically through the use of medical images such as magnetic resonance imaging, ultrasound, and photoacoustic imaging to identify lesions. For example, the identification of prostate lesions involves acquiring magnetic resonance imaging, ultrasound imaging, and photoacoustic imaging of the prostate, and combining these images for multimodal identification of prostate lesions to improve accuracy. Accordingly, these images need to be registered and fused to accurately display prostate cancer. Furthermore, multimodal imaging information from prostate cancer provides guidance for prostate cancer puncture procedures.

[0003] Therefore, needle guidance is very important in prostate lesion puncture and directly affects the puncture effect.

[0004] At present, it is relatively difficult to directly guide the puncture needle using multimodal imaging of prostate lesions without a field of view. The information is mixed, making it difficult to directly grasp the effective information related to the puncture needle guidance. Summary of the Invention

[0005] The purpose of the present invention is to provide a puncture planning method, system and storage medium based on prostate multimodal imaging to solve the technical problem in the prior art that it is relatively difficult to directly guide the puncture needle using multimodal imaging of prostate lesions without a field of view, and the information is mixed, making it difficult to directly grasp the effective information related to puncture needle guidance.

[0006] In order to solve the above technical problems, the present invention specifically provides the following technical solutions: A puncture planning method based on prostate multimodal imaging comprises the following steps: Acquire microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate site; The medical image processing software 3D Slicer was used to register and fuse microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging to obtain multimodal prostate imaging. Marking the prostate cancer area in prostate multimodal imaging and planning a first path with the prostate cancer area as the target; acquiring in real time an ultrasonic image of the puncture needle as it moves along the first path; Detecting and segmenting the puncture needle in the ultrasound image, and determining the positioning information of the puncture needle tip; The first path is registered with the prostate multimodal imaging in the ultrasound image, and the positioning information of the puncture needle tip is corrected in real time in the registered ultrasound image according to the first path and displayed in the ultrasound image.

[0007] As a preferred embodiment of the present invention, a method for marking a prostate cancer area in prostate multimodal imaging includes: The U-Net network is used to segment the prostate cancer area and the surrounding tissue area in prostate multimodal imaging.

[0008] As a preferred solution of the present invention, the first path planning method includes: The multiple optimization objectives for planning the first path include: Maximize the distance between the needle tip and the surrounding tissue area , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of each point on the outline of the jth surrounding tissue region, n is all the path points in the first path, k is the total number of surrounding tissue regions, is the Euclidean distance operation formula, and max is the maximization identifier; The goal is to maximize the overlap between the puncture needle tip and the prostate cancer area , , where is the curve fitted by the puncture needle tip from the i-th to the n-th path points in the first path, is the straight line obtained by fitting the maximum diameter of the prostate cancer region, is the similarity operator; Minimize the distance traveled by the needle tip , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of the i+1th path point of the puncture needle tip in the first path, and min is the minimization identifier; The puncture path of the needle tip is smooth , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of the i+1th path point of the puncture needle tip in the first path, is the coordinate value of the i-1th path point of the puncture needle tip in the first path; The prostate region and the rectal region are used as the solution space, and a multi-objective optimization algorithm is used to solve the 、 、 and An optimization solution is performed to obtain a first path, and the first path is smoothed.

[0009] As a preferred embodiment of the present invention, the method for detecting and segmenting the puncture needle in the ultrasound image and determining the positioning information of the puncture needle tip includes: The YOLO3 algorithm is used to pre-position the puncture needle in the ultrasound image and obtain the puncture needle bounding box; The improved U-Net network is used to segment the puncture needle in the ultrasound image within the puncture needle boundary box to obtain a puncture needle segmentation binary image; Linear regression is used to locate the puncture needle tip in the puncture needle segmentation binary image to obtain the position coordinates of the puncture needle tip.

[0010] As a preferred solution of the present invention, the method for determining the puncture needle bounding box includes: Input the ultrasound image into the YOLO3 algorithm to obtain the puncture needle bounding box; When all sides of the puncture needle bounding box are smaller than the preset size, a square box of the preset size is formed by expanding the center point of the puncture needle bounding box in all directions, which serves as the corrected puncture needle bounding box. At the same time, the ultrasound image in the square box is the ultrasound image in the puncture needle bounding box. When at least one side of the puncture needle bounding box is larger than the preset size, a square box that completely contains the puncture needle bounding box is formed with the center point of the puncture needle bounding box as the center point and the longest side as the side length. The square box and the ultrasound image in the square box are simultaneously scaled to the preset size as the corrected puncture needle bounding box and the ultrasound image in the puncture needle bounding box.

[0011] As a preferred solution of the present invention, the improved U-Net network consists of 50 convolutional layers, including 9 stages, namely 4 contraction paths, 1 center path, and 4 expansion paths; At the beginning of each stage, a 1×1 convolution layer is added to perform pooling operations on the input feature map channels; The contraction path is a 1×1 convolutional layer followed by two residual blocks with skip connections. The residual blocks are subjected to a 2×2 maximum pooling operation with a stride of 2. Each residual block consists of two 3×3 convolution kernels, and a rectified linear unit (ReLU) is connected after each convolution kernel. The expansion path first upsamples the input feature map, then performs a 2×2 convolution to halve the number of feature channels, and connects the output feature map to the corresponding feature map of the contraction path.

[0012] As a preferred embodiment of the present invention, a method for locating the puncture needle tip in a puncture needle segmentation binary image using linear regression includes: Perform linear fitting on the puncture needle to obtain the puncture needle fitting curve ; Get the pixel coordinates of the puncture needle in the puncture needle segmentation binary image , is the pixel coordinate of the lth point on the puncture needle, m is the total number of pixels on the puncture needle, and the least squares method is used to fit the curve of the puncture needle to form 、 、 and The solution function , ; Will Convert to matrix form to get ; in, , , ; Solving via matrix calculus The minimum point is obtained ,according to Obtain , and the obtained Substitute into the puncture needle fitting curve; The criteria functions for searching the needle tip position based on the puncture needle fitting curve include: Strength reduction criteria for puncture needle tip ; Gradient descent criterion for the needle tip ; Needle Size Guidelines ; Where, is the position coordinate of the puncture needle tip, is the position coordinate of the puncture needle starting point on the puncture needle fitting curve, is the position coordinate of the puncture needle end point on the puncture needle fitting curve, is the position coordinate of the point behind the needle tip on the puncture needle fitting curve, Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The gradient amplitude at , Segmentation of the puncture needle in the binary image The gradient amplitude at , Segmentation of the puncture needle in the binary image The gradient amplitude at , L is the length of the puncture needle, max is the maximization identifier, min is the minimization identifier, is the Euclidean distance operation formula; The criterion function is solved using the puncture needle fitting curve to obtain the position coordinates of the puncture needle tip.

[0013] As a preferred embodiment of the present invention, the puncture needle fitting curve is located at arrive The local lines between the two images are superimposed on the ultrasound image within the puncture needle boundary box to obtain the puncture needle enhanced ultrasound image, which is then registered and fused with the prostate multimodal imaging.

[0014] As a preferred embodiment of the present invention, the present invention provides a puncture planning system based on prostate multimodal imaging, which is applied to a puncture planning method based on prostate multimodal imaging. The system includes: The first data processing unit is used to obtain microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate area, and to register and fuse the microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging using the medical image processing software 3D Slicer to obtain multimodal prostate imaging; a path planning unit, configured to mark a prostate cancer region in prostate multimodal imaging and plan a first path with the prostate cancer region as a target; an ultrasound guidance unit, configured to obtain in real time an ultrasound image of the puncture needle as it moves along the first path; a second data processing unit, configured to detect and segment the puncture needle in the ultrasound image and determine positioning information of the puncture needle tip; The registration correction unit is used to register the first path with the prostate multimodal imaging to the ultrasound image, and to perform real-time correction on the positioning information of the puncture needle tip in the registered ultrasound image according to the first path, and to display it in the ultrasound image.

[0015] As a preferred embodiment of the present invention, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, a puncture planning method based on prostate multimodal imaging is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention first performs prostate multimodal imaging to accurately display the details of the prostate cancer area, and plans the puncture path of prostate cancer on the prostate multimodal imaging, thereby providing guidance information for the puncture, thereby improving the accuracy of puncture path planning, and accurately positioning the puncture needle shape and positioning information through ultrasound guidance during the puncture process, so as to grasp the relative position between the puncture needle tip and the guidance path, so as to use the planned path to provide accurate guidance or correction information for the puncture process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0018] Figure 1 A flow chart of the guided puncture method provided in an embodiment of the present invention; Figure 2 A block diagram of a guided puncture system provided in an embodiment of the present invention; Figure 3 A flowchart of puncture needle detection, segmentation, and needle tip positioning provided by an embodiment of the present invention; Figure 4 This is a diagram of the puncture needle detection, segmentation, and needle tip positioning results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, the present invention provides a puncture planning method based on prostate multimodal imaging, comprising the following steps: Acquire microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate site; The medical image processing software 3D Slicer was used to register and fuse microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging to obtain multimodal prostate imaging. Mark the prostate cancer area in prostate multimodal imaging and plan the first path with the prostate cancer area as the target, which is essentially the planned path to guide the puncture needle; Acquiring in real time an ultrasound image of the puncture needle as it moves along the first path, the ultrasound image usually being acquired through a transrectal ultrasound probe; Detecting and segmenting the puncture needle in the ultrasound image, and determining the positioning information of the puncture needle tip; The first path is registered with the prostate multimodal imaging in the ultrasound image, and correction information of the puncture needle's travel process is determined in the registered ultrasound image using positioning information of the first path and the puncture needle tip.

[0021] In this embodiment, the following data are obtained: The first data, the multimodal imaging data of the prostate tissue, is subjected to object analysis to obtain the condition of the target tissue and to plan the puncture path of the puncture needle according to the condition of the tissue.

[0022] The second data is the needle's trajectory data. Essentially, this involves analyzing and processing image data during the trajectory, understanding the deviation between the needle tip and the planned path, and designing a corrective guidance method for the needle. This method can be used with a constructed human model or a model of the entire prostate area, or other tissue models (again, using surrounding tissue as obstacles to plan the puncture path from the needle insertion point to the target).

[0023] In addition, this method can also be applied to other mechanical operation path planning in situations where there is no field of view.

[0024] In the present invention, a pre-puncture analysis is first performed to obtain the individual structure of the puncture object, and then the optimal guidance path for transrectal prostate cancer puncture is formulated based on the individual structure of the puncture object.

[0025] After developing the optimal guidance path adapted to the individual structure, the present invention uses it to guide the puncture needle during the puncture operation, and combines it with intraoperative ultrasound guidance to obtain an intraoperative ultrasound image. The puncture needle is segmented in the ultrasound image and the needle tip position is located, thereby determining whether the puncture needle has deviated from the planned first path, and providing movement correction information for the deviated puncture needle based on the deviation displayed in the image.

[0026] The present invention obtains multimodal imaging of prostate cancer during pre-operative analysis, namely, the fusion of magnetic resonance imaging, photoacoustic imaging, and ultrasound imaging, which can accurately display the details of the prostate cancer area. Path planning based on multimodal imaging can produce high-quality path planning results. The present invention segmented the prostate cancer area based on multimodal imaging to understand the individual situation, as follows: Methods for marking prostate cancer areas in multimodal prostate imaging include: The U-Net network is used to segment the prostate cancer area and surrounding tissue areas (urethra, ejaculatory duct, bladder, etc.) in prostate multimodal imaging.

[0027] After completing the organizational structure segmentation, the present invention performs path planning based on the organizational structure segmentation results, as follows: The planning methods for the first path include: The multiple optimization objectives for planning the first path include: Maximize the distance between the needle tip and the surrounding tissue area , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of each point on the outline of the jth surrounding tissue region, n is all the path points in the first path, k is the total number of surrounding tissue regions, is the Euclidean distance operation formula, and max is the maximization identifier; Target The safety optimization goal is characterized so that the puncture needle avoids the surrounding tissue area as much as possible during the puncture process, thereby ensuring safety.

[0028] The goal is to maximize the overlap between the puncture needle tip and the prostate cancer area , , where is the curve fitted by the puncture needle tip from the i-th to the n-th path points in the first path, is the straight line obtained by fitting the maximum diameter of the prostate cancer region, is the similarity operator; Target The optimization goal of the sampling effect is characterized so that the puncture needle can overlap with the prostate cancer area as much as possible during the puncture process, thereby ensuring the maximum acquisition of prostate cancer tissue.

[0029] Minimize the distance traveled by the needle tip , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of the i+1th path point of the puncture needle tip in the first path, and min is the minimization identifier; Target The distance optimization goal is characterized, so that the puncture path is as short as possible and the puncture process is completed as quickly as possible to ensure puncture efficiency.

[0030] The puncture path of the needle tip is smooth , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of the i+1th path point of the puncture needle tip in the first path, is the coordinate value of the i-1th path point of the puncture needle tip in the first path; Target The optimization goal of path smoothness is characterized to make the puncture process as smooth as possible to reach the prostate cancer area, avoid unnecessary damage, and ensure path continuity and safety.

[0031] The prostate region and the rectal region are used as the solution space, and a multi-objective optimization algorithm is used to solve the 、 、 and An optimization solution is performed to obtain a first path, and the first path is smoothed.

[0032] The present invention 、 、 and Planning the puncture path can obtain a safe, smooth and efficient path.

[0033] The present invention applies ultrasound guidance during the puncture process, which can segment the puncture needle in the ultrasound image, obtain the puncture needle shape, and locate the needle tip position of the puncture needle, as follows: like Figure 3 and Figure 4 As shown, the method for detecting and segmenting the puncture needle in the ultrasound image and determining the positioning information of the puncture needle tip includes: The YOLO3 algorithm is used to pre-position the puncture needle in the ultrasound image and obtain the puncture needle bounding box; The improved U-Net network is used to segment the puncture needle in the ultrasound image within the puncture needle boundary box to obtain a puncture needle segmentation binary image; Linear regression is used to locate the puncture needle tip in the puncture needle segmentation binary image to obtain the position coordinates of the puncture needle tip.

[0034] The method for determining the puncture needle bounding box includes: Input the ultrasound image into the YOLO3 algorithm to obtain the puncture needle bounding box; The size and proportions of the ROI bounding box determined by YOLO depend on the needle size and insertion depth. Because the segmentation framework used requires the image to be of a predetermined size, such as 256×256 pixels, the probability of obtaining a ROI bounding box of this size is low in practice. To address this issue, the present invention proposes a correction technique, which is implemented as follows: When all sides of the puncture needle bounding box are smaller than the preset size, a square box of the preset size is formed by expanding the center point of the puncture needle bounding box in all directions, which serves as the corrected puncture needle bounding box. At the same time, the ultrasound image in the square box is the ultrasound image in the puncture needle bounding box. When at least one side of the puncture needle bounding box is larger than the preset size, a square box that completely contains the puncture needle bounding box is formed with the center point of the puncture needle bounding box as the center point and the longest side as the side length. The square box and the ultrasound image in the square box are simultaneously scaled to the preset size as the corrected puncture needle bounding box and the ultrasound image in the puncture needle bounding box.

[0035] In other words, the small frame processing of the present invention uses the center point of the original bounding box as an anchor point and directly expands outward to generate a 256x256 pixel frame, with the original small bounding box located in the center of the new frame.

[0036] The large frame processing uses the center point of the original bounding box as the anchor point, first constructs a minimum square that can completely contain the original ROI (the side length is equal to the long side of the original box), and then scales this square area to 256x256 pixels.

[0037] This correction ensures that a bounding box image of the preset size is obtained regardless of the original bounding box size, without losing the puncture needle pixels.

[0038] The improved U-Net network consists of 50 convolutional layers, including 9 stages, 4 contraction paths, 1 center path, and 4 expansion paths; The entire network consists of 50 convolutional layers, resulting in a significant decrease in speed compared to the general U-Net model. This decrease is due to the increased computational effort. To address this issue, a 1×1 convolutional layer is added at the beginning of each stage to perform pooling on the input feature map channels. The contraction path is a 1×1 convolutional layer followed by two residual blocks with skip connections. The residual blocks are subjected to a 2×2 maximum pooling operation with a stride of 2. Each residual block consists of two 3×3 convolution kernels, and a rectified linear unit (ReLU) is connected after each convolution kernel. The expansion path first upsamples the input feature map, then performs a 2×2 convolution to halve the number of feature channels, and connects the output feature map to the corresponding feature map of the contraction path.

[0039] This paper uses the least squares method to perform linear fitting on the puncture needle segmentation results. After obtaining the puncture needle fitting curve, the needle tip position can be located and the information of the puncture needle region can be enhanced based on the puncture needle segmentation results to highlight the puncture needle shape. The details are as follows: Methods for locating the puncture needle tip in a puncture needle segmentation binary image using linear regression include: Perform linear fitting on the puncture needle to obtain the puncture needle fitting curve ; Get the pixel coordinates of the puncture needle in the puncture needle segmentation binary image , is the pixel coordinate of the lth point on the puncture needle, m is the total number of pixels on the puncture needle, and the least squares method is used to fit the curve of the puncture needle to form 、 、 and The solution function , ; Will Convert to matrix form to get ; in, , , ; Solving via matrix calculus The minimum point is obtained ,according to Obtain , and the obtained Substitute into the puncture needle fitting curve; The present invention determines the fitting curve of the puncture needle through the above calculation process, and the needle tip position of the puncture needle can be searched along the fitting curve, as follows: The criteria functions for searching the needle tip position based on the puncture needle fitting curve include: Strength reduction criteria for puncture needle tip ; The grayscale intensity of the puncture needle tip will be greatly reduced compared to the needle body, and the grayscale intensity of the puncture needle tip will also be greatly reduced as the needle tip disappears when extending forward along the puncture needle fitting curve. Therefore, the present invention establishes an intensity reduction criterion to search and solve the needle tip area. is the position coordinate of the puncture needle end point on the puncture needle fitting curve, which is equivalent to the position point closest to the needle tip on the puncture needle in the image. Therefore, the puncture needle tip will be Appear nearby, Maximize, expect to search for There is a point where the strength is greatly reduced, which is the needle tip position. Maximize, expect the intensity between the needle tip point found and the point extended forward along the puncture needle fitting curve to be greatly reduced, further verifying the accuracy of the needle tip search, that is, there should be a difference in grayscale intensity between the needle tip found and its front and back positions. Further, Maximization indicates that there should be a large grayscale difference between the front and back positions of the needle tip, constraining the first two and , to avoid misjudgment of the needle tip position and inaccurate positioning results.

[0040] Gradient descent criterion for the needle tip ; Similarly, the image gradient at the puncture needle tip is significantly reduced compared to the needle body. When the puncture needle tip extends forward along the puncture needle fitting curve, the image gradient also significantly decreases due to the disappearance of the needle tip. Therefore, the present invention establishes a gradient descent criterion to search for a solution in the needle tip area. is the position coordinate of the puncture needle end point on the puncture needle fitting curve, which is equivalent to the position point closest to the needle tip on the puncture needle in the image. Therefore, the puncture needle tip will be Appear nearby, Maximize, expect to search for The point where the gradient decreases significantly is the needle tip position. Maximize, expect the gradient between the needle tip point found and the point extended forward along the puncture needle fitting curve to be greatly reduced, further verifying the accuracy of the needle tip search, that is, there should be a difference in image gradient between the needle tip found and its front and back positions. Further, Maximization shows that there is a large gradient difference between the position in front of the needle tip and the position behind the needle tip, constraining the first two and , to avoid misjudgment of the needle tip position, or random errors.

[0041] in, , where the horizontal gradient , vertical gradient It is calculated using the Sobel operator.

[0042] Needle Size Guidelines ; The present invention also uses the inherent length L of the puncture needle to set the size criteria. is the position coordinate of the starting point of the puncture needle on the puncture needle fitting curve, which is equivalent to the needle tail of the puncture needle. Therefore, the distance between the needle tip and the needle tail should be equal to the inherent length of the puncture needle. For size guidelines.

[0043] The present invention utilizes the above three criteria to perform optimization and solution on the fitting curve to obtain the position coordinates of the needle tip, thereby realizing needle tip positioning.

[0044] Where, is the position coordinate of the puncture needle tip, is the position coordinate of the puncture needle starting point on the puncture needle fitting curve, is the position coordinate of the puncture needle end point on the puncture needle fitting curve, is the position coordinate of the point behind the needle tip on the puncture needle fitting curve, Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The gradient amplitude at , Segmentation of the puncture needle in the binary image The gradient amplitude at , Segmentation of the puncture needle in the binary image The gradient amplitude at , L is the length of the puncture needle, max is the maximization identifier, min is the minimization identifier, is the Euclidean distance operation formula.

[0045] The criterion function is solved using the puncture needle fitting curve to obtain the position coordinates of the puncture needle tip.

[0046] Place the puncture needle on the fitting curve arrive The local lines between the two images are superimposed on the ultrasound image within the puncture needle boundary box to obtain the puncture needle enhanced ultrasound image, which is then registered and fused with the prostate multimodal imaging.

[0047] The method of registering the first path from the medical image to the ultrasound image includes: The multimodal imaging is used as a reference image and the ultrasound image is used as a floating image. The multimodal imaging and the ultrasound image are registered and fused to obtain an ultrasound image containing the first path after registration.

[0048] When the position coordinates of the puncture needle tip are on the first path, the next path point is used as guidance information to guide the puncture needle tip to continue puncturing; When the position coordinates of the puncture needle tip deviate from the first path, the path point on the first path at the closest distance to the puncture needle tip is used as correction information to correct the offset part of the puncture needle during the puncture process so that it returns to the first path and continues to move forward.

[0049] like Figure 2 As shown, the present invention provides a puncture planning system based on prostate multimodal imaging, which is applied to a puncture planning method based on prostate multimodal imaging. The system includes: The first data processing unit is used to obtain microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate area, and to register and fuse the microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging using the medical image processing software 3D Slicer to obtain multimodal prostate imaging; a path planning unit, configured to mark a prostate cancer region in prostate multimodal imaging and plan a first path with the prostate cancer region as a target; an ultrasound guidance unit, configured to obtain in real time an ultrasound image of the puncture needle as it moves along the first path; a second data processing unit, configured to detect and segment the puncture needle in the ultrasound image and determine positioning information of the puncture needle tip; The registration correction unit is used to register the first path from the prostate multimodal imaging to the ultrasound image, and determine the correction information of the puncture needle's travel process in the registered ultrasound image using the positioning information of the first path and the puncture needle tip.

[0050] The present invention provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, a puncture planning method based on prostate multimodal imaging is implemented.

[0051] The present invention first performs prostate multimodal imaging to accurately display the details of the prostate cancer area, and plans the puncture path of prostate cancer on the prostate multimodal imaging, thereby providing guidance information for puncture, thereby improving the accuracy of puncture path planning, and accurately positioning the puncture needle shape and positioning information through ultrasound guidance during the puncture guidance process, so as to grasp the relative position between the puncture needle tip and the guidance path, so as to use the planned path to provide accurate guidance or correction information for the puncture process.

[0052] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A puncture planning method based on prostate multimodal imaging, characterized in that: The following steps are involved: Acquire microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate site; The medical image processing software 3D Slicer was used to register and fuse microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging to obtain multimodal prostate imaging. Marking the prostate cancer area in prostate multimodal imaging and planning a first path with the prostate cancer area as the target; acquiring in real time an ultrasonic image of the puncture needle as it moves along the first path; Detecting and segmenting the puncture needle in the ultrasound image, and determining the positioning information of the puncture needle tip; The first path is registered with the prostate multimodal imaging in the ultrasound image, and correction information of the puncture needle's travel process is determined in the registered ultrasound image using positioning information of the first path and the puncture needle tip.

2. The puncture planning method based on prostate multimodal imaging according to claim 1, characterized in that: Methods for marking prostate cancer areas in multimodal prostate imaging include: The U-Net network is used to segment the prostate cancer area and the surrounding tissue area in prostate multimodal imaging.

3. The puncture planning method based on prostate multimodal imaging according to claim 2, characterized in that: The first path planning method includes: The multiple optimization objectives for planning the first path include: Maximize the distance between the needle tip and the surrounding tissue area , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of each point on the outline of the jth surrounding tissue region, n is all the path points in the first path, k is the total number of surrounding tissue regions, is the Euclidean distance operation formula, and max is the maximization identifier; The goal is to maximize the overlap between the puncture needle tip and the prostate cancer area , , where is the curve fitted by the puncture needle tip from the i-th to the n-th path points in the first path, is the straight line obtained by fitting the maximum diameter of the prostate cancer region, is the similarity operator; Minimize the distance traveled by the needle tip , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of the i+1th path point of the puncture needle tip in the first path, and min is the minimization identifier; The puncture path of the needle tip is smooth , , where is the coordinate value of the i-th path point of the puncture needle tip in the first path, is the coordinate value of the i+1th path point of the puncture needle tip in the first path, is the coordinate value of the i-1th path point of the puncture needle tip in the first path; The prostate region and the rectal region are used as the solution space, and a multi-objective optimization algorithm is used to solve the 、 、 and An optimization solution is performed to obtain a first path, and the first path is smoothed.

4. The puncture planning method based on prostate multimodal imaging according to claim 1, characterized in that: The method for detecting and segmenting the puncture needle in the ultrasound image and determining the positioning information of the puncture needle tip includes: The YOLO3 algorithm is used to pre-position the puncture needle in the ultrasound image and obtain the puncture needle bounding box; The improved U-Net network is used to segment the puncture needle in the ultrasound image within the puncture needle boundary box to obtain a puncture needle segmentation binary image; Linear regression is used to locate the puncture needle tip in the puncture needle segmentation binary image to obtain the position coordinates of the puncture needle tip.

5. The puncture planning method based on prostate multimodal imaging according to claim 4, characterized in that: The method for determining the puncture needle bounding box includes: Input the ultrasound image into the YOLO3 algorithm to obtain the puncture needle bounding box; When all sides of the puncture needle bounding box are smaller than the preset size, a square box of the preset size is formed by expanding the center point of the puncture needle bounding box in all directions, which serves as the corrected puncture needle bounding box. At the same time, the ultrasound image in the square box is the ultrasound image in the puncture needle bounding box. When at least one side of the puncture needle bounding box is larger than the preset size, a square box that completely contains the puncture needle bounding box is formed with the center point of the puncture needle bounding box as the center point and the longest side as the side length. The square box and the ultrasound image in the square box are simultaneously scaled to the preset size as the corrected puncture needle bounding box and the ultrasound image in the puncture needle bounding box.

6. The puncture planning method based on prostate multimodal imaging according to claim 5, characterized in that: The improved U-Net network consists of 50 convolutional layers, including 9 stages, including 4 contraction paths, 1 center path, and 4 expansion paths; At the beginning of each stage, a 1×1 convolution layer is added to perform pooling operations on the input feature map channels; The contraction path is a 1×1 convolutional layer followed by two residual blocks with skip connections. The residual blocks are subjected to a 2×2 maximum pooling operation with a stride of 2. Each residual block consists of two 3×3 convolution kernels, and a rectified linear unit (ReLU) is connected after each convolution kernel. The expansion path first upsamples the input feature map, then performs a 2×2 convolution to halve the number of feature channels, and connects the output feature map to the corresponding feature map of the contraction path.

7. The puncture planning method based on prostate multimodal imaging according to claim 5, characterized in that: Methods for locating the puncture needle tip in a puncture needle segmentation binary image using linear regression include: Perform linear fitting on the puncture needle to obtain the puncture needle fitting curve ; Get the pixel coordinates of the puncture needle in the puncture needle segmentation binary image , is the pixel coordinate of the lth point on the puncture needle, m is the total number of pixels on the puncture needle, and the least squares method is used to fit the curve of the puncture needle to form 、 、 and The solution function , ; Will Convert to matrix form to get ; in, , , ; Solving via matrix calculus The minimum point is obtained ,according to Obtain , and the obtained Substitute into the puncture needle fitting curve; The criteria functions for searching the needle tip position based on the puncture needle fitting curve include: Strength reduction criteria for puncture needle tip ; Gradient descent criterion for the needle tip ; Needle Size Guidelines ; Where, is the position coordinate of the puncture needle tip, is the position coordinate of the puncture needle starting point on the puncture needle fitting curve, is the position coordinate of the puncture needle end point on the puncture needle fitting curve, is the position coordinate of the point behind the needle tip on the puncture needle fitting curve, Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The grayscale intensity at Segmentation of the puncture needle in the binary image The gradient amplitude at , Segmentation of the puncture needle in the binary image The gradient amplitude at , Segmentation of the puncture needle in the binary image The gradient amplitude at , L is the length of the puncture needle, max is the maximization identifier, min is the minimization identifier, is the Euclidean distance operation formula; The criterion function is solved using the puncture needle fitting curve to obtain the position coordinates of the puncture needle tip.

8. The puncture planning method based on prostate multimodal imaging according to claim 7, characterized in that: Place the puncture needle on the fitting curve arrive The local lines between the two images are superimposed on the ultrasound image within the puncture needle boundary box to obtain the puncture needle enhanced ultrasound image, which is then registered and fused with the prostate multimodal imaging.

9. A puncture planning system based on prostate multimodal imaging, characterized in that: A puncture planning method based on prostate multimodal imaging as described in any one of claims 1 to 8, the system comprising: The first data processing unit is used to obtain microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging of the same prostate area, and to register and fuse the microvascular photoacoustic imaging, ultrasound elastography, and magnetic resonance imaging using the medical image processing software 3D Slicer to obtain multimodal prostate imaging; a path planning unit, configured to mark a prostate cancer region in prostate multimodal imaging and plan a first path with the prostate cancer region as a target; an ultrasound guidance unit, configured to obtain in real time an ultrasound image of the puncture needle as it moves along the first path; a second data processing unit, configured to detect and segment the puncture needle in the ultrasound image and determine positioning information of the puncture needle tip; The registration correction unit is used to register the first path from the prostate multimodal imaging to the ultrasound image, and determine the correction information of the puncture needle's travel process in the registered ultrasound image using the positioning information of the first path and the puncture needle tip.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 8 is implemented.

Images