Biopsy needle positioning system and method and medical equipment system
By setting markers on the biopsy needle cannula and estimating the position and orientation of the markers using RGB and depth images, the problems of high cost and strong operator dependence of robotic arms are solved, and efficient and accurate positioning of biopsy needles is achieved.
Patent Information
- Application Number
- CN202411298803.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2026-03-20
AI Technical Summary
Robotic arms are expensive and require a large deployment space in biopsy procedures. They are also highly dependent on the operator's experience, which can lead to errors in insertion point and angle, reducing work efficiency and increasing patient suffering.
A fixed marker is placed on the cannula of the biopsy needle. The position and orientation of the marker are estimated using RGB and depth images. The insertion point and angle are calculated using a network model.
It reduces biopsy costs, improves work efficiency, and enables real-time accurate positioning of biopsy needles without the need for robotic arms or recalibration.
Smart Images
Figure CN121694871A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical imaging, and in particular to a biopsy needle positioning system and positioning method, as well as a medical device system. Background Technology
[0002] Biopsy involves taking cells, tissues, or fluids from a patient for examination to determine if the patient has a disease or if the disease is malignant. Typically, biopsies require additional equipment, such as robotic arms or simple auxiliary grippers. Robotic arms offer greater deployment space and can automatically provide precise insertion points and angles without the need for operators. Compared to robotic arms, simple auxiliary grippers help secure the biopsy needle, but each procedure must be performed manually.
[0003] Biopsy plays an important role in pathological diagnosis. Through biopsy, clinicians can make diagnoses of lesions or provide clues for disease diagnosis, understand the nature and development trend of lesions, and provide reference for clinical medication.
[0004] It should be noted that the above introduction to the technical background is only for the purpose of providing a clear and complete explanation of the technical solutions of this application and facilitating understanding by those skilled in the art. It should not be assumed that these technical solutions are known to those skilled in the art simply because they have been described in the background section of this application. Summary of the Invention
[0005] The inventors discovered that, in practice, although robotic arms can provide precise insertion points and angles, they are expensive and require a large deployment space. When the biopsy needle insertion point and angle change, the robotic arm must be recalibrated and re-calibrated, wasting a lot of time. In addition, the use of simple auxiliary clamps largely depends on the operator's experience. Inexperienced operators may cause incorrect insertion points and angles, thereby reducing work efficiency and increasing patient suffering.
[0006] To address at least one or more of the aforementioned problems, this application provides a biopsy needle positioning system, a positioning method, and a medical device system. By setting a marker fixed to the cannula carrying the biopsy needle, the true position and orientation of the marker are estimated based on the acquired RGB image and depth image corresponding to the marker. A network model is then used to obtain the estimated position and orientation of the marker, thereby determining the insertion point and angle of the biopsy needle. This reduces biopsy costs and improves work efficiency.
[0007] According to one aspect of an embodiment of this application, a biopsy needle positioning system is provided, the system comprising:
[0008] A marker assembly, the marker assembly including a marker and a cannula with a biopsy needle mounted on it fixed relative to the marker;
[0009] An image acquisition device, installed at a location capable of capturing images of the marker, is used to acquire an RGB image of the marker and a depth image corresponding to the RGB image; and
[0010] The positioning device uses a trained network model to extract the RGB features and point cloud features of the marker based on the RGB image and depth image of the marker, and estimates the predicted position and orientation of the marker based on the RGB features and point cloud features, thereby obtaining the insertion point and insertion angle of the biopsy needle.
[0011] In some embodiments, the marker assembly further includes a bracket connected to one end of the sleeve, the marker being mounted on the bracket.
[0012] In some embodiments, the bracket is L-shaped.
[0013] In some embodiments, the marker is a cube, sphere, or cone.
[0014] In some embodiments, the positioning device uses a segmentation network to segment the RGB image and separate the marker portion from the RGB image; uses the marker portion as input to the RGB network to extract the RGB features of the marker portion; uses the point cloud portion corresponding to the marker portion on the depth image as input to the point cloud network to extract the point cloud features of the point cloud portion; and obtains the true position and orientation of the marker based on the RGB features and the point cloud features.
[0015] In some embodiments, the positioning device also projects the depth image into two dimensions and samples the resulting two-dimensional points, using a boundary prediction network to predict the boundary of the marker;
[0016] The positioning device obtains the true position and orientation of the marker based on the RGB features, the point cloud features, and the boundary of the marker.
[0017] In some embodiments, the system further includes:
[0018] The training device uses multiple RGB images of the marker at different angles and depth images corresponding to each RGB image acquired by the image acquisition device, as well as the actual position and pose of the marker at the aforementioned different angles, to train the network model used by the positioning device.
[0019] In some embodiments, for each RGB image and its corresponding depth image, the training device sets the coordinates P of multiple marker points on the marker in the world coordinate system. world Convert to coordinates P in the camera coordinate system camera And the coordinates P of the multiple marker points in the camera coordinate system camera Convert to coordinates P in the CT coordinate system ct The estimated position and orientation of the marker at the current angle are obtained.
[0020] In some embodiments, the training device calculates the coordinates P of the plurality of marker points in the world coordinate system according to the following formula. world Convert to coordinates P in the camera coordinate system camera :
[0021] P camera =R wc ·P world +t wc
[0022] Among them, R wc and t wc R is an external parameter of the image acquisition device. wc Let t be a rotation matrix. wc is the translation vector, representing the position and orientation of the camera coordinate system relative to the world coordinate system, respectively.
[0023] In some embodiments, the training device assigns the coordinates P of the plurality of marker points in the camera coordinate system according to the following formula. camer a is converted to coordinate P in the CT coordinate system. ct :
[0024] P ct =R ct ·P camera +t ct
[0025] Among them, R ct Let t be a rotation matrix. ct Let R be the translation vector, and let R be the position and orientation of the CT coordinate system relative to the camera coordinate system, respectively. ct and t ct It was obtained by singular value decomposition.
[0026] In some embodiments, the plurality of marker points are located at different positions on the marker, and the plurality of marker points characterize the outline of the marker.
[0027] In some embodiments, the system further includes:
[0028] The display device displays the insertion point and insertion angle of the biopsy needle.
[0029] According to another aspect of the embodiments of this application, a biopsy needle localization method is provided, the method comprising:
[0030] The RGB image of the marker and the corresponding depth image are acquired, and the marker is connected and fixed to a cannula with a biopsy needle fixed relative to the marker.
[0031] Using a trained network model, RGB features and point cloud features of the markers are extracted from their RGB and depth images.
[0032] Based on the RGB features and point cloud features, the estimated position and orientation of the marker are estimated, thereby obtaining the insertion point and angle of the biopsy needle.
[0033] According to another aspect of the embodiments of this application, a medical device system is provided, the medical device system including the biopsy needle positioning system described in the embodiments of the first aspect above.
[0034] One of the beneficial effects of this application's embodiments is that by setting a marker fixed on the cannula carrying the biopsy needle, the true position and orientation of the marker are estimated based on the acquired RGB image corresponding to the marker and the depth image corresponding to the RGB image, and the estimated position and orientation of the marker are obtained using a network model. Therefore, by using the position and orientation of the marker, which is fixed relative to the biopsy needle, to calculate the insertion point and angle of the biopsy needle, real-time and accurate positioning of the biopsy needle can be achieved without a robotic arm and recalibration process, thereby saving biopsy costs and improving work efficiency.
[0035] Embodiments of this application are disclosed in detail with reference to the following description and accompanying drawings. It should be understood that the scope of embodiments of this application is not limited thereto. Within the spirit and scope of the appended claims, embodiments of this application include many changes, modifications, and equivalents.
[0036] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.
[0037] It should be emphasized that the term "including / contains / has" as used herein refers to the presence of a feature, whole or component, but does not exclude the presence or addition of one or more other features, wholes or components. Attached Figure Description
[0038] The above and other objects, features and advantages of the embodiments of this application will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:
[0039] Figure 1 This is a schematic diagram of a biopsy needle positioning system according to an embodiment of this application;
[0040] Figure 2 This is a schematic diagram of a sleeve according to an embodiment of this application;
[0041] Figure 3 This is a schematic diagram of a marker according to an embodiment of this application;
[0042] Figure 4 This is a schematic diagram of a marker component according to an embodiment of this application;
[0043] Figure 5 This is a schematic diagram illustrating the working process of the positioning device according to an embodiment of this application;
[0044] Figure 6 This is a schematic diagram illustrating the working process of the trained network model according to an embodiment of this application;
[0045] Figure 7 This is a schematic diagram of the operation process of the training device according to an embodiment of this application;
[0046] Figure 8 This is a schematic diagram of the transformation from the world coordinate system to the camera coordinate system;
[0047] Figure 9 This is a schematic diagram of a biopsy needle positioning method according to an embodiment of this application;
[0048] Figure 10 This is a schematic diagram of a CT imaging device according to an embodiment of this application;
[0049] Figure 11 This is a schematic diagram of a CT imaging system according to an embodiment of this application. Detailed Implementation
[0050] Referring to the accompanying drawings, the foregoing and other features of this application will become apparent from the following description. Specific embodiments of this application are specifically disclosed in the description and drawings, illustrating partial embodiments in which the principles of this application can be employed. It should be understood that this application is not limited to the described embodiments, but includes all modifications and equivalents falling within the scope of the appended claims.
[0051] In the embodiments of this application, the terms "first," "second," "upper," "lower," etc., are used to distinguish different elements by their names, but do not indicate the spatial arrangement or temporal order of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one or more of the terms listed in connection with the application and all combinations thereof. The terms "comprising," "including," "having," etc., refer to the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.
[0052] In the embodiments of this application, the singular forms "a," "the," etc., including the plural forms, should be broadly understood as "a kind" or "a class" rather than limited to the meaning of "an." Furthermore, the term "the" should be understood to include both the singular and plural forms, unless the context explicitly indicates otherwise. Additionally, the term "according to" should be understood as "at least partially based on…," and the term "based on" should be understood as "at least partially based on…," unless the context explicitly indicates otherwise.
[0053] Features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.
[0054] This application provides a biopsy needle positioning system.
[0055] Figure 1 This is a schematic diagram of a biopsy needle positioning system according to an embodiment of this application. Figure 1 As shown, the biopsy needle positioning system 100 includes:
[0056] The marker assembly 101 includes a marker 1011 and a cannula 1012, wherein the cannula 1012 is equipped with a biopsy needle 1013 and the cannula 1012 is fixed relative to the marker 1011.
[0057] Image acquisition device 102 is installed at a position where it can capture the marker 1011, and is used to acquire the RGB image of the marker 1011 and the depth image corresponding to the RGB image;
[0058] The positioning device 103 uses a trained network model to extract the RGB features and point cloud features of the marker 1011 based on the RGB image and depth image of the marker 1011. Based on the RGB features and point cloud features, it estimates the predicted position and orientation of the marker 1011, thereby obtaining the insertion point and insertion angle of the biopsy needle.
[0059] According to the above embodiments, by setting a marker fixed on the cannula carrying the biopsy needle, the true position and orientation of the marker are estimated based on the acquired RGB image and depth image corresponding to the marker, and the predicted position and orientation of the marker are obtained using a network model. Therefore, by using the position and orientation of the marker, which is fixed relative to the biopsy needle, to calculate the insertion point and angle of the biopsy needle, real-time and accurate positioning of the biopsy needle can be achieved without a robotic arm or recalibration process, thereby saving biopsy costs and improving work efficiency.
[0060] The marker components in the embodiments of this application will be described below.
[0061] Figure 4 This is a schematic diagram of a marker component according to an embodiment of this application. Figure 4 As shown, the marker assembly 101 includes a marker 1011 and a sleeve 1012. Figure 2 This is a schematic diagram of a sleeve according to an embodiment of this application.
[0062] In some embodiments, such as Figure 2 and Figure 4 As shown, the marker assembly 101 includes a cannula 1012 for accommodating a biopsy needle (not shown in the figure). The biopsy needle can be securely placed in the cannula 1012, that is, the relative position of the cannula 1012 and the biopsy needle is fixed, so as to facilitate subsequent movement operations and the acquisition of the position and orientation of the biopsy needle.
[0063] In the above embodiments, there are no restrictions on the material and shape of the cannula 1012 or the type of biopsy needle. The cannula 1012 can accommodate and fix different types of biopsy needles, and the biopsy needles can perform live tissue examinations.
[0064] In some embodiments, the sleeve 1012 has a support for placing the marker 1011, such as Figure 2 and Figure 4 As shown, a bracket 1012(a) is provided in the head region of the sleeve 1012, and the bracket 1012(a) is used to fix the marker 1011. Figure 2 and Figure 4 In the example, taking L-shaped bracket 1012(a) as an example, this application does not limit the structure of the bracket, as long as it can be used to fix the marker 1011 so that the marker 1011 is equivalent to the sleeve 1012 for fixing.
[0065] Figure 3 This is a schematic diagram of a marker in an embodiment of this application, such as... Figure 3 and Figure 4 As shown, the marker component 101 includes a marker 1011.
[0066] In at least one example, such as Figure 3 As shown, the marker 1011 is a cube with a side length of 40 mm. The cube has a support frame 1011(a) to ensure that the shape of the marker 1011 will not be deformed due to compression, thereby improving the accuracy of determining the position and orientation of the marker 1011. The side length of the marker 1011 can also be other lengths, as long as it facilitates the insertion of the biopsy needle and the calculation of the position and orientation of the marker 1011.
[0067] The marker 1011 can also be other shapes, such as a symmetrical sphere or cone. When the marker 1011 is a three-dimensional symmetrical figure, it is only necessary to calculate the position and orientation of a part of the marker 1011 to obtain the position and orientation of the entire marker 1011, and then obtain the insertion point and angle of the biopsy needle. The marker 1011 can also be a non-three-dimensional symmetrical figure. In this case, it is necessary to calculate its position and orientation for the entire marker 1011.
[0068] In the above embodiment, the surface of the marker 1011 can be covered with black, which helps to block the reflection of light so that it can be captured by the camera, thereby improving the accuracy of the image acquisition device 102.
[0069] In the above embodiments, the marker 1011 and the sleeve 1012 of the marker assembly 101 can be fixed by means of fitting or snapping, etc. Figure 2 and Figure 3 For example, the sleeve 1012 has a protrusion 1012(b), and the marker 1011 has an insertion portion 1011(b). The protrusion 1012(b) is inserted into the insertion portion 1011(b), so that the marker 1011 is fixed relative to the sleeve 1012, forming a shape as shown. Figure 4 The marker assembly 101 shown, in addition to the sleeve 1012 and the marker 1011, can also be integrally formed, or can be fixed in other ways.
[0070] like Figure 4 As shown, the marker 1011 is fixed at the tail of the cannula 1012. The marker 1011 can also be fixed at other parts of the cannula 1012, as long as the image acquisition device 102 can capture the marker and does not affect the insertion of the biopsy needle.
[0071] According to the above embodiments, by fixing a marker that can be captured by an image acquisition device on the cannula, the position and orientation of the biopsy needle can be calculated by estimating the position and orientation of the marker, thereby obtaining the insertion point and angle of the biopsy needle.
[0072] In some embodiments, an image acquisition device 102 is used to acquire RGB images of markers 1011 and depth images corresponding to the RGB images. The image acquisition device 102 is capable of acquiring views of multiple markers 1011 from different angles. The image acquisition device can be an RGB-D camera, such as a D345 camera, or other cameras capable of capturing RGB images and their corresponding depth images.
[0073] In the above embodiments, the image acquisition device 102 is installed at a position that can capture the marker 1011, for example, on the opposite side of the operator and 50 centimeters away from the operator's hand, or it can be installed at other positions as long as the marker 1011 can be captured.
[0074] In some embodiments, after the image acquisition device 102 acquires the RGB image and depth image of the marker 1011, the positioning device 103 can use a trained network to estimate the position and pose of the marker 1011 based on the RGB image and depth image of the marker 1011. The trained network includes a segmentation network, an RGB network, and a point cloud network.
[0075] The operation of the positioning device 103 in the embodiments of this application will be described below with reference to the accompanying drawings.
[0076] Figure 5 This is a schematic diagram illustrating the operation of the positioning device 103 according to an embodiment of this application. Figure 6 This is a schematic diagram illustrating the working process of the trained network model described above. Figure 6 In the example, RGB and depth images are taken as input, and the output is the estimated position and pose of marker 1011, etc. Figure 5 and Figure 6 As shown, the positioning device 103 can process the RGB image of the marker 1011 and the depth image corresponding to the RGB image obtained by the image acquisition device 102, respectively, so as to obtain the position and orientation of the marker 1011.
[0077] like Figure 5 As shown, the positioning device 103 uses a segmentation network to segment the RGB image and separate the marker portion from the RGB image. This removes information unrelated to the marker 1011 and avoids the image background from affecting the detection of the marker 1011.
[0078] In the above embodiments, the marker portion can be used as input to an RGB network, which processes the marker portion to extract its RGB features. Correspondingly, the point cloud portion corresponding to the marker portion on the depth image corresponding to the RGB image can be used as input to a point cloud network, which processes the point cloud portion to extract its point cloud features.
[0079] According to the above embodiments, the RBG features and point cloud features of the marker portion are obtained, and the positioning device 103 can estimate the position and orientation of the marker based on the RBG features and point cloud features.
[0080] In some embodiments, the positioning device 103 may use a pose prediction network to obtain the true position and pose of the marker 1011 based on the RGB features and point cloud features of the marker 1011.
[0081] This application does not limit the network structure of the trained network. In at least one example, the trained network may include a Full Flow Bidirectional Fusion Network (FFB6D), wherein the segmentation network, RGR network, point cloud network, and pose prediction network can be implemented in any feasible manner. The following uses... Figure 5 and Figure 6 The trained network will be illustrated using an example.
[0082] like Figure 5 As shown, the marker portion separated from the RGB image is convolved using an RGB network. In each convolutional layer, the convolved features are fused with the corresponding point cloud depth layer in the point cloud network, performing RGB-to-point cloud fusion and point cloud-to-RGB fusion to obtain fused features. RGB and point cloud information are then extracted from the fused features and input into the RGB and point cloud networks respectively. The fusion and extraction operations are repeated in the next convolutional layer and point cloud depth layer, and so on, to obtain RGB features and point cloud features. The RGB features are the output of the RGB network, and the point cloud features are the output of the point cloud network. The number of fusion and extraction operations can be set according to specific needs, and this application does not impose any restrictions on this.
[0083] Therefore, fusing the RGB image information and the corresponding depth image information in the captured image can achieve data complementarity, better present the appearance and geometric features required for pose estimation, and improve the accuracy of position and pose estimation.
[0084] In some embodiments, the positioning device 103 may further predict the boundary of the marker 1011 and calculate the position and orientation of the marker 1011 based on the RGB features, point cloud features and the boundary of the marker 1011.
[0085] by Figure 5 For example, the positioning device 103 projects the depth image corresponding to the RGB image of the marker 1011 into a two-dimensional image to obtain two-dimensional points of the depth image. It then samples the two-dimensional points and inputs the obtained sampled points into a boundary prediction network to predict the boundary of the marker 1011. Based on the boundary of the marker 1011, the aforementioned RGB features and point cloud features, the true position and orientation of the marker 1011 are obtained. This further supplements the boundary prediction of the marker and improves the accuracy of position and orientation estimation.
[0086] In the above embodiments, the network structure of the boundary prediction network is not limited. It can be implemented as part of the pre-trained network or through a separate deep learning network.
[0087] In at least one example, the pose prediction network can use the Iterative ClosestPoint (ICP) algorithm to estimate the true position and pose, or it can use other methods to estimate the true position and pose. In this case, the estimation of the true position and pose of marker 1011 is the estimation of the position and pose of marker 1011 in the world coordinate system.
[0088] In some embodiments, such as Figure 5 As shown, the positioning device 103 uses a trained network model to process the true position and orientation of the marker 1011 and outputs the estimated position and orientation of the marker 1011. In at least one example, such as Figure 1 As shown, the biopsy needle positioning system 100 includes a training device 104. The training device 104 uses multiple RGB images of the marker at different angles and the depth images corresponding to each RGB image acquired by the image acquisition device 102, as well as the real position and orientation of the marker at the aforementioned different angles, to train the network model used by the positioning device 103.
[0089] Figure 7 This is a schematic diagram of the operation process of the training device in the embodiments of this application. Figure 8 This is a schematic diagram of the transformation from the world coordinate system to the camera coordinate system.
[0090] In some embodiments, the training device 104, for each RGB image of the marker 1011 at different angles and the corresponding depth image of the RGB image acquired by the image acquisition device 102, sets the coordinates P of multiple marker points on the marker 1011 in the world coordinate system.world Convert to coordinates P in the camera coordinate system camera And assign the coordinates P of multiple marker points in the camera coordinate system camera Convert to coordinates P in the CT coordinate system ct The estimated position and orientation of the marker at the current angle are obtained.
[0091] The marker points on marker 1011 are located at different positions on marker 1011, representing the outline of marker 1011. The number and position of the marker points can be set according to actual needs, as long as they can completely represent the outline of marker 1011. Furthermore, the coordinates P of multiple marker points in the world coordinate system... world It can be obtained from the RGB image and its corresponding depth image, as well as the actual position and orientation of marker 1011.
[0092] In some embodiments, the training device 104 uses the following formula to calculate the coordinates P of multiple marker points in the world coordinate system. world Convert to coordinates P in the camera coordinate system camera :
[0093] P camera =R wc ·P world +t wc
[0094] Among them, R wc and t wc R is an external parameter of the image acquisition device 102. wc Let t be a rotation matrix. wc is the translation vector, representing the position and orientation of the world coordinate system relative to the camera coordinate system, respectively.
[0095] In at least one example, R wc It is a 3x3 rotation matrix, t wc It is a 3*1 translation vector, R wc and t wc It can be obtained using the Perspective-n-Point (PnP) algorithm, or other algorithms, by calculating R. wc and t wc This allows us to obtain the transformation relationship from the world coordinate system to the camera coordinate system, enabling us to transform the position and orientation of marker 1011 in the world coordinate system for representation in the camera coordinate system.
[0096] by Figure 8 For example, the perspective point algorithm is used to calculate R. wc and t wc Given P i The coordinates of point P in the world coordinate system and P iThe coordinates of a point in the pixel coordinate system are calculated using a formula to obtain the rotation matrix and translation vector of the camera coordinate system relative to the world coordinate system, thus enabling the conversion of coordinates in the world coordinate system to coordinates in the camera coordinate system.
[0097] In some embodiments, the training device 104 uses the following formula to calculate the coordinates P of multiple marker points in the camera coordinate system. camera Convert to coordinates P in the CT coordinate system ct :
[0098] P ct =R ct ·P camera +t ct
[0099] Among them, R ct Let t be a rotation matrix. ct Let R be the translation vector, and let R be the position and orientation of the CT coordinate system relative to the camera coordinate system, respectively. ct and t ct It can be obtained by Singular Value Decomposition (SVD), or other algorithms can be used.
[0100] In at least one example, the training device 104 uses RGB images and corresponding depth images from multiple angles, and utilizes singular value decomposition to calculate R. ct and t ct Furthermore, the least squares method is used to improve the rotation matrix R mentioned above. ct Translation vector t ct The accuracy.
[0101] By calculating R ct and t ct This allows us to obtain the transformation relationship from the camera coordinate system to the CT coordinate system, and further transform the position and orientation of marker 1011 into the CT coordinate system for representation.
[0102] In the above embodiments, it should be noted that the marker 1011 is fixed to the sleeve 1012 and moves with the sleeve 1012. Therefore, the transformation relationship between the world coordinate system and the camera coordinate system, that is, R... wc and t wc The value of changes with the movement of the biopsy needle in RGB images at different angles and in the corresponding depth images; for the transformation between the camera coordinate system and the CT coordinate system, R ct and t ct The value of R can be directly calculated based on the relative positions of the image acquisition device and the CT device. Furthermore, since the relative positions of the image acquisition device 102 and the CT device are fixed in most cases during a biopsy, R...ct and t ct The value is also fixed.
[0103] According to the above embodiment, the network model is trained so that the network model obtains the coordinates of multiple marker points of marker 1011 in the CT coordinate system, and outputs the estimated position and pose of marker 1011 based on the coordinates, that is, the representation of marker 1011 in the CT coordinate system.
[0104] It should be noted that in the above embodiments, only the training process of the transformation of the marker 1011 under different coordinate systems has been described. This application is not limited to this. The training device 104 can train all other parts in the network model, such as the segmentation network, RGB network, point cloud network, etc., so as to obtain a trained network model.
[0105] In some embodiments, such as Figure 5 As shown, the positioning device can further calculate the estimated position and orientation of the biopsy needle, which is fixed relative to the marker 1011, based on the estimated position and orientation of the marker 1011. That is, it calculates the position and orientation of the biopsy needle in the CT coordinate system and obtains the insertion point and angle of the biopsy needle based on the estimated position and orientation of the biopsy needle.
[0106] In some embodiments, such as Figure 1 As shown, the biopsy needle positioning system 100 also includes a display device 105, which can display the insertion point and insertion angle of the biopsy needle.
[0107] In some possible implementations, the display device 105 can display the insertion point and angle of the biopsy needle. For example, when the operator moves the biopsy needle, the display device 105 can display a green or other colored bounding box to prompt the operator to operate at the current insertion point and angle or at other insertion points and angles. Alternatively, a prompting device (not shown) can be used to provide voice prompts to inform the operator of the correct insertion point and angle of the biopsy needle.
[0108] In at least one example, the CT device can be calibrated for the relative position of the image acquisition device and the CT device, ensuring that the position and orientation between the CT coordinate system and the camera coordinate system do not change due to external factors, thus preventing R from being affected. ct and t ct The calculations are inaccurate. Therefore, the estimated position and orientation of the marker 1011 can be further calibrated so that the display device 105 can provide a more accurate biopsy needle insertion point and biopsy needle insertion angle.
[0109] According to the above embodiments, the cost of biopsy is reduced and the work efficiency is improved.
[0110] The above embodiments are merely illustrative examples of embodiments of this application, but this application is not limited thereto, and appropriate modifications can be made based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.
[0111] As described in the above embodiments, by setting a marker fixed on the cannula carrying the biopsy needle, the true position and orientation of the marker are estimated based on the acquired RGB image and depth image corresponding to the marker, and the predicted position and orientation of the marker are obtained using a network model. Therefore, by using the position and orientation of a marker fixed relative to the biopsy needle to calculate the insertion point and angle of the biopsy needle, real-time and accurate positioning of the biopsy needle can be achieved without a robotic arm or recalibration process, thereby saving biopsy costs and improving work efficiency.
[0112] This application also provides a biopsy needle positioning method, and the same content as the foregoing embodiments will not be repeated.
[0113] Figure 9 This is a schematic diagram of a biopsy needle localization method according to an embodiment of this application, as shown below. Figure 9 As shown, the biopsy needle localization method includes the following steps:
[0114] 901. Acquire an RGB image of the marker and a depth image corresponding to the RGB image, wherein the marker is connected and fixed to a cannula with a biopsy needle fixed relative to the marker;
[0115] 902. Using the trained network model, extract the RGB features and point cloud features of the marker based on the RGB image and depth image of the marker;
[0116] 903. Based on the RGB features and point cloud features, estimate the predicted position and orientation of the marker, and then obtain the insertion point and insertion angle of the biopsy needle.
[0117] It is worth noting that the above appendix Figure 9 The embodiments of this application have only been illustrated schematically, and the application is not limited thereto. For example, the execution order between various operations can be appropriately adjusted, and other operations can be added or some operations can be removed. Those skilled in the art can make appropriate modifications based on the above description, and are not limited to the above-described embodiments. Figure 9 The records.
[0118] It is worth noting that the above description only covers the steps relevant to this application, but this application is not limited thereto. The biopsy needle localization method may also include other steps, and for details of these steps, please refer to relevant technologies.
[0119] This application also provides a medical device system. It includes a biopsy needle positioning system 100 as described in the first aspect of the embodiment, the contents of which are incorporated herein by reference.
[0120] The following is an exemplary description of the medical device system.
[0121] The medical device systems described in this article, that is, devices and systems for acquiring medical imaging data, are applicable to a variety of medical imaging modalities, including but not limited to CT (computed tomography) equipment, PET (positron emission tomography)-CT, or any other suitable medical imaging equipment.
[0122] The system for acquiring medical imaging data may include the aforementioned medical imaging equipment, a separate computer device connected to the medical imaging equipment, or a computer device connected to an internet cloud, which is connected via the internet to the medical imaging equipment or a storage device for storing medical images. The imaging method may be implemented independently or in combination by the aforementioned medical imaging equipment, the computer device connected to the medical imaging equipment, and the computer device connected to the internet cloud.
[0123] Exemplarily, embodiments of this application have been described above in conjunction with X-ray computed tomography (CT) equipment. Those skilled in the art will understand that embodiments of this application can also be applied to other medical imaging equipment.
[0124] Figure 10 This is a schematic diagram of a CT imaging device according to an embodiment of this application, illustrating the CT imaging device 1000. For example... Figure 10 As shown, the CT imaging device 1000 includes a scanning gantry 1001 and a patient table 1002; the scanning gantry 1001 has an X-ray source 1003, which projects an X-ray beam toward a detector assembly or collimator 1004 on the opposite side of the scanning gantry 1001. The subject 1005 can lie flat on the patient table 1002 and move into the scanning gantry opening 1006 as the patient table 1002 moves; medical image data of the subject 1005 can be obtained by scanning with the X-ray source 1003.
[0125] Figure 11 This is a schematic diagram of a CT imaging system according to an embodiment of this application, illustrating a block diagram of the CT imaging system 1100. Figure 11 As shown, the detector assembly 1104 includes multiple detector units 1104a and a data acquisition system (DAS) 1104b. The multiple detector units 1104a sense projected X-rays passing through the object being detected 1105.
[0126] DAS1104b converts the collected information into projection data for subsequent processing based on the sensing of detector unit 1104a. During the scan that acquires X-ray projection data, the scanning gantry 1101 and the components mounted thereon rotate around the rotation center 1101c.
[0127] The rotation of the scanning gantry 1101 and the operation of the X-ray source 1103 are controlled by the control mechanism 1103 of the CT imaging system 1100. The control mechanism 1103 includes an X-ray controller 1103a that provides power and timing signals to the X-ray source 1103, and a scanning gantry motor controller 1103b that controls the rotational speed and position of the scanning gantry 1101. The image reconstruction unit 1104 receives projection data from the DAS 1104b and performs image reconstruction. The reconstructed image is transmitted as input to the computer 1105, which stores the image in a mass storage device 1106.
[0128] Computer 1105 also receives commands and scanning parameters from the operator via console 1107. Console 1107 has some form of operator interface, such as a keyboard, mouse, voice-activated controller, or any other suitable input device. An associated display 1108 allows the operator to view reconstructed images and other data from computer 1105. Commands and parameters provided by the operator are used by computer 1105 to provide control signals and information to DAS 104b, X-ray controller 1103a, and scanning gantry motor controller 1103b. Additionally, computer 1105 operates patient table motor controller 1109, controlling patient table 1102 to position the subject 1105 and scanning gantry 1101. Specifically, patient table 1102 moves the subject 1105, wholly or partially, via... Figure 11 The scanning rack opening is 1106.
[0129] The above illustrations depict devices and systems for acquiring medical imaging data (or medical images or medical image data) according to embodiments of this application, but this application is not limited thereto. Medical imaging devices may be CT equipment, PET-CT, or any other suitable imaging equipment. Storage devices may be located within the medical imaging device, on a server outside the medical imaging device, in a separate medical image storage system (such as a PACS, Picture Archiving and Communication System), and / or in a remote cloud storage system.
[0130] Furthermore, medical imaging workstations can be located locally on the medical imaging equipment, meaning they are situated close to the equipment, and both can be located in the same scanning room, radiology department, or within the same hospital. Meanwhile, the medical image cloud platform analysis system can be located away from the medical imaging equipment, for example, in the cloud where it communicates with the equipment.
[0131] As an example, after a medical institution completes an imaging scan using medical imaging equipment, the scanned data is stored in a storage device. A medical imaging workstation can directly read the scanned data and perform image processing through its processor. As another example, a medical image cloud platform analysis system can remotely access medical images stored in the storage device to provide "Software as a Service" (SaaS). SaaS can exist between hospitals, between hospitals and imaging centers, or between hospitals and third-party online medical service providers.
[0132] This application also provides a non-transitory computer-readable medium storing a computer program having at least one code segment that can be executed by a machine to cause the machine to perform the steps of the method described in the foregoing embodiments. Since the specific implementation of the method has been described in the preceding embodiments, its content is incorporated herein by reference and will not be repeated here.
[0133] The methods described above in this application can be implemented in hardware or in combination with software. This application relates to a computer-readable program that, when executed by a logic component, enables the logic component to implement the constituent parts described above, or to implement the various methods or steps described above. This application also relates to a storage medium for storing the above program, such as a hard disk, magnetic disk, optical disk, DVD, flash memory, etc.
[0134] The methods described in conjunction with the embodiments of this application can be directly embodied in hardware, software modules executed by a processor, or a combination of both. For example, one or more and / or combinations of one or more functional block diagrams shown in the figures can correspond to either software modules in a computer program flow or hardware modules. These software modules can correspond to the respective steps shown in the figures. These hardware modules can be implemented, for example, using a field-programmable gate array (FPGA).
[0135] The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor, enabling the processor to read information from and write information to the storage medium; or the storage medium can be an integral part of the processor. The processor and storage medium can reside in an ASIC. The software module can be stored in the memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the device (such as a mobile terminal) uses a high-capacity MEGA-SIM card or a high-capacity flash memory device, the software module can be stored in the MEGA-SIM card or the high-capacity flash memory device.
[0136] One or more and / or one or more combinations of functional blocks described in the accompanying drawings can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described herein. One or more and / or one or more combinations of functional blocks described in the accompanying drawings can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.
[0137] The present application has been described above with reference to specific embodiments. However, those skilled in the art should understand that these descriptions are exemplary and not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on its spirit and principles, and these modifications and variations are also within the scope of the present application.
[0138] Preferred embodiments of this application have been described above with reference to the accompanying drawings. Many features and advantages of these embodiments are apparent from this detailed description, and therefore the appended claims are intended to cover all such features and advantages of these embodiments that fall within their true spirit and scope. Furthermore, since many modifications and variations will readily occur to those skilled in the art, the embodiments of this application are not intended to be limited to the precise structures and operations illustrated and described, but rather to encompass all suitable modifications, variations, and equivalents falling within their scope.
Claims
1. A biopsy needle positioning system, characterized in that, The system includes: A marker assembly, the marker assembly including a marker and a cannula with a biopsy needle mounted on it fixed relative to the marker; An image acquisition device, installed at a location capable of capturing images of the marker, is used to acquire an RGB image of the marker and a depth image corresponding to the RGB image; and The positioning device uses a trained network model to extract the RGB features and point cloud features of the marker based on the RGB image and depth image of the marker, and estimates the predicted position and orientation of the marker based on the RGB features and point cloud features, thereby obtaining the insertion point and insertion angle of the biopsy needle.
2. The system according to claim 1, wherein, The marker assembly also includes a bracket connected to one end of the sleeve, on which the marker is mounted.
3. The system according to claim 2, wherein, The bracket is L-shaped.
4. The system according to claim 2, wherein, The marker is a cube, sphere, or cone.
5. The system according to claim 1, wherein, The positioning device uses a segmentation network to segment the RGB image and separate the marker portion from the RGB image; the marker portion is used as the input of the RGB network to extract the RGB features of the marker portion; The point cloud portion corresponding to the marker portion on the depth image is used as the input to the point cloud network to extract the point cloud features of the point cloud portion. The true position and orientation of the marker are obtained based on the RGB features and the point cloud features.
6. The system according to claim 5, wherein, The positioning device also projects the depth image into two dimensions, samples the resulting two-dimensional points, and uses a boundary prediction network to predict the boundary of the marker. The positioning device obtains the true position and orientation of the marker based on the RGB features, the point cloud features, and the boundary of the marker.
7. The system according to claim 1, wherein, The system also includes: The training device uses multiple RGB images of the marker at different angles and depth images corresponding to each RGB image acquired by the image acquisition device, as well as the actual position and pose of the marker at the aforementioned different angles, to train the network model used by the positioning device.
8. The system according to claim 7, wherein, For each RGB image and its corresponding depth image, the training device sets the coordinates P of multiple marker points on the marker in the world coordinate system. world Convert to coordinates P in the camera coordinate system camera And the coordinates P of the multiple marker points in the camera coordinate system camera Convert to coordinates P in the CT coordinate system ct The estimated position and orientation of the marker at the current angle are obtained.
9. The system according to claim 8, wherein, The training device calculates the coordinates P of the multiple marker points in the world coordinate system according to the following formula. world Convert to coordinates P in the camera coordinate system camera : P camera =R wc ·P world +t wc Among them, R wc and t wc R is an external parameter of the image acquisition device. wc Let t be a rotation matrix. wc is the translation vector, representing the position and orientation of the camera coordinate system relative to the world coordinate system, respectively.
10. The system according to claim 8, wherein, The training device calculates the coordinates P of the multiple marker points in the camera coordinate system according to the following formula. camera Convert to coordinates P in the CT coordinate system ct : P ct =R ct ·P camera +t ct Among them, R ct Let t be a rotation matrix. ct Let R be the translation vector, and let R be the position and orientation of the CT coordinate system relative to the camera coordinate system, respectively. ct and t ct It was obtained by singular value decomposition.
11. The system according to claim 8, wherein, The plurality of marker points are located at different positions on the marker, and the plurality of marker points characterize the outline of the marker.
12. The system according to claim 1, wherein, The system also includes: The display device displays the insertion point and insertion angle of the biopsy needle.
13. A method for locating a biopsy needle, characterized in that, The method includes: The RGB image of the marker and the corresponding depth image are acquired, and the marker is connected and fixed to a cannula with a biopsy needle fixed relative to the marker. Using a trained network model, RGB features and point cloud features of the markers are extracted from their RGB and depth images. Based on the RGB features and point cloud features, the estimated position and orientation of the marker are estimated, thereby obtaining the insertion point and angle of the biopsy needle.
14. A medical device system, characterized in that, The medical device system includes the biopsy needle positioning system according to any one of claims 1-12.