Ultrasonic image-based apparatus for monitoring positional deviation of surgical instrument

Abstract

An ultrasonic image-based apparatus for monitoring positional deviation of a surgical instrument, comprising a surgical instrument (112), an ultrasonic subsystem comprising an ultrasonic probe (122), and a control subsystem comprising a processor. The control subsystem is separately connected to the surgical instrument (112) and the ultrasonic subsystem, and the ultrasonic subsystem continuously acquires ultrasonic images of a target tissue site (200) and transmits the acquired series of ultrasonic image frames to the processor of the control subsystem. The surgical instrument (112) moves in the target tissue site (200) according to a surgical planning path set by the processor. The tail end of the surgical instrument (112) is provided with an arc-shaped top end portion (22). The processor comprises an image processing program. The image processing program performs calculation on the basis of the series of ultrasonic image frames, so as to determine a deviation between the position of a tracking sub-region in a current image frame and that of the tracking sub-region in a previous image frame, the tracking sub-region being a region comprising the arc-shaped top end portion (22) in the ultrasonic image. The apparatus can implement accurate, real-time, and effective monitoring of the displacement of the surgical instrument (112) on the basis of the ultrasonic image, ensuring surgical safety.

Description

A surgical instrument position deviation monitoring device based on ultrasonic images Technical Field

[0001] The present application relates to the field of medical devices, and in particular to a surgical instrument position deviation monitoring device based on ultrasonic images. Furthermore, it particularly relates to a displacement monitoring device for a water jet during a water jet minimally invasive surgical robot operation. Background Art

[0002] Currently, the medical device sector is experiencing rapid development in automated surgical planning technologies, such as surgical navigation and surgical planning. Typically, after a surgical plan is developed, the surgical instruments, driven by a surgical robot, move according to the planned path and complete the procedure. For such surgeries, monitoring the displacement of surgical instruments is particularly important. Unnecessary positional deviations of the surgical instruments during surgery can make it difficult to complete the procedure according to the planned path, or even if the procedure is completed, it may deviate significantly from the pre-set plan. Even slight deviations in position can significantly reduce surgical accuracy.

[0003] For example, a waterjet minimally invasive surgery system based on surgical planning consists of a waterjet subsystem, ultrasound equipment, and a corresponding control system. During a prostate surgery, for example, a waterjet procedure uses a robotic arm to insert the waterjet into the target tissue area, guided by a rectal ultrasound probe. Once the waterjet is in place, the host computer software plans the waterjet's scanning path and water jet intensity. The robotic arm guides the waterjet and performs the surgery using the planned parameters. From planning to completion, the positions of the waterjet, ultrasound probe, and patient must remain unchanged; otherwise, the planned parameters become invalid and the surgery could fail.

[0004] However, such positional deviations are inevitable during surgery. For example, unwanted movement from the patient, operator, or other surroundings can cause surgical instruments to shift, making planned procedures unsustainable. Therefore, monitoring the displacement of surgical instruments is crucial to ensure accurate and safe surgery, and is a key factor in determining surgical quality.

[0005] In the existing technology, the common displacement monitoring idea is to use optical and / or magnetic navigation to calibrate and record the spatial position of the water jet and ultrasonic probe. This method requires the installation of additional sensors or marking points and a pre-calibration process to monitor the spatial position movement of the water jet and ultrasonic probe. The cost is high, and the additional markers also need to consider other processes such as disinfection, which brings many inconveniences.

[0006] Some researchers have also proposed using images to monitor displacement, but this approach presents numerous challenges. For one thing, imaging methods such as X-rays, CT scans, and MRI, which can produce high-definition images, are difficult to directly capture in real time during surgery due to the radioactivity, bulkiness, and slow imaging process of their equipment. This makes their use in monitoring real-time displacement even more impossible. Ultrasound imaging is a common imaging method used during surgery, but its inherent image quality makes it extremely difficult to identify surgical instruments in real time. For example, a water jet minimally invasive surgical system appears as a long straight line in an ultrasound image, which is closely spaced from the sheath or diagnostic endoscope used as an insertion aid (as shown in Figure 3). This results in numerous artifacts during ultrasound imaging, making displacement monitoring nearly impossible.

[0007] Summary of the Invention

[0008] The present application proposes a device for monitoring the position deviation of a surgical instrument based on ultrasonic images, comprising: a surgical instrument, an ultrasonic subsystem including an ultrasonic probe, and a control subsystem including a processor, wherein the control subsystem is connected to the surgical instrument and the ultrasonic subsystem respectively; the surgical instrument moves in the target tissue area according to the surgical planning path set by the processor; the device is characterized in that: the end of the surgical instrument is provided with an arc-shaped top portion, the arc-shaped top portion includes a curved outer surface and an inner surface, the inner surface is an arc-shaped surface that is relatively closer to the injection area of ​​the surgical instrument, and the outer surface is an arc-shaped surface that is relatively further away from the injection area of ​​the surgical instrument; after the surgical instrument is in place, the ultrasonic probe begins to continuously acquire ultrasonic images of the target tissue area and transmits the acquired series of ultrasonic image frames to the processor of the control subsystem, The processor includes an image processing program, which performs calculations based on the series of ultrasound image frames to determine the position deviation between the tracking sub-area in the current frame image and the tracking sub-area in the previous frame image. The tracking sub-area is a local area in the ultrasound image that contains the arc-shaped top portion. The image processing program extracts the image features of the tracking sub-area portion through a feature extraction method to obtain the image features of the arc-shaped top portion for subsequent displacement tracking; wherein, the surgical instrument includes a water jet and a sheath, the sheath includes a slender support member body, and the arc-shaped top portion is extended at the end of the slender support member body; or the surgical instrument includes a water jet, the water jet includes a slender knife tube and a knife head located at the end of the knife tube, and the knife head has an arc shape to form the arc-shaped top portion.

[0009] Preferably, it further includes a fixed base, which is connected to the surgical instrument via a first robotic arm and a first adapter, and is connected to the ultrasound probe via a second robotic arm and a second adapter, and the ultrasound probe is a transrectal ultrasound probe.

[0010] Preferably, the inner surface and the outer surface together form an arc-shaped top portion, and the bending angle formed is 30°-75°.

[0011] Preferably, the arc-shaped top portion comprises tungsten, and / or the arc-shaped top portion comprises a smooth outer surface and a rough inner surface, and the rough inner surface is formed by embedding irregular concave-convex pits, grooves, steps or frosting.

[0012] Preferably, the distance between the ultrasonic probe and the sheath is R, and the curvature of the arc-shaped top portion is K=1 / R.

[0013] Preferably, the tracking sub-region is a minimum image region including all arc-shaped top portions.

[0014] Preferably, when the monitored position deviation exceeds a set safety displacement distance, the device alarms or performs a surgery pause.

[0015] Preferably, the image processing program further performs the following steps: identifying and marking the tracking sub-area on the initial frame image, then updating the tracking sub-area in each subsequent frame image, and calculating the position deviation between the tracking sub-area in the current frame image and the expected tracking sub-area; the position of the expected tracking sub-area is the position of the tracking sub-area in the previous frame image, or the position of the tracking sub-area in the previous frame image and the position after superposition of the path of planned motion between the two frames.

[0016] Preferably, the center position coordinates of the tracking sub-area are set as the original coordinate origin for tracking in the initial frame image, and then the tracking sub-area is updated in each subsequent frame image and the center position coordinates of the updated tracking sub-area are calculated, and the deviation between the center position coordinates of the tracking sub-area in the current frame image and the center position coordinates of the expected tracking sub-area is calculated.

[0017] Preferably, the image processing program further displays the boundary points or center position points of the tracking sub-region in the current frame image in different colors and / or prompts text information thereof according to the size of the position deviation.

[0018] Advantageously, the image processing procedure performs identification and tracking based on a small sample of tracking sub-regions.

[0019] Preferably, the image processing program further performs the following steps: constructing an offline feature template library based on the tracking sub-region; determining the offline feature template f1(x) that best matches the initial frame image and the tracking sub-region in the initial frame image; determining the best feature template F(x) of the next frame image based on each current frame image; and determining the tracking sub-region in the next frame image based on the determined best feature template F(x).

[0020] Preferably, the step of constructing an offline feature template library based on the tracking sub-region includes the step of offline acquiring an original feature template of the tracking sub-region, wherein the original feature template is a combination of an ultrasound image of the tracking sub-region and feature information thereof.

[0021] Preferably, the step of constructing an offline feature template library based on the tracking sub-region further includes the step of constructing an offline feature template library based on the original feature template, and the offline feature template library includes multiple feature templates generated based on the original feature template.

[0022] Preferably, the offline feature template library includes offline feature templates of different depth parameters, and offline feature templates of different scales formed by determining scale factors according to different depth parameters.

[0023] Preferably, the step of determining the offline feature template f1(x) that best matches the initial frame image and the tracking sub-area in the initial frame image includes: after the ultrasound probe and the surgical instrument are in place, the ultrasound probe starts to acquire the initial frame image; based on the depth parameters of the ultrasound probe, multiple candidate feature templates with corresponding depth parameters are selected from the offline feature template library; the matching degree of the initial frame image and the multiple candidate feature templates is calculated one by one, and the feature template with the highest matching degree is recorded as the offline feature template f1(x) with the best matching degree, and the area with the highest matching degree is the tracking sub-area in the initial frame image.

[0024] Preferably, in the step of determining the optimal feature template F(x) of the next frame image based on each current frame image, the optimal feature template F(x) of the next frame image is determined based at least on the image features of the tracking sub-area in the feature template f1(x) and the image features of the tracking sub-area in the current frame image.

[0025] Preferably, in the step of determining the optimal feature template F(x) of the next frame image based on each current frame image, the deep features of the tracking sub-area in the feature template f1(x) selected from the offline feature template library, the features of the tracking sub-area in the current frame image, and the pre-trained deep learning neural network are extracted, and the optimal feature template F(x) of the next frame image is obtained through weight fusion.

[0026] Preferably, the window size of the tracking sub-area can be adaptively adjusted.

[0027] Preferably, by collecting the current depth parameter of ultrasound in real time and making a judgment, when the depth parameter changes between two adjacent ultrasound image frames, the scale factor in the parameter set is adaptively updated according to the set scale factor set at the corresponding depth.

[0028] The technical solution provided by this application can solve the aforementioned technical problems and has at least one of the following technical effects: it can overcome the technical difficulties of monitoring surgical instruments based on ultrasonic images, making it possible to accurately, in real time, and effectively monitor the displacement of surgical instruments based on ultrasonic images. Compared with traditional displacement monitoring methods, it significantly reduces costs and reduces tedious operating steps such as disinfection of additional markers; with the help of the intraoperative ultrasonic imaging subsystem of the surgical system, the displacement of surgical instruments can be monitored, and the monitoring results can be fed back and prompted in real time, providing convenience for the operator or doctor to perform the operation. At the same time, accurate and real-time displacement monitoring helps to implement the safe execution of the surgical plan in strict accordance with the surgical plan, improve the quality of the operation, and reduce medical accidents. In addition, the displacement monitoring information can also be directly used to adjust the surgical plan, which is conducive to the precise control of the automated surgical process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0030] FIG1 is a schematic structural diagram of a water jet minimally invasive surgery system provided according to an embodiment of the present application.

[0031] FIG2 is a schematic diagram of the spatial distribution of an ultrasound probe and surgical instruments provided according to an embodiment of the present application.

[0032] FIG. 3 is an ultrasonic image of a conventional water jet and a target tissue area provided according to the background art.

[0033] FIG4 is a schematic diagram of ultrasound reflection by the curved top portion of the surgical instrument provided in accordance with an embodiment of the present application.

[0034] FIG5 is a schematic diagram of the main structure of a water jet subsystem provided according to an embodiment of the present application.

[0035] FIG6 is a schematic diagram showing the reflection of ultrasonic waves by the rough inner surface of the arc-shaped top portion of the surgical instrument provided in accordance with an embodiment of the present application.

[0036] FIG. 7 is an ultrasound image including a tracking sub-region provided according to an embodiment of the present application.

[0037] FIG7 a is a local ultrasound image including a tracking sub-region provided according to an embodiment of the present application.

[0038] FIG7 b is a schematic diagram of image features of the tracking sub-region after image features of the tracking sub-region are extracted according to an embodiment of the present application.

[0039] FIG8 is a schematic diagram of tracking the displacement of a surgical instrument based on a tracking sub-area according to an embodiment of the present application.

[0040] FIG9 is a schematic diagram of a process of determining and updating a template of a tracking sub-area according to an embodiment of the present application.

[0041] FIG10 is a schematic diagram of the system workflow of the surgical instrument displacement monitoring device provided in an embodiment of the present application when used in a surgical system. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] According to an embodiment of the present application, a surgical instrument displacement monitoring device based on ultrasound images is provided, comprising: a surgical instrument, an ultrasound subsystem, and a control subsystem. The ultrasound subsystem comprises an ultrasound probe, the control subsystem comprises a processor, the control subsystem is connected to the surgical instrument and the ultrasound subsystem respectively, the ultrasound subsystem continuously acquires ultrasound images of the target tissue area and transmits the acquired series of ultrasound image frames to the processor of the control subsystem, and the surgical instrument moves in the target tissue area according to the surgical planning path set by the processor. Taking a water jet minimally invasive surgical system as an example, referring to FIG1 , the water jet minimally invasive surgical system comprises: a water jet subsystem, an ultrasound subsystem, and a control subsystem, the control subsystem is connected to the ultrasound subsystem and the water jet subsystem respectively, the control subsystem can acquire image data from the ultrasound subsystem, control parameter data from the water jet subsystem, and can also issue control instructions to adjust the imaging parameters of the ultrasound subsystem and the control parameters of the water jet subsystem. Among them, the ultrasound subsystem comprises an ultrasound probe, and when used for prostate tissue-related surgery, transrectal ultrasound TRUS (transrectal of ultrasound) is preferably used. The ultrasound subsystem continuously acquires ultrasound images of the target tissue area and transmits the acquired ultrasound images to a processor in the control subsystem. The control subsystem includes a processor that receives the images from the ultrasound subsystem and analyzes and processes the images.

[0044] Figure 2 shows a schematic diagram of the spatial arrangement of an ultrasound probe and a surgical instrument provided according to an embodiment of the present application. The fixed base 100 serving as a fixed reference component is connected to a surgical instrument 112 through a first robotic arm 110 and a first adapter 111, and is connected to an ultrasound probe 122 through a second robotic arm 120 and a second adapter 121.

[0045] When used in prostate tissue-related surgery, the surgical instrument 112 is moved through the urethra or perineum to the target tissue area 200 , and the ultrasound probe 122 inserted through the rectum is directed toward the urethra or perineum where the surgical instrument 112 is located to obtain an ultrasound image of the target tissue area 200 .

[0046] As mentioned in the background art, while the ultrasound subsystem can capture real-time images of the surgical instrument and target tissue area, the image clarity is limited. Therefore, conventional surgical planning requires the use of preoperative MR or CT images. Intraoperative ultrasound images are typically used only to assist in understanding the general situation during the procedure and are limited to physicians with extensive knowledge and experience in ultrasound imaging of target tissue. In particular, when the surgical instrument is a water jet, since both the water jet and the sheath that assists the water jet in entering the target tissue area are linear structures, the ultrasound image near the water jet contains numerous artifacts. Figure 3 shows a conventional ultrasound image of the water jet and target tissue area provided by the background art. The location of the water jet cannot be clearly determined from the ultrasound image alone, making it even more difficult to monitor the displacement of the water jet, particularly the energy jet at the end of the water jet.

[0047] To address this issue, this application first focuses on improving the shape of the surgical instrument's tip, configuring it to include a curved tip. As shown in Figure 4, ultrasound waves emitted by an ultrasound probe U are reflected back to the probe U via the curved tip M. Compared to a linear structure, more ultrasound reflection beams L are reflected back to the probe U, thereby enhancing the visibility of the affected area in the ultrasound image.

[0048] Referring to FIG5 , there is shown a schematic diagram of the main structure of the water jet subsystem in a water jet displacement monitoring device according to an embodiment of the present application. In this embodiment, the surgical instrument comprises a water jet 1 and a sheath 2. The sheath 2 is used for inserting into the target tissue area and comprises a slender support body 21. The arcuate top portion 22 is extended at the end of the slender support body 21. The water jet 1 moves along the channel defined by the sheath 2 according to the surgical planning path set by the processor. The sheath 2 is used for inserting into the target tissue area and can provide the function of fixing and / or supporting the surrounding tissue. The water jet 1 comprises a blade tube 11 and a blade head 12. The blade head 12 and the blade tube 11 move along the channel defined by the sheath 2 according to the surgical planning path set by the processor. Preferably, the arcuate top portion 22 is also provided with a suction through hole, which communicates with the suction channel inside the support body 21.

[0049] In an alternative embodiment, the water jet 1 is configured to have an arc-shaped top portion. That is, the surgical instrument includes a water jet, which includes an elongated blade tube 11 and a blade head 12 located at the end of the blade tube 11, wherein the blade head 12 has an arc shape, forming the arc-shaped top portion.

[0050] As shown in FIG5 , the arcuate top portion 22 includes a curved outer surface 22a and an inner surface 22b. In an alternative embodiment, when the water jet cutter head is configured to have an arcuate shape, the cutter head 12 includes a curved outer surface 12a and an inner surface 12b (not shown). The inner surface is a curved surface that is relatively closer to the water jet spray area, and the outer surface is a curved surface that is relatively further away from the water jet spray area. The outer surface is curved and smooth, which can be more conducive to reducing trauma to surrounding tissues during the insertion of the surgical instrument into the target tissue area, thereby providing better protection for the surrounding tissues. The inner surface is a curved arc that matches the outer surface, and together with the outer surface, it forms an arcuate top portion. In order to achieve a better protection effect during insertion into the urethra, the bending angle is preferably selected to be 30°-75°. Different from traditional straight surgical instruments (water jets or sheaths), setting the end of the surgical instrument to be curved is conducive to providing better recognition in ultrasound images.

[0051] The curved tip portion may comprise tungsten. It is advantageous to construct the curved tip portion from tungsten or to add a tungsten-containing coating to the outer and / or inner surfaces of the curved tip portion. Furthermore, the entire surgical instrument may also be constructed from tungsten. Because the acoustic impedance of tungsten is approximately 70 times that of human soft tissue, the curved tip portion 22 comprising tungsten may also improve its visibility in ultrasound images.

[0052] As shown in Figure 6, when the inner surface is roughened, the roughened surface can be formed by embedding a large number of irregularly shaped pits or grooves, steps, or frosted surfaces. Roughening the inner surface, particularly when the roughened surface includes an irregular structure comprising a random combination of acute and obtuse angles, is beneficial for improving the visibility of the curved tip in ultrasound images. This is because the roughened arc surface enhances the ultrasonic waves reflected into the ultrasound probe sensor while suppressing or reducing ultrasonic waves reflected in other directions.

[0053] In a preferred embodiment, the ultrasound probe in the ultrasound subsystem is a convex array probe, and the curvature K of the arc-shaped top portion is related to the distance R between the ultrasound probe and the sheath. The curvature K is calculated as K=1 / R.

[0054] In the prostate tissue water jet surgery scenario, if the distance from the urethra or perineum (where the sheath is located) to the rectum (where the ultrasound probe is located) after the water jet is inserted is R, it is preferred to set the curvature K of the arc top to 1 / R. The arc top with this curvature range can obtain a better reflection and development effect in the ultrasound image, providing better recognition.

[0055] Through one or more of the above settings, the recognition of surgical instruments in ultrasonic images can be greatly improved. The ultrasonic image obtained in a certain embodiment is shown in Figure 7. The outline of the arc-shaped top portion 22 can be seen more clearly in the rectangular frame in Figure 7, making it possible to extract image features based on the outline within the rectangular frame area and track based on the extracted features.

[0056] In the water jet subsystem shown in Figure 5, the water jet moves along the planned path. Its typical motion modes include the following two:

[0057] Movement Mode 1: Driven by the first robotic arm 110 or similar component, the sheath and water jet move as a unit to a first target position, or return from the first target position to the initial position. In this scenario, the relative positions of the sheath and water jet are fixed, as are the relative positions of the water jet and the curved tip of the sheath. The water jet is fixed relative to the curved tip, but displacement of the water jet may occur, for example due to movement of the sheath as a whole. Therefore, the displacement of the curved tip is equal to the displacement of the water jet.

[0058] Movement mode 2: The sheath remains fixed at the first target position, and the water jet moves along the path defined by the sheath to the second target position, or returns from the second target position to the first target position. In this case, the movement of the water jet relative to the sheath is the movement of the water jet when it is driven by the control subsystem to perform the ablation / cutting operation under the guidance of the surgical planning plan, and its movement distance can be calculated based on the surgical planning plan and / or the water jet control parameters. At this time, the displacement of the water jet includes both the displacement caused by unnecessary movement of the sheath and the displacement caused by deviation of the water jet control parameters. Since the problem of water jet control parameter deviation can be detected and corrected in a timely manner by adopting PID feedback control, etc., it can be ignored here. Therefore, the displacement of the water jet can still be represented by the displacement of the arc top.

[0059] Based on the above considerations, this application proposes the idea of ​​tracking the displacement of the water jet by tracking the displacement of the arc-shaped top portion, and further improves the tracking effect by specifically optimizing the tracking scheme.

[0060] In this application, the image area within the rectangular frame shown in Figure 7 is defined as a tracking subregion. In this application, a tracking subregion refers to the area within the ultrasound image that contains the arc-shaped top portion. The tracking subregion is preferably a portion of the entire ultrasound image. Optimally, the smallest image area that contains the entire arc-shaped top portion can be selected. By selecting a smaller area as the tracking subregion, the computational effort for tracking can be greatly reduced, improving computational speed and thus facilitating real-time tracking.

[0061] By improving the structure of the surgical instrument, the tracking sub-area where the arc-shaped top portion is located is the most recognizable area in the ultrasound image of the target tissue area, providing possibilities for image recognition and image tracking. In addition, this sub-area contains the characteristics of the arc-shaped top portion, and by tracking the displacement of the arc-shaped top portion, the displacement tracking of the surgical instrument is achieved.

[0062] Figure 7a shows a local original ultrasound image containing a tracking sub-region. The area within the rectangular frame in the figure is the tracking sub-region. The image features of the tracking sub-region are extracted by feature extraction methods, such as the gradient direction histogram, to obtain stable and clear image features of the arc top, as shown in Figure 7b, for subsequent displacement tracking.

[0063] After the surgical instrument is in place (for example, after the sheath 2 and the water jet 1 move as a whole to the first target position), the ultrasonic probe starts to acquire a series of ultrasonic image frames. The first ultrasonic image acquired is called the initial frame image. The image that undergoes specific image analysis and processing for each frame is called the current frame image. The frame image before the current frame image is called the previous frame or previous frame image, and the frame image after the current frame is called the next frame or next frame image.

[0064] Figure 8 is a schematic diagram of tracking the displacement of the area based on the tracking sub-area. The ultrasound probe continuously acquires a series of ultrasound image frames. The rectangular box on the right represents the position of the tracking sub-area determined based on the initial frame image, and the rectangular box on the left represents the position of the tracking sub-area on the current frame image. The arrow indicates the direction of the position deviation of the tracking sub-area, and the specific coordinate value of the position deviation is marked above the rectangular box on the left.

[0065] According to a preferred embodiment, the boundary or center position of the tracking sub-region in the previous frame image may also be marked as a preset original color value in the current frame image.

[0066] The boundary points or center points of the tracking sub-region in the current frame image may also be displayed in different colors according to the size of the position deviation.

[0067] Considering the resolution of ultrasound images and the impact of surgical instrument displacement on human tissue, a displacement safety distance can be set. For example, in a water jet prostate surgery scenario, a displacement safety distance of 5mm is set. When the monitored position deviation is between 0 and 5mm, the boundary or center point of the tracking sub-area in the current frame image is displayed in orange. When the monitored position deviation is greater than 5mm, the boundary or center point of the tracking sub-area in the current frame image is displayed in red.

[0068] Preferably, the value of the position deviation can also be displayed simultaneously with the current frame image.

[0069] Preferably, a predetermined safety strategy may be executed according to the position deviation. For example, when it is determined that the position deviation exceeds a set safety displacement distance threshold, an alarm may be issued and / or the operation may be suspended.

[0070] According to the displacement monitoring method provided in the embodiment of the present application, the tracking sub-area is first identified and marked on the initial frame image, and then the tracking sub-area is updated in each subsequent frame image, the tracking sub-area of ​​the current frame is determined, and the position deviation between the tracking sub-area in the current frame image and the expected tracking sub-area is calculated, and the position deviation is used as the displacement value of the water jet.

[0071] Preferably, the center position coordinates of the tracking sub-area can be set as the original coordinate origin for tracking in the initial frame image, and then the tracking sub-area is updated in each subsequent frame image and the center position coordinates of the updated tracking sub-area are calculated, and the position deviation between the center position coordinates of the tracking sub-area in the current frame image and the center position coordinates of the expected tracking sub-area is calculated, and the position deviation is used as the displacement value of the water jet.

[0072] When the arc-shaped tip portion is a fixed component (such as a sheath) during surgery, the estimated position of the tracking sub-region is the position of the tracking sub-region in the previous frame of image.

[0073] When the arc-shaped top portion is a moving part during surgery (such as a water jet cutter head), the expected position of the tracking sub-area is the position of the tracking sub-area in the previous frame image and the position after the path of the planned movement between the two frames is superimposed.

[0074] As mentioned above, due to excessive artifacts and other inherent factors in ultrasound images, it is difficult to identify surgical instruments in real time. This application first proposes the idea of ​​monitoring the displacement of surgical instruments by monitoring the displacement of the arc-shaped top part, and improves the structural design of the surgical instrument as a whole, especially the top part, so that the recognition of this area in the ultrasound image can be improved, and further combines deep learning algorithms to improve the accuracy of image feature recognition in the tracking sub-area and improve the computational efficiency of data processing.

[0075] The deep learning algorithms commonly used in the field of image processing require a large number of samples and hardware resources for training. In contrast, this application proposes a small sample recognition and tracking method based on tracking sub-regions. The method includes the following steps: constructing an offline feature template library based on the tracking sub-region, determining the current feature template in real time based on the newly input current frame image, and determining the tracking sub-region in the current frame image according to a predetermined search range and search strategy in the current frame image based on the feature information in the current feature template.

[0076] The above method steps are described in detail below with reference to FIG9 .

[0077] Step S1: construct an offline feature template library based on the tracking sub-region.

[0078] Step S11: obtaining the original feature template of the tracking sub-region offline.

[0079] The original feature template is based on a combination of the ultrasound image of the tracked subregion and its feature information, such as fHOG features. Offline acquisition methods include, but are not limited to, pre-operatively acquiring an image of the surgical instrument containing the curved tip portion using an ultrasound probe and extracting its feature information; or obtaining a pre-provided image of the surgical instrument containing the curved tip portion and extracting its feature information.

[0080] Step S12: construct an offline feature template library based on the original feature template. The offline feature template library contains multiple feature templates generated based on the original feature template. Constructing the offline feature template library may include the following steps:

[0081] S121: For the same target field of view or target tissue area, ultrasound images acquired by ultrasound probes with different depth parameters have different pixel scales. Therefore, expanding the original feature template into feature templates at different depth parameters can be applied to different depth parameter scenarios, effectively improving the accuracy of image feature matching calculations. Taking the prostate tissue water jet surgery scenario as an example, feature templates at depth parameters of 39mm, 47mm, 55mm, and 63mm can be extracted to construct a feature template library applicable to different depth parameters.

[0082] S122: Because the sheath and water jet may have angle deviations and scales when inserted into the target tissue area, n feature templates of different scales and angles are generated based on the original feature template and a pre-set scale factor. This constructs a feature template library applicable to different scales and angles, thereby effectively improving the accuracy of image feature matching calculations.

[0083] Taking the prostate tissue water jet surgery scenario as an example, unlike general target tracking, after the water jet is inserted into the target tissue area, the sheath and the ultrasound probe are usually required to be basically level. The scale of the water jet in the ultrasound image will not change significantly when the ultrasound depth is fixed. The main reason for the scale change is the switching of the ultrasound depth parameter and the depth of the water jet in the current frame. At the same depth, considering that the ultrasound probe itself has a certain width, when the water jet undergoes a certain deflection, the scale of the projection onto the two-dimensional plane may change. Therefore, the angle parameters are preferably {0°, ±5°, ±10°}.

[0084] Furthermore, at different depth parameters, the scale factor is determined through multi-scale sampling. That is, the scale factor is determined based on the depth parameter, which can improve computational efficiency and reduce the number of template samples. For example, at a depth of 63mm, the scale factor R is set to {0.99, 1.0, 1.01}; at a depth of 55mm, the scale factor R is set to {0.98, 1.0, 1.02}; at a depth of 47mm, the scale factor R is set to {0.96, 1.0, 1.04}; and at a depth of 39mm, the scale factor R is set to {0.95, 1.0, 1.05}. The scale factor is set based on experimental results at different depths during design.

[0085] Based on steps S121 and S122, a feature template library suitable for different depths, scales, and angles can be obtained. This allows for the determination of optimal feature templates for scenarios with varying depth, scale, and angle parameters. This facilitates the elimination of unnecessary error factors during subsequent similarity matching calculations, enabling the calculated results to more accurately reflect objective positional deviation information. In particular, after extensive experimentation, the inventors of this application proposed the aforementioned parameter selection method for prostate tissue waterjet surgery, which achieves a moderate expansion of the original feature templates. The resulting offline feature template library, with a moderate data size, improves computational accuracy while also balancing computational efficiency.

[0086] Step S2: Determine the offline feature template f1(x) that best matches the initial frame image and the tracking sub-region in the initial frame image.

[0087] After the ultrasound probe and surgical instruments are in place, the ultrasound probe starts to acquire a series of ultrasound image frames. The first ultrasound image acquired is called an initial frame image.

[0088] In step S21 , based on the depth parameter of the ultrasound probe, a plurality of candidate feature templates corresponding to the depth parameter are selected from an offline feature template library, wherein the plurality of candidate feature templates are feature templates based on different angle parameters and different scale factors.

[0089] In step S22, the matching degree between the initial frame image and the multiple candidate feature templates is calculated one by one. The feature template with the highest matching degree is recorded as the best matching offline feature template f1(x). The area with the highest matching degree is the tracking subregion in the initial frame image. x is the image of the tracking subregion. Therefore, when the best offline feature template f1(x) is selected, the tracking subregion in the initial frame image is also determined accordingly.

[0090] In some cases (for example, when the difference in the amount of calculation is not significant), step S21 may be omitted, and step S22 may be directly executed to obtain the best matching offline feature template f1(x).

[0091] Step S3: updating and determining the best feature template F(x) for tracking the next frame image based on the current frame image.

[0092] After tracking begins, the ultrasound probe continuously acquires a series of ultrasound image frames of the target tissue area and transmits them to the processor. Due to the inherent poor image quality of images acquired by the ultrasound probe, contrast differences or unknown artifacts may appear in the current frame image. Therefore, directly performing search and matching operations based solely on the feature template f1(x) in the offline feature template library is ineffective and requires further correction of the feature template. Therefore, starting from the initial frame, the information of each current frame image is also combined to update and determine the optimal feature template F(x) for the next frame image. This is used for searching and matching operations in the tracking sub-region in the next frame image.

[0093] This application adopts a dynamic feature fusion extraction method, which includes: based on the features of the tracking sub-area in the feature template f1(x) selected from the offline feature template library (such as fHOG features) and the features of the tracking sub-area in the current frame image (such as fHOG features), they are fused according to preset weights to obtain the optimal feature template F(x) for the next frame image.

[0094] Furthermore, not only the features of the tracking sub-region in the feature template f1(x) selected from the offline feature template library (such as fHOG features) and the features of the tracking sub-region in the current frame image (such as fHOG features) are extracted based on the pre-trained deep learning neural network (for example, the dimensionality reduced features extracted by the Conv4 layer in resnet50 trained by imagenet100), and the best feature template F(x) for the next frame image is fused through weights.

[0095] The above-mentioned dynamic feature fusion method has at least the following advantages: by introducing the feature template f1(x) in the offline feature template library, the cumulative error caused by the continuous updating process can be effectively avoided; and the deep semantic information of the tracking sub-area in the current frame image is extracted through the pre-trained deep learning neural network, thereby effectively improving the stability of the extracted image features.

[0096] Step S4: determining the tracking sub-region in the current frame image based on the optimal feature template F(x).

[0097] Get the current frame image, search in the current frame image according to the set strategy, and find the area with the maximum response result of the best feature template F(x), which is the tracking sub-area in the current frame image.

[0098] According to a preferred embodiment of the present application, a tracking solution is also provided that can adaptively adjust the size of the tracking sub-region window in a water jet surgery scenario. Unlike general target tracking problems, the scale of the water jet in the ultrasound image does not change significantly at a fixed ultrasound depth. The main reason for the scale change is the switching of the ultrasound depth parameter, not the depth of the target in the current frame. At the same depth, considering that the ultrasound probe itself has a certain width, the target may change its scale when projected onto the two-dimensional plane when it undergoes a certain deflection. Therefore, in this embodiment, by real-time acquisition and judgment of the current ultrasound depth parameter, when the depth parameter changes between two adjacent ultrasound frames, the scale factor in the parameter set is adaptively updated according to the set of scale factors at the corresponding depth. The scale factor set is calculated based on the corresponding relationship between the current switching depth and the target depth, as well as the scale coefficients for different depths in the table below. Depth coefficients commonly used in water jet surgery, such as 39mm, 47mm, 55mm, and 63mm, are selected for depth testing to obtain the corresponding scale coefficients. The depth-scale coefficient index table is shown in the following table:

[0099] Define the scale factor R = {delta_size, 1, 1 / delta_size}, where delta_size = (current ultrasound depth scale coefficient) / (target ultrasound depth scale coefficient after switching). For example, if the current ultrasound depth of view is 47 mm and the ultrasound depth after switching is 55 mm, the current ultrasound depth scale coefficient in the index table is 1.30, and the target ultrasound depth scale coefficient after switching is 1.13. Delta_size = 1.30 / 1.13 = 1.14, and the scale factor R = {0.87, 1, 1, 14}. By selecting the scale factor under the corresponding depth parameter, all scale factors are calculated and multiplied by the original tracking sub-region window size to obtain the new tracking sub-region window size. The new tracking sub-region window size is passed through the filter H and the maximum response result is calculated to obtain the optimal scale. The tracking sub-region window size is updated in the next iteration.

[0100] In the tracking scheme that adaptively adjusts the window size of the tracking sub-area, when the depth parameter between two adjacent ultrasound image frames changes, the optimal offline feature template f1(x) also needs to be updated to the new window size. The updating steps are the same as the aforementioned step S2. Accordingly, F(x) is also updated to the adjusted window size of the tracking sub-area.

[0101] As shown in Figure 10, the surgical instrument displacement monitoring device provided by the present application is used for surgical planning, and the method and steps of the surgical planning scheme. As shown in the figure, after the planned surgery begins, a safety policy corresponding to the displacement distance is set, and the ultrasound image is identified and marked as a tracking sub-area. Tracking is performed based on the characteristics of the tracking sub-area and the tracking information is visualized. If the displacement distance is determined to exceed the safe displacement distance based on the tracking information, an alarm is issued; otherwise, the planned surgery continues.

[0102] The inventive concept of the present application is not limited to prostate tissue surgery, but is applicable to other similar surgical environments. Such similar surgical environments refer to: an ultrasound probe configured to acquire real-time ultrasound images of the target tissue region containing the surgical instrument, that is, the surgical instrument inserted into the target tissue region should be within the field of view of the ultrasound probe. Furthermore, although a water jet is used as an example in some embodiments, it should be understood that the inventive concept of the present application is not limited to a water jet. Surgical instruments (such as electrocautery, laser ablation devices, etc.) driven and controlled by a surgical robot or a robotic arm and moving based on a planned surgical path to perform surgery can also adopt the ultrasound image-based monitoring solution proposed in this application when their displacement needs to be monitored.

[0103] The embodiments provided by the present application can solve the aforementioned technical problems and have at least one of the following technical effects: it overcomes the technical difficulties of monitoring the displacement of surgical instruments based on ultrasonic images, making it possible to accurately, in real time, and effectively monitor the displacement of surgical instruments based on ultrasonic images. Compared with the traditional method of monitoring the displacement of surgical instruments, it significantly reduces costs and reduces tedious operating steps such as disinfection of additional markers; with the help of the intraoperative ultrasonic imaging subsystem of the surgical system, the displacement of surgical instruments can be monitored, and the monitoring results can be fed back and prompted in real time, providing convenience for the operator or doctor during surgery. At the same time, accurate and real-time displacement monitoring helps to implement the safe execution of the surgical plan in strict accordance with the surgical plan, improve the quality of surgery, and reduce medical accidents. In addition, the displacement monitoring information can also be directly used to adjust the surgical planning plan, which is conducive to the precise control of the automated surgical process.

[0104] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A surgical instrument position deviation monitoring device based on ultrasonic images, comprising: A surgical instrument, an ultrasonic subsystem including an ultrasonic probe, and a control subsystem including a processor, wherein the control subsystem is connected to the surgical instrument and the ultrasonic subsystem respectively; the surgical instrument moves in the target tissue area according to the surgical planning path set by the processor; it is characterized in that: the end of the surgical instrument is provided with an arc-shaped top portion, the arc-shaped top portion includes a curved outer surface and an inner surface, the inner surface is an arc-shaped surface that is relatively closer to the injection area of ​​the surgical instrument, and the outer surface is an arc-shaped surface that is relatively more deviated from the injection area of ​​the surgical instrument; after the surgical instrument is in place, the ultrasonic probe begins to continuously acquire ultrasonic images of the target tissue area and transmits the acquired series of ultrasonic image frames to the processor of the control subsystem, the processor includes an image processing program, and the The image processing program performs calculations based on the series of ultrasound image frames to determine the position deviation between the tracking sub-area in the current frame image and the tracking sub-area in the previous frame image, wherein the tracking sub-area is a local area in the ultrasound image that includes the arc-shaped top portion, and the image processing program extracts the image features of the tracking sub-area portion by a feature extraction method to obtain the image features of the arc-shaped top portion for subsequent displacement tracking; wherein the surgical instrument comprises a water knife and a sheath, the sheath comprises a slender support member body, and the arc-shaped top portion is extended at the end of the slender support member body; or the surgical instrument comprises a water knife, the water knife comprises a slender knife tube and a knife head located at the end of the knife tube, the knife head has an arc shape to form the arc-shaped top portion.

2. The surgical instrument position deviation monitoring device according to claim 1, characterized in that: It also includes a fixed base, which is connected to the surgical instrument through a first mechanical arm and a first adapter, and is connected to the ultrasound probe through a second mechanical arm and a second adapter. The ultrasound probe is a transrectal ultrasound probe.

3. The surgical instrument position deviation monitoring device according to claim 1, characterized in that: The inner surface and the outer surface together form an arc-shaped top portion, and the bending angle formed is 30°-75°.

4. The surgical instrument position deviation monitoring device according to claim 1, characterized in that: The arc-shaped top portion includes tungsten, and / or the arc-shaped top portion includes a smooth outer surface and a rough inner surface, and the rough inner surface is formed by embedding irregular concave-convex pits, grooves, steps or frosting.

5. The surgical instrument position deviation monitoring device according to claim 1, characterized in that: The distance between the ultrasonic probe and the sheath is R, and the curvature of the arc-shaped top end is K=1 / R.

6. The surgical instrument position deviation monitoring device according to claim 1, characterized in that: The tracking sub-region is the smallest image region including all arc-shaped top portions.

7. The surgical instrument position deviation monitoring device according to claim 1, characterized in that: When the monitored position deviation exceeds the set safety displacement distance, the device alarms or performs a surgery pause.

8. The surgical instrument position deviation monitoring device according to any one of claims 1 to 7, characterized in that: The image processing program also performs the following steps: identifying and marking the tracking sub-area on the initial frame image, then updating the tracking sub-area in each subsequent frame image, and calculating the position deviation between the tracking sub-area in the current frame image and the expected tracking sub-area; the position of the expected tracking sub-area is the position of the tracking sub-area in the previous frame image, or the position of the tracking sub-area in the previous frame image and the position after superposition of the path of planned movement between the two frames.

9. The surgical instrument position deviation monitoring device according to any one of claims 1 to 7, characterized in that: The image processing program also performs the following steps: setting the center position coordinates of the tracking sub-area on the initial frame image as the original coordinate origin for tracking, then updating the tracking sub-area in each subsequent frame image and calculating the center position coordinates of the updated tracking sub-area, and calculating the deviation between the center position coordinates of the tracking sub-area in the current frame image and the expected center position coordinates of the tracking sub-area.

10. The surgical instrument position deviation monitoring device according to any one of claims 1 to 7, characterized in that: The image processing program also displays the boundary points or center position points of the tracking sub-area in the current frame image in different colors and / or prompts text information thereof according to the size of the position deviation.

11. The surgical instrument position deviation monitoring device according to any one of claims 1 to 7, characterized in that: The image processing procedure performs identification and tracking based on a small sample of tracking sub-regions.

12. The surgical instrument position deviation monitoring device according to any one of claims 1 to 7, characterized in that: The image processing program further performs the following steps: constructing an offline feature template library based on the tracking sub-region; Determine the offline feature template f1(x) that best matches the initial frame image and the tracking sub-region in the initial frame image; determine the best feature template F(x) of the next frame image based on each current frame image; determine the tracking sub-region in the next frame image based on the determined best feature template F(x).

13. The surgical instrument position deviation monitoring device according to claim 12, characterized in that: The step of constructing an offline feature template library based on the tracking sub-region includes the step of offline acquiring an original feature template of the tracking sub-region, wherein the original feature template is a combination of an ultrasound image of the tracking sub-region and feature information thereof.

14. The surgical instrument position deviation monitoring device according to claim 13, characterized in that: The step of forming an offline feature template library based on the tracking sub-region further includes the step of forming an offline feature template library based on the original feature template, wherein the offline feature template library includes a plurality of feature templates generated based on the original feature template.

15. The surgical instrument position deviation monitoring device according to claim 14, characterized in that: The offline feature template library includes offline feature templates with different depth parameters, and offline feature templates with different scales formed by determining scale factors according to different depth parameters.

16. The surgical instrument position deviation monitoring device according to claim 12, characterized in that: The steps of determining the offline feature template f1(x) that best matches the initial frame image and the tracking sub-area in the initial frame image include: after the ultrasound probe and the surgical instrument are in place, the ultrasound probe starts to acquire the initial frame image; based on the depth parameters of the ultrasound probe, multiple candidate feature templates with corresponding depth parameters are selected from the offline feature template library; the matching degree of the initial frame image and the multiple candidate feature templates is calculated one by one, and the feature template with the highest matching degree is recorded as the offline feature template f1(x) with the best matching degree, and the area with the highest matching degree is the tracking sub-area in the initial frame image.

17. The surgical instrument position deviation monitoring device according to claim 12, characterized in that: In the step of determining the best feature template F(x) for the next frame image according to each current frame image, the best feature template F(x) for the next frame image is determined based at least on the image features of the tracking sub-area in the feature template f1(x) and the image features of the tracking sub-area in the current frame image.

18. The surgical instrument position deviation monitoring device according to claim 17, characterized in that: In the step of determining the best feature template F(x) for the next frame image according to each current frame image, the deep features of the tracking sub-area in the current frame image are extracted based on the features of the tracking sub-area in the feature template f1(x) selected from the offline feature template library, the features of the tracking sub-area in the current frame image, and the pre-trained deep learning neural network, and the best feature template F(x) for the next frame image is obtained through weight fusion.

19. The surgical instrument position deviation monitoring device according to claim 12, characterized in that: The window size of the tracking sub-area can be adjusted adaptively.

20. The surgical instrument position deviation monitoring device according to claim 19, characterized in that: By collecting the current depth parameters of ultrasound in real time and making judgments, when the depth parameters between two adjacent frames of ultrasound images change, the scale factors in the parameter set are adaptively updated according to the set scale factor set at the corresponding depth.

Images