A water jet AR display method, device and system based on ultrasonic image

Abstract

The present application provides a water jet AR display method, device and system based on ultrasonic images, the method comprising the steps of: obtaining a template image of a surgical instrument including a water jet cutter head, and marking a plurality of first key feature points in the template image, wherein the first key feature points include a knife hole feature point; obtaining an ultrasonic image of the surgical instrument including a water jet cutter head and identifying a plurality of second key feature points in the ultrasonic image, wherein the second key feature points correspond to the first key feature points, and the plurality of second key feature points also include a knife hole feature point; forming the water jet AR display image based on the ultrasonic image based on the position coordinates of the plurality of first key feature points in the template image, and the position coordinates of the plurality of second key feature points in the ultrasonic image; the present application can provide effective visual assistance for medical operations, reduce surgical risks, and improve medical effects.

Description

Technical Field

[0001] The present application relates to a water jet AR display method, device and system based on ultrasonic images. Background Art

[0002] In surgical scenarios, due to the real-time and convenience of ultrasound images, real-time ultrasound images are usually used to assist surgical navigation or monitor surgical progress. The ultrasound probe is supported by a mechanical arm or a bracket or a similar mechanism to provide real-time image information. When using a water jet to perform minimally invasive surgery or similar operations, such as inserting a water jet through the urethra into the patient's prostate tissue to perform surgery, in order to improve the accuracy of the surgery, it is necessary to use real-time ultrasound images to plan paths and provide navigation. However, ultrasound images are sometimes difficult to provide sufficiently clear and accurate displays. Accurate reading of ultrasound image information usually requires more experience or special attention. If the operator does not have enough professional knowledge of ultrasound imaging, it is difficult to accurately understand the information in the ultrasound image. Even experienced doctors need to pay special attention to reading ultrasound images in complex surgical scenarios. Especially in water jet surgery scenarios, the special operating environment of the water jet and surrounding tissues makes it more difficult to accurately read ultrasound image information. The doctor's reading speed slows down and there may be errors, which may bring uncertainty to path planning and surgical operations, which is undesirable.

[0003] And it is not limited to surgical scenarios. In non-surgical scenarios, when the operator places the operating component water jet in an environment that cannot be seen intuitively, the operator may also need to use ultrasound images to obtain environmental information. In order to improve the accuracy of the operation and improve the recognition of ultrasound image information to further improve the operator's user experience, higher requirements are also placed on ultrasound image information acquisition and image display. Summary of the invention

[0004] In order to solve the above problems, the present application provides a water jet AR display method, device and system based on ultrasonic images. The provided water jet AR display method based on ultrasonic images includes the following steps:

[0005] Acquire a template image of a surgical instrument including a water jet cutter head, and mark a plurality of first key feature points in the template image, wherein the first key feature points include a cutter hole feature point;

[0006] Acquire an ultrasonic image of a surgical instrument including a water jet cutter head, and identify a plurality of second key feature points in the ultrasonic image, wherein the second key feature points correspond to the first key feature points, and the plurality of second key feature points also include a cutter hole feature point;

[0007] The water jet AR display image based on the ultrasound image is formed based on the position coordinates of the multiple first key feature points in the template image and the position coordinates of the multiple second key feature points in the ultrasound image.

[0008] Furthermore, the steps of acquiring an ultrasonic image of a surgical instrument including a water jet cutter head and identifying a knife hole feature point among a plurality of second key feature points in the ultrasonic image include: controlling the water jet to perform a water spraying action, the ejected high-pressure jet forms a rapidly fluctuating air mass in a liquid immersion environment, performing jet feature recognition on the contour features of the air mass based on the detected air mass fluctuation area in the ultrasonic image, and generating a first auxiliary line for locating the knife hole feature point.

[0009] Furthermore, the step of acquiring an ultrasonic image of a surgical instrument including a water jet cutter head and identifying a knife hole feature point among a plurality of second key feature points in the ultrasonic image also includes: performing edge extraction using static features of the water jet cutter head in the ultrasonic image to generate a second auxiliary line for locating the knife hole feature point.

[0010] Furthermore, the exact position of the knife hole in the ultrasonic image is determined according to the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

[0011] Furthermore, the step of acquiring an ultrasonic image of a surgical instrument including a water jet cutter head and identifying a plurality of second key feature points in the ultrasonic image also includes: translating the water jet cutter head for a distance along the main axis of the water jet cutter head and performing a water spraying action to acquire corresponding position points; and / or, acquiring an initial position point of an endoscope included in the surgical instrument, and driving the endoscope to retract to a terminal position to acquire an endoscope stop position point; and / or, acquiring a plurality of position points on the tube wall contour of the main axis of the water jet cutter head.

[0012] Furthermore, whether the fitting result of the tube wall contour is reliable is determined based on a reliability judgment threshold. When the judgment result is considered unreliable, the water jet cutter head is controlled to translate a certain distance along the water jet cutter head main axis and perform a water spraying action to obtain a corresponding position point; and / or, the initial position point of the endoscope included in the surgical instrument is obtained, and the endoscope is driven to retract to the end position to obtain the endoscope stop position point.

[0013] The present application also proposes a water jet AR display device based on ultrasonic images, comprising the following modules:

[0014] A template image acquisition module, used to acquire a template image of a surgical instrument including a water jet cutter head, and mark a plurality of first key feature points in the template image, wherein the first key feature points include a cutter hole feature point;

[0015] An ultrasonic image acquisition module, used to acquire an ultrasonic image of a surgical instrument including a water jet cutter head and identify a plurality of second key feature points in the ultrasonic image, wherein the second key feature points correspond to the first key feature points;

[0016] The AR image generation module is used to form the water jet AR display image based on the ultrasound image based on the position coordinates of the multiple first key feature points in the template image and the position coordinates of the multiple second key feature points in the ultrasound image.

[0017] Furthermore, the ultrasonic image acquisition module also controls the water jet to perform a water spraying action, and the ejected high-pressure jet forms a rapidly fluctuating air mass in the liquid immersion environment. In the ultrasonic image, based on the detected air mass fluctuation area, the contour features of the air mass are subjected to jet feature recognition, and a first auxiliary line is generated for locating the knife hole feature point.

[0018] Furthermore, the ultrasonic image acquisition module also uses the static features of the water jet cutter head in the ultrasonic image to perform edge extraction and generate a second auxiliary line for locating the cutter hole feature point.

[0019] Furthermore, the ultrasonic image acquisition module also determines the exact position of the knife hole in the ultrasonic image according to the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

[0020] Furthermore, the ultrasonic image acquisition module also translates the water jet cutter head along the water jet cutter head main body axis for a certain distance and performs a water spraying action to obtain a corresponding position point; and / or obtains an initial position point of the endoscope included in the surgical instrument, and drives the endoscope to retract to the end position to obtain an endoscope stop position point; and / or obtains multiple position points on the tube wall contour of the water jet cutter head main body axis.

[0021] Furthermore, the ultrasonic image acquisition module also determines whether the fitting result of the tube wall contour is reliable based on a reliability judgment threshold. When the judgment result is considered unreliable, the water jet cutter head is controlled to translate a certain distance along the water jet cutter head main axis and perform a water spraying action to obtain a corresponding position point; and / or, an initial position point of the endoscope included in the surgical instrument is obtained, and the endoscope is driven to retract to the end position to obtain a stop position point of the endoscope.

[0022] The present application also proposes a water jet AR display system based on ultrasonic images, comprising: a surgical instrument suitable for being inserted into a human body to perform surgery, an ultrasonic imaging device, a processor, a memory and a display device, wherein the surgical instrument comprises a water jet cutter head and an endoscope, wherein the main axis of the water jet cutter head and the main axis of the endoscope are substantially parallel, the ultrasonic imaging device is placed below the surgical instrument, the processor is used to execute a computer program stored in the memory and executable on the processor, wherein the computer program implements the method described above when executed by the processor, and the display device is used to display the water jet AR display image based on the ultrasonic image.

[0023] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the present application makes full use of the advantages of real-time ultrasonic images in medical scenarios, provides water jet AR display based on real-time ultrasonic image information, and can superimpose the water jet structure image on the ultrasonic image according to the actual posture and proportion of the water jet structure in the ultrasonic image, so as to realize the visualization of the position of the water jet hole in the surgical instrument containing the water jet when it moves along the main axis of the water jet head, and the position of the endoscope lens when the endoscope moves along the main axis of the endoscope in the ultrasonic image, which can provide effective and intuitive visual assistance for doctors or operators, thereby significantly improving the perception ability and operation accuracy of doctors during the operation or operators during the operation, reducing surgical risks or operational errors, improving medical effects or operation success rates, and comprehensively improving the accuracy and safety of surgery or operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 It is a schematic diagram of a usage scenario of a surgical instrument including a water jet according to an embodiment of the present application.

[0026] Figure 2 It is a schematic diagram of the structure of a surgical instrument including a water jet according to an embodiment of the present application.

[0027] FIG3( a ) is a schematic diagram of an ultrasonic image of a water jet cutter head under an ideal condition.

[0028] FIG3( b ) is a schematic diagram of an ultrasonic image of a water jet cutter head in an actual surgical scenario.

[0029] Figure 4 It is a schematic diagram illustrating the principle of feature recognition of a water jet ejected from a nozzle according to an embodiment of the present application.

[0030] Figure 5This is a schematic diagram showing the distortion of the main axis of the water jet cutter head in the ultrasound image.

[0031] Figure 6 Schematic diagram of a water jet AR display interface based on ultrasound images according to an embodiment of the present application.

[0032] Figure 7 This is a schematic diagram of an interface for providing AR display of the position and working status of a water jet based on an ultrasound image during a water jet surgery according to an embodiment of the present application.

[0033] Figure 8 The figure is a flowchart of a method embodiment according to the present application.

[0034] Fig. 9 Schematic diagram of modules included in a device according to an embodiment of the present application.

[0035] Fig.10 Schematic diagram of a system structure according to an embodiment of the present application.

[0036] Reference numerals:

[0037] 1 Water jet cutter head; 10 Water jet cutter head top; 10a Cutting hole; 11 Water jet cutter head main shaft; 110 Water jet adapter; 2 Ultrasonic probe; 21 Ultrasonic probe main shaft; 210 Ultrasonic adapter; 3 Endoscope; 30 Endoscope top; 31 Endoscope main shaft; 4 Sheath; 40 Sheath top; 41 Sheath main shaft; 5 Shaft sleeve; 51 Retractable main shaft DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings.

[0039] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0040] like Figure 1As shown, after the water jet cutter head 1 enters the target tissue area, it sprays a high-pressure water jet through the cutter hole 10a at the front end of the cutter head to perform surgery, and uses ultrasound probe imaging to monitor the surgical process or provide surgical navigation. In the scenario of ablation of prostate hyperplasia, the ultrasound probe 2 is inserted through the rectum below the target tissue area, and the imaging element of the ultrasound probe 2 is facing upward, that is, roughly toward the direction of the water jet cutter head, so that the ultrasound probe 2 can be used to obtain an image of the water jet cutter head 1 and the target prostate tissue area where it is located, and the ultrasound image can be used to monitor the surgical process of the water jet cutter head 1 in real time. Specifically, the ultrasound probe 2, for example, uses a dual-plane ultrasound probe to obtain a cross-sectional image and a sagittal image of the water jet cutter head 1; the ultrasound probe 2 can also use a three-dimensional ultrasound probe to directly obtain a three-dimensional image of the water jet cutter head 1 and the target tissue area where it is located. After the water jet head 1 completely enters the area where the target tissue is located, at least the sagittal image, preferably in combination with the cross-sectional image, can be used to obtain information reflecting the real-time movement of the water jet A and display it through a display device, thereby providing the doctor or operator with basic data for surgical planning and providing surgical process monitoring data.

[0041] In order to ensure that the sagittal ultrasound image obtains the real-time motion information of the process of performing surgery by the high-pressure water jet of the water jet head 1, the imaging element of the ultrasound probe 2 should be placed below the water jet head 1 and facing the water jet head 1, the ultrasound probe 2 includes an ultrasound probe main axis 21, the water jet head includes a water jet head main axis 11, the ultrasound probe main axis 21 is maintained in the working position by the ultrasound adapter 210 located at the proximal end, and the water jet head main axis 11 is maintained in the working position by the water jet adapter 110 located at the proximal end. During the working process, it is ideal that the ultrasound probe main axis 21 and the water jet head main axis 11 remain parallel to each other, the ultrasound probe clamped by the ultrasound adapter 210 can translate along the extension direction of the main axis 21, and the water jet head clamped by the water jet adapter 110 can translate along the extension direction of the water jet head main axis 11, and the translation directions of the two should be as parallel as possible.

[0042] In certain embodiments of the present application, an endoscope 3 is also included in the area where the target tissue is located simultaneously with the water jet cutter head 1. The endoscope 3 includes an endoscope main body axis 31. The endoscope 3 and the water jet cutter head 1 are inserted into the area where the target tissue is located via the same channel or different but parallel channels. During operation, the endoscope main body axis 31, the ultrasound probe main body axis 21, and the water jet main body axis 11 are substantially parallel to each other.

[0043] Figure 2FIG. 1 is a schematic diagram of a surgical instrument including a water jet according to an embodiment of the present application. In this embodiment, the surgical instrument includes a sheath, a water jet head, and an endoscope. As shown in the figure, the front end of the surgical instrument includes a sleeve 5, and the sleeve 5 includes a retractable main shaft 51. The retractable main shaft 51 is provided with a retractable channel for the water jet head 1 and a retractable channel for the endoscope 3. Figure 2 In the state shown, the water jet cutter head 1 and the endoscope 3 are both retracted in the telescopic main shaft 51, and the top portion 10 of the water jet cutter head and the top portion 30 of the endoscope are both located in the telescopic channel of the telescopic main shaft 51. Although not shown in the figure, those skilled in the art should understand that the water jet cutter head 1 and the endoscope 3 can move independently or jointly in the telescopic channel of the telescopic main shaft 51, and the basic movement of the water jet cutter head 1 and the endoscope 3 includes translation along the direction defined by the channel. The water jet cutter head 1 and the endoscope 3 are both main bodies with a certain rigidity, and the top portion 10 of the water jet cutter head and the top portion 30 of the endoscope can continue to extend forward from the top portion of the sleeve 5. Preferably, the front end of the surgical instrument also includes a sheath 4, and the sheath 4 includes a main shaft 41 and a top portion 40.

[0044] The endoscope and the water jet are components whose movements can be controlled independently. For example, the movement control process can be that the sheath 4, the water jet head 1, and the endoscope 3 are first brought into the target operation area as a whole. After they are in place, they are ready for subsequent planning or surgery. This position is defined as the original position. The original position means: the position when the sheath 4, the water jet head 1, and the endoscope 3 are assembled and inserted into the target tissue area in the form of a closure, for example. In this position, the sheath 4, the water jet head 1, and the endoscope 3 maintain a predetermined assembly relationship and are ready to start the subsequent operation steps. Afterwards, the sheath 4 remains in place, which is the original position of the sheath, and the endoscope 3 is withdrawn from the original position of the endoscope to the predetermined observation position, and the water jet head 1 is withdrawn from the original position of the water jet head to the predetermined initial position near the endoscope 3.

[0045] Ideally, the image of the water jet cutter head acquired by the ultrasonic probe 2 located below the water jet cutter head 1 should be as shown in FIG. 3 (a). Without considering any interference, the position of the sheath top portion 40 of the sheath 4, the position of the water jet cutter head top portion 10 of the water jet cutter head 1 and / or the position of the endoscope top portion 30 of the endoscope 3 can be clearly seen from the ultrasonic image.

[0046] However, due to the limitation of the imaging level of the ultrasonic image, and because the sheath main axis 41, the water jet cutter main axis 11, and the endoscope main axis 31 are all roughly horizontal, the three channels are parallel and very close to each other, and are close to each other in the ultrasonic image. The real-time image obtained by the ultrasonic probe is very likely to produce artifacts, and then the image information of the tissue area where the surgical instrument is inserted is superimposed. In particular, during the water jet surgery, the end of the water jet cutter 1 may have an extended virtual shadow or the straight line where the water jet cutter main axis is located may be deformed in the ultrasonic image due to artifacts or the water jet cutter main axis is not completely facing the sagittal plane. After the endoscope 3 is retracted to the predetermined observation position, the image between the endoscope 3 and the sheath 4 may also have line breaks. The above situations make the image information of the water jet cutter 1 and the endoscope 3 seen from the sagittal ultrasonic image relatively blurred, as shown in Figure 3 (b). In this case, even experienced doctors find it difficult to quickly and accurately identify key information such as the position of the top portion 40 of the sheath, the top portion 10 of the water jet cutter head and / or the top portion 30 of the endoscope in the image, making it difficult to effectively use real-time ultrasound images for surgical guidance and monitoring during the operation, and also difficult to achieve accurate intraoperative image planning.

[0047] Therefore, it is necessary to provide a solution that can provide clear, accurate, and vivid real-time image display of surgical instruments including water jets. Such display is provided at least through sagittal ultrasound images and can also be provided through three-dimensional ultrasound images. The provided visualization allows doctors or users to observe at a glance the accurate position and morphological information of each component in the surgical instrument including water jets and plan or decide the next operation accordingly, thereby implementing more accurate planning or surgery.

[0048] For details, see Figure 8 The water jet AR display method based on ultrasonic images provided in this application comprises the following steps:

[0049] Step S1: Acquire a template image of a surgical instrument including a water jet cutter head, mark a plurality of first key feature points in the template image, and save the marked template image in a computer storage unit.

[0050] As mentioned above, the surgical instrument including a water jet cutter head includes at least a water jet cutter head, and preferably, also includes a sheath and / or an endoscope. During use, the sheath and / or the endoscope can be assembled into one with the water jet cutter head to facilitate insertion into the target tissue area, and after insertion, at least the endoscope and the water jet cutter head can move independently. Therefore, the template image of the surgical instrument including a water jet cutter head should at least include: a template image of the whole surgical instrument assembled with the sheath, the water jet cutter head, and the endoscope, a template image of the water jet cutter head, a template image of the endoscope, and a template image of the sheath. That is, the corresponding template images are obtained for the whole instrument and each component.

[0051] Taking the acquisition of the template image of the water jet cutter head as an example, first establish a three-dimensional model of the water jet cutter head, use a three-dimensional scanning device to perform a three-dimensional scan on the water jet cutter head, obtain its detailed geometric shape, ensure the high accuracy and integrity of the scan data for subsequent processing; use three-dimensional modeling software to process the scan data, remove noise and redundant data, and ensure the smoothness and accuracy of the model; perform detail corrections to ensure that the model is consistent with the actual water jet cutter head; and convert the processed scan data into a standard three-dimensional image. In some embodiments, the three-dimensional image can be directly used as a template image; in other embodiments, the template image of the water jet cutter head is two-dimensional, and the three-dimensional image is projected onto a two-dimensional plane to generate a two-dimensional image with multiple perspectives. Preferably, the three-dimensional scanning selects a structured light scanning method, which can quickly capture high-resolution detail images at a relatively low cost. It should be noted that the methods for obtaining the overall surgical instrument template image, the endoscope template image, and the sheath template image are similar to this, and will not be repeated here.

[0052] After obtaining a template image of a surgical instrument including a water jet cutter head, it is necessary to add key feature point marks to the template image. This can be achieved by a user performing an operation in a user interaction interface, or by marking based on the recognition results of the feature points through a software program. The key feature points that need to be marked include at least the location of the cutter hole 10a.

[0053] In a preferred embodiment, the marked first key feature point also includes the outer contour feature point of the water jet tip top portion 10 and / or the sheath top portion 40 and / or the endoscope top portion 30, such as the position point or point group representing the front end of the outer contour.

[0054] In a further preferred embodiment, the marked first key feature point also includes a plurality of feature points capable of defining the outer contour line of the water jet cutter head main axis 11 and / or the sheath main axis 41 and / or the endoscope main axis 31 .

[0055] Step S2: Acquire an ultrasonic image of a surgical instrument including a water jet cutter head and identify a plurality of second key feature points in the ultrasonic image.

[0056] The surgical instrument including the water jet cutter head and the ultrasonic probe 2 are placed in their original positions, and an ultrasonic image of the surgical instrument including the water jet cutter head is acquired. As described above, the original position means that the ultrasonic probe main body axis 21 is substantially parallel to the water jet cutter head main body axis 11, and the ultrasonic probe 2 is located below the water jet cutter head 1, and the imaging element of the ultrasonic probe is arranged toward the water jet cutter head 1, so that real-time ultrasonic image information of the water jet cutter head 1 can be acquired.

[0057] Further, a plurality of second key feature points in the ultrasound image are determined, and the second key feature points correspond to the first key feature points.

[0058] The second key feature point identified in the ultrasonic image at least includes the position P0 (x0, y0) of the knife hole 10a in the ultrasonic image when the water jet cutter head 1 is located at the original position of the water jet cutter head.

[0059] When the first key feature point marked in step S1 also includes the outer contour feature points of the water jet cutter head top portion 10 and / or the sheath top portion 40 and / or the endoscope top portion 30, the outer contour feature points representing the water jet cutter head top portion 10 and / or the sheath top portion 40 and / or the endoscope top portion 30 are correspondingly identified in the ultrasound image.

[0060] When the first key feature point marked in step S1 also includes multiple feature points that can define the outer contour line of the water jet cutter head main axis 11 and / or the sheath main axis 41 and / or the endoscope main axis 31, the multiple feature points that define the outer contour line of the water jet cutter head main axis 11 and / or the sheath main axis 41 and / or the endoscope main axis 31 are correspondingly identified in the ultrasound image.

[0061] In order to accurately identify the location of the knife hole (nozzle) 10a in the ultrasonic image, the dynamic characteristics of the nozzle jet can be used to perform jet feature recognition, and / or the static characteristics of the nozzle in the ultrasonic image can be used to perform edge extraction, so as to accurately locate the knife hole 10a in the ultrasonic image, which are described in detail below.

[0062] Jet feature recognition

[0063] like Figure 4 As shown in Figure 1, the nozzle jet is reflected in the image as a rapidly fluctuating air mass. The movement of these air masses can be used to infer the position of the nozzle. The details are as follows:

[0064] The water jet is controlled to perform the water spraying action. The ejected high-pressure jet forms rapidly fluctuating air masses in the liquid immersion environment. The regular changes of these air masses based on growth and collapse are related to the Strouhal Number of the water jet. In the ultrasonic image, they appear as rapidly fluctuating white air mass areas, which are significantly distinguished from the surrounding black areas. Jet feature recognition can then be performed and used to determine the location of the nozzle.

[0065] First, the ultrasonic image is acquired after the water jet is controlled to perform the water jet action, and the acquired ultrasonic image is preprocessed to enhance the image contrast and clarity; the white air mass fluctuation area in the image is detected by the frame difference method or the background subtraction method; according to the detected white air mass fluctuation area, a first auxiliary line is generated for positioning. Specifically, the white air mass fluctuation area is identified and marked in the image, and an auxiliary line that can roughly bisect the white area is generated in the detected white air mass fluctuation area. A yellow dotted line can be added to the image. This auxiliary line should bisect the area of ​​air mass fluctuation as much as possible, and the auxiliary line straight line is usually roughly vertical in the ultrasonic image, and the knife hole is on this auxiliary line straight line.

[0066] Static edge extraction

[0067] like Figure 5 As shown, another auxiliary line where the nozzle is located is determined by edge extraction method using the static features of the water jet cutter head in the ultrasonic image.

[0068] Specifically, edge extraction is performed on the image of the water jet cutter head, and for example, a Canny edge detection algorithm or a Sobel operator may be used to enhance the contour of the water jet cutter head and perform edge extraction. Preferably, time averaging is performed on multiple images within a period of time (for example, during a micro jet) to enhance the stable contour features of the water jet cutter head.

[0069] Since the overall material of the water jet cutter head 1 is usually stainless steel with high acoustic impedance, it is reflected as a bright long straight line in the ultrasonic image. According to the detected edge contour, a straight auxiliary line in the roughly horizontal direction is fitted using methods such as Hough transform. Since the long straight line presented by the main axis 11 of the water jet cutter head on the ultrasonic image may be tilted or interfered by artifacts, the roughly horizontal straight auxiliary line fitted here should not be too long, and its width can be equivalent to or slightly longer than the width of the water jet air mass.

[0070] Furthermore, the nozzle position is precisely located by cross-analyzing the results of jet feature recognition and edge extraction. The first straight auxiliary line generated by jet feature recognition and the second straight auxiliary line fitted by static edge extraction are cross-calculated. The intersection of the two straight auxiliary lines is the position of the nozzle, and the position coordinates of the nozzle in the ultrasound image can be determined.

[0071] Through the above steps, the dynamic jet feature recognition and static image analysis technology are combined to effectively identify and track the position coordinates of the nozzle (i.e., the water jet hole 10a) in the ultrasound image. When the water jet cutter head is at the original position P0, the pixel position coordinates of the corresponding hole feature point in the ultrasound image are imageP0 (x0, y0). By accurately locating the nozzle position in the real-time ultrasound image, a reliable basis is provided for AR imaging.

[0072] Furthermore, the position coordinates of other second key feature points in the ultrasound image are obtained by the following steps:

[0073] In one embodiment, the second key feature point is determined based on movement information of the water jet cutter head.

[0074] The water jet adapter 110 is used to drive the water jet cutter head 1 to translate a predetermined distance S from the original position along the direction of the water jet cutter head main body axis 11 to reach the second position P2.

[0075] The predetermined distance S can be determined by the working parameters of the water jet drive mechanism or a position sensor or other physical distance detection means. For example, the moving distance can be detected by setting a grating scale, a magnetic scale, a piezoelectric couple, etc., or the moving distance can be confirmed based on the distance scale parameter change information on the moving path where the water jet cutter head is located.

[0076] At the second position P2, the water jet is controlled to perform the water spraying action again, and similar steps as mentioned above are adopted to use the dynamic characteristics of the nozzle jet to perform jet feature recognition and / or use the static characteristics of the nozzle in the ultrasonic image to perform edge extraction, and identify the position coordinates imageP2 (x2, y2) of the second position P2 where the knife hole is located when the water jet performs the water spraying action again.

[0077] After determining the original position coordinates imageP0 (x0, y0) and the second position coordinates imageP2 (x2, y2) in the ultrasound image, a connecting line is established between the two position points to determine the extension direction or inclination angle of the water jet cutter head main axis 11 on the ultrasound image for subsequent AR display.

[0078] It should be noted that the water jet cutter head may not be limited to moving once, but may also be moved multiple times to identify the position coordinates of the water jet nozzle in the ultrasound image each time, forming multiple second key feature points. When collecting multiple second key feature points, although it may mean a decrease in calculation speed, the calculation accuracy of the system will be improved. And because the channel established by the sheath is fixed and this step is performed before the operation, multiple second key feature points will provide more accurate calibration of the moving direction of the water jet, which makes the lost efficiency worthwhile.

[0079] In an alternative embodiment, the second key feature point is determined based on movement information of the endoscope.

[0080] Since the nozzle is a part of the water jet head 10 in the original position, and both the nozzle and the endoscope 3 are placed in the retractable main body shaft 51 of the surgical instrument and the physical distance between the two is determined by the assembly relationship, the original position P1 of the lens of the endoscope 3 can be determined based on the predetermined assembly relationship and the original position P0 of the nozzle, and the original position imageP1(x1,y1) of the lens of the endoscope 3 in the ultrasound image can be determined.

[0081] Next, the endoscope 3 is controlled to retract along the telescopic channel, and the endoscope retraction distance L is detected by the endoscope driver (not shown) or position sensor (not shown), and the position P3 (x3, y3) of the endoscope lens when the endoscope stops moving can be determined in combination with the retraction movement direction of the endoscope. Among them, the difference image can be obtained by performing pixel-level subtraction operation on the two ultrasonic images before and after the endoscope moves, and the position imgaeP3 (x3, y3) where the endoscope stops moving in the ultrasonic image can be determined based on the difference image. In the ultrasonic image, based on the line connecting imageP1 (x1, y1) and imageP3 (x3, y3), the moving direction of the endoscope can be determined. This direction also represents the extension direction of the main axis of the endoscope, and is parallel to the extension direction of the main axis of the water jet cutter head. The advantage of this method is that only one water spraying action needs to be performed, and the nozzle position can be determined without moving the water jet cutter head.

[0082] The endoscope may also not be limited to moving once, but may be moved multiple times, and the position coordinates of the endoscope in the ultrasound image each time are identified to form multiple second key feature points. Similar to the above, when collecting multiple second key feature points of the endoscope, although it may mean a decrease in the calculation speed, the calculation accuracy of the system will be improved. And because the channel established by the sheath is fixed and this step is performed before the operation, multiple second key feature points will provide more accurate calibration of the moving direction of the water jet, which makes the lost efficiency worthwhile.

[0083] In an alternative embodiment, the second key feature point may also be determined based on fitting of the tube wall contour of the front end of the surgical instrument including the water jet cutting head.

[0084] Still assume that the position of the nozzle in the initial state is the original position P0, and the coordinates on the ultrasound image are imageP0 (x0, y0); further, extract the tube wall contour fitting to determine the direction of the water jet cutter head main axis 11 in the ultrasound image. Usually, the water jet cutter head main axis 11 is made of stainless steel or similar metals, and may show certain distortion / artifacts in the ultrasound image as a whole, such as Figure 5As shown. When the distortion / artifact in the image is small, the image of the main axis 11 of the water jet cutter head can be directly edge extracted and fitted, and the fitting line is considered to be the actual direction of the water jet cutter head in the human body. When the distortion / artifact is large, this method will be biased. At this time, a reliability judgment threshold can be established as a criterion for judging whether the fitting result is acceptable. Specifically, the distance from each point to the fitting line is calculated. For each fitting point, its distance to all fitting lines is calculated to determine whether it is a point with a large error; according to the preset distance threshold (for example, 10 pixels), the points with large errors are removed. This can reduce the impact of errors caused by distortion and artifacts on the fitting results.

[0085] Further, the least squares method is used to fit the straight line; and the parameters (slope and intercept) of the fitted straight line, as well as the goodness of fit (e.g., R² value) are calculated. The goodness of fit is used to measure the degree of fit between the fitted straight line and the data points. After removing the gross errors, all the fitting points are used to calculate their average error to the fitted straight line. The smaller the average error, the more accurate the fitted straight line. In the preferred example, the R² value is calculated to measure the quality of the fitted straight line, ranging from 0 to 1. The closer the value is to 1, the better the fit.

[0086] Furthermore, a custom threshold (for example, R² = 0.95) is set as a criterion for judging whether the fitting result is acceptable; and the fitting result is judged according to the calculated R² value. If the R² value is greater than or equal to the set threshold, the fitting result is considered reliable, and the fitting straight line is accepted as the direction of the main axis of the water jet cutter head in the ultrasound image; otherwise, the fitting result is considered unreliable and needs to be readjusted or replaced with other methods for direction positioning.

[0087] In some embodiments, other marking points identifiable in the ultrasonic image may be added to the water jet cutter head 1. For example, marking points may be formed on the water jet cutter head main shaft 11 using a high acoustic impedance material (such as a metal material or a special coating), and small metal particles may be embedded in the surface of the water jet cutter head main shaft 11. Alternatively, grooves may be engraved on the outer surface of the water jet cutter head main shaft 11 using a laser engraving machine, or convex points may be added, so as to form obvious reflection points in the ultrasonic image.

[0088] In an alternative embodiment, similar to the above-mentioned scheme, the wall contour of the sheath or endoscope can also be extracted for fitting, and the wall contour and / or extension direction of the water jet cutter head can be determined based on the physical position relationship between the sheath or endoscope and the water jet cutter head based on assembly.

[0089] Through the above detailed steps, the tube wall contour in the endoscopic ultrasound image can be extracted and the direction of the main axis of the water jet cutter head can be fitted. The key steps include image preprocessing, edge extraction, straight line fitting, removal of points with large distortion / artifacts, calculation of fitting errors, and judgment of the reliability of the fitting results. Combining these methods, the direction of the water jet cutter head can be accurately determined in the ultrasound image, providing effective navigation and reference for surgery.

[0090] Step S5: Based on the position coordinates of the plurality of first key feature points in the template image and the position coordinates of the plurality of second key feature points in the ultrasound image, the water jet template image is superimposed on the ultrasound image for display to form the water jet AR display image based on the ultrasound image, such as Figure 6 shown.

[0091] As mentioned above, a plurality of corresponding key feature points are selected or determined in both the water jet template image and the ultrasonic image, and these feature points at least include the knife hole feature point.

[0092] Furthermore, according to the selected ultrasound probe, the pixel / mm ratio of the ultrasound imaging can be determined, and the distance in the image corresponding to the actual length of the water jet template can be determined, thereby determining the display ratio of the water jet template.

[0093] First, determine the pixel-to-millimeter ratio of the ultrasound image. Specifically, use an object of known size for calibration, place the object under the ultrasound imaging device for scanning, and obtain its image; measure the length of the object in the ultrasound image (in pixels); divide the actual length of the known object (in millimeters) by the measured pixel length to obtain the pixel-to-millimeter ratio D.

[0094] Assume that a calibration object with a length of 50 mm is used for calibration: place the calibration object under the ultrasonic imaging device, scan and acquire an image; measure the length of the calibration object in the image, assuming that the measured length is 250 pixels; calculate the pixel-to-millimeters ratio: pixel-to-millimeters ratio = 250 pixels / 50 mm = 5 pixels / mm.

[0095] Furthermore, the display ratio of the water jet template in the image is determined. Specifically, the actual length L (in millimeters) of the water jet is measured; the actual length of the water jet is multiplied by the pixel-millimeter ratio to obtain the pixel length Lpixel of the water jet head in the ultrasound image, that is, Lpixel=L×D; according to the calculated pixel length Lpixel, the display ratio of the template image of the surgical instrument including the water jet head is adjusted to be consistent with the ratio in the ultrasound image.

[0096] Assuming that the actual length L of the water jet cutter head is 150 mm, the pixel length Lpixel of the water jet cutter head in the ultrasound image is calculated using the calculated pixel-to-millimeter ratio D: the pixel length Lpixel of the water jet cutter head in the ultrasound image = 150 mm × 5 pixels / mm = 750 pixels. Next, adjust the display ratio of the template image and scale the template image to a length of 750 pixels to make it consistent with the ratio in the ultrasound image; in an alternative embodiment, the scaling can also be implemented using image processing software such as OpenCV programming.

[0097] After the length of the water jet cutter head is determined, since the ratio of the physical length and physical width of the water jet cutter head can be determined in advance by measuring the actual object, it can also be determined by marking the contour line in the two-dimensional image. According to the pre-agreed length-to-width ratio, the width of the water jet cutter head can be displayed proportionally.

[0098] Furthermore, in a preferred embodiment, when acquiring an ultrasonic image of a surgical instrument including a water jet cutter head and identifying multiple second key feature points in the ultrasonic image, the endoscope or the water jet cutter head is moved multiple times. As described above, multiple movements can advantageously provide more accurate calibration. This calibration is usually necessary because: considering that the long straight line of the water jet blade may be deformed in the ultrasound image, and the endoscope and water jet head may appear as extended dotted lines due to artifacts, the error of only identifying the position of the water jet blade and the endoscope lens through the ultrasound image as the specific position of the water jet is relatively large. At the same time, the visualization of the water jet in the ultrasound image requires accurate tracking of the position of the knife hole and the endoscope lens on the image, and the ultrasound probe used to obtain the ultrasound sagittal image has a certain width, so that as long as the main axis of the water jet blade is within the width range corresponding to the ultrasound probe, an ultrasound sagittal image can be formed. When the axis of the main axis of the water jet blade has a certain angle with the sagittal plane, the movement of the water jet blade or the endoscope lens in the ultrasound image is the projection of the actual water jet blade or the movement distance in the sagittal plane. At this time, the knife hole movement distance of the visualized water jet template image in the sagittal plane needs to be calibrated to achieve accurate tracking of the knife hole and the endoscope lens position in the visualized image.

[0099] Specifically, a proportional correction may be provided based on the distances over which the endoscope or water jet cutter head is moved multiple times and the ultrasound image position.

[0100] Taking the movement of the endoscope lens as an example, assuming that when the endoscope is at the first position E0, the lens position in the ultrasonic image is imageE0 (x0, y0), and the endoscope lens is moved from the first position E0 to the second position E1 by a distance S1 along the extension direction of the endoscope body axis, and the position of the endoscope lens at this time is identified in the ultrasonic image as imageE1 (x1, y1); the endoscope lens is continued to be moved by a distance S2 along the extension direction of the endoscope body axis to the third position E2, and the position of the endoscope lens at this time is identified as imageE2 (x2, y2) in the ultrasonic image.

[0101] Considering that in the sagittal image, the main axis of the endoscope extends mainly in the x direction, the calculation method of the calibration coefficient k in the x direction is as follows:

[0102] worldE0 = T imageE0

[0103] worldE1 = T imageE1

[0104] worldE2 = T imageE2

[0105] k=mean(S2 / sqrt((worldE2(x)- worldE1(x))^2-(worldE2(y)- worldE1(y))^2), S1 / sqrt((worldE1(x)- worldE0(x))^2-(worldE1(y)-worldE0(y))^2));

[0106] Wherein, T is the transformation matrix between the ultrasound image coordinate system obtained by the ultrasound probe and the physical coordinate system, which can be obtained by calibration after the ultrasound probe is determined. worldE0 represents the physical coordinate of the E0 position, worldE1 represents the physical coordinate of the E1 position, and worldE2 represents the physical coordinate of the E2 position. mean is the mean function, and sqrt is the square root function.

[0107] The distances S1 and S2 can be determined by the working parameters of the endoscope drive mechanism or a position sensor or other physical distance detection means. For example, the moving distance can be detected by setting a grating scale, a magnetic scale, a piezoelectric couple, etc., or the moving distance can be confirmed based on the distance scale parameter change information on the moving path of the endoscope.

[0108] The above is only an example. It should be noted that the calibration coefficient can also be obtained by moving the water jet cutter head multiple times and performing similar operations.

[0109] After the calibration coefficient k is determined, in the subsequent AR display, the original position of the water jet hole imageP0 (x0, y0) is used as the center, and the length in the x direction is multiplied by the calibration coefficient and displayed.

[0110] Through the aforementioned steps, the generated ultrasound-based AR image of the surgical instrument including the water jet can provide more comprehensive and intuitive display information during the operation, for example, see Figure 7 On the augmented reality ultrasound image, the actual movement position and working status of the water jet can be simulated and displayed, and the position of the nozzle can be simulated and displayed.

[0111] According to an embodiment of the present application, the present application also provides a water jet AR display device based on ultrasound images, which is used to implement the aforementioned water jet AR display method based on ultrasound images. The device includes: an ultrasound imaging device for obtaining real-time ultrasound images of water jets; and an AR display device, which can be a head-mounted display or augmented reality glasses, or a conventional display screen. Among them, the water jet template can be preset data, or it can be newly created or updated through a visible light sensor and a processor including functions such as alignment. In addition, although the present application uses an example of real-time monitoring of the surgical process of a water jet (usually inserted through the urethra) with the help of a TRUS probe (transrectal ultrasound probe), it should be understood that the present application is not limited to the above-mentioned surgical scenarios.

[0112] See also Fig. 9 , the present application also proposes a water jet AR display device based on ultrasonic images, comprising the following modules:

[0113] A template image acquisition module for a water jet cutter head, used to acquire a template image of a water jet cutter head and mark a first key feature point in the template image, wherein the key feature point includes a cutter hole;

[0114] An ultrasonic image acquisition module of a water jet cutter head, used to acquire an ultrasonic image of the water jet cutter head and identify a second key feature point of the water jet cutter head in the ultrasonic image, wherein the second key feature point corresponds to the first key feature point;

[0115] The AR image generation module is used to form the water jet AR display image based on the ultrasound image based on the position coordinates of the first key feature point in the template image and the position coordinates of the second key feature point in the ultrasound image.

[0116] See also Fig.10 The present application also proposes a water jet AR display system based on ultrasonic images, comprising: a surgical instrument suitable for being inserted into a human body to perform surgery, an ultrasonic imaging device, a processor, a memory and a display device, wherein the surgical instrument comprises a water jet cutter head and an endoscope, wherein the main axis of the water jet cutter head and the main axis of the endoscope are substantially parallel, the ultrasonic imaging device is placed below the surgical instrument, the processor is used to execute a computer program stored in the memory and executable on the processor, wherein the computer program implements the aforementioned method when executed by the processor, and the display device is used to display the water jet AR display image based on the ultrasonic image.

[0117] In summary, this application can provide effective visual assistance for medical operations through precise image processing and real-time tracking technology. By providing intuitive visual assistance, it can significantly improve the doctor's perception ability and operation accuracy during the operation, reduce surgical risks, improve medical effects, and comprehensively improve the accuracy and safety of the operation.

[0118] It should be noted that the above modules are divided according to module functions, and can be configured and implemented as needed in terms of physical location, hardware and software / firmware architecture.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in conjunction with software and hardware. In a typical configuration, the equipment of the present application includes one or more processors (one in CPU, FGAP, MUC), input / output user interface, network interface, and memory.

[0120] Therefore, the present application also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method described in any one of the embodiments of the first aspect of the present application is implemented.

[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] In addition, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0123] Therefore, the present application also proposes a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any embodiment of the present application are implemented. For example, the memory of the present application may include a non-permanent memory in a computer-readable medium, a random access memory (RAM) and / or a non-volatile memory, such as a read-only memory (ROM) or a flash RAM.

[0124] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0125] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A water jet AR display method based on ultrasound images in a non-surgical scenario, comprising the following steps: Acquire a template image of a device including a water jet cutter head, and mark a plurality of first key feature points in the template image, wherein the first key feature points include a cutter hole feature point; Acquire an ultrasonic image of an instrument including a water jet cutter head and identify a plurality of second key feature points in the ultrasonic image, wherein the second key feature points correspond to the first key feature points, wherein: Using the dynamic features of the nozzle jet to perform jet feature recognition, and using the static features of the nozzle in the ultrasonic image to perform edge extraction to determine the knife hole position; The water jet AR display image based on the ultrasound image is formed based on the position coordinates of the multiple first key feature points in the template image and the position coordinates of the multiple second key feature points in the ultrasound image.

2. The water jet AR display method according to claim 1, characterized in that: The step of obtaining an ultrasonic image of the instrument including the water jet cutter head and identifying a plurality of second key feature points in the ultrasonic image comprises: Controlling the water jet to perform a water spraying action, the sprayed high-pressure jet forms a rapidly fluctuating air mass in the liquid immersion environment, and performing jet feature recognition on the contour features of the air mass according to the detected air mass fluctuation area in the ultrasonic image to generate a first auxiliary line; The static features of the water jet cutter head in the ultrasonic image are used to extract the edge and generate the second auxiliary line; The accurate position of the knife hole in the ultrasonic image is determined according to the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

3. A water jet AR display device based on ultrasonic images, comprising the following modules: A template image acquisition module, used to acquire a template image of a device including a water jet cutter head, and mark a plurality of first key feature points in the template image, wherein the first key feature points include a cutter hole feature point; An ultrasonic image acquisition module is used to acquire an ultrasonic image of an instrument including a water jet cutter head, and identify a plurality of second key feature points in the ultrasonic image, wherein the second key feature points correspond to the first key feature points, wherein: Using the dynamic features of the nozzle jet to perform jet feature recognition, and using the static features of the nozzle in the ultrasonic image to perform edge extraction to determine the knife hole position; The AR image generation module is used to form the water jet AR display image based on the ultrasound image based on the position coordinates of the multiple first key feature points in the template image and the position coordinates of the multiple second key feature points in the ultrasound image.

4. The water jet AR display device according to claim 3, characterized in that: The ultrasonic image acquisition module determines the knife hole position by the following steps: Controlling the water jet to perform a water spraying action, the sprayed high-pressure jet forms a rapidly fluctuating air mass in the liquid immersion environment, and performing jet feature recognition on the contour features of the air mass according to the detected air mass fluctuation area in the ultrasonic image to generate a first auxiliary line; The static features of the water jet cutter head in the ultrasonic image are used to extract the edge and generate the second auxiliary line; The accurate position of the knife hole in the ultrasonic image is determined according to the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

5. The water jet AR display device according to claim 3, characterized in that: The ultrasound image acquisition module is also used to perform the following steps: The water jet cutter head is translated for a certain distance along the main axis of the water jet cutter head and a water spraying action is performed to obtain a corresponding position point; and / or, an initial position point of an endoscope contained in the instrument is obtained, and the endoscope is driven to retract to a terminal position to obtain a stop position point of the endoscope; and / or, a plurality of position points on the tube wall contour of the main axis of the water jet cutter head are obtained.

6. The water jet AR display device according to claim 5, characterized in that: The ultrasonic image acquisition module is also used to perform the following steps: determine whether the fitting result of the tube wall contour is reliable according to a reliability judgment threshold; when the judgment result is considered unreliable, control the water jet cutter head to translate a certain distance along the water jet cutter head main body axis and perform a water spraying action to obtain a corresponding position point; and / or, obtain an initial position point of the endoscope contained in the instrument, and drive the endoscope to retract to the end position to obtain a stop position point of the endoscope.

7. A water jet AR display system based on ultrasonic images, comprising: A surgical instrument suitable for being inserted into a human body to perform surgery, an ultrasonic imaging device, a processor, a memory, and a display device, wherein the surgical instrument comprises a water jet cutter head and an endoscope, wherein the main axis of the water jet cutter head and the main axis of the endoscope are substantially parallel, and the ultrasonic imaging device is placed below the surgical instrument, and the processor is used to execute a computer program stored in the memory and executable on the processor, wherein the computer program implements the following steps when executed by the processor: Acquire a template image of a device including a water jet cutter head, and mark a plurality of first key feature points in the template image, wherein the first key feature points include a cutter hole feature point; Acquire an ultrasonic image of a device including a water jet cutter head and identify a plurality of second key feature points in the ultrasonic image, wherein the second key feature points correspond to the first key feature points, wherein the dynamic features of the nozzle jet are used to perform jet feature recognition, and the static features of the nozzle in the ultrasonic image are used to perform edge extraction to determine the location of the cutter hole; Based on the position coordinates of the plurality of first key feature points in the template image and the position coordinates of the plurality of second key feature points in the ultrasound image, forming the ultrasound-image-based water jet AR display image; The display device is used to display the water jet AR display image based on the ultrasonic image.

8. The water jet AR display system according to claim 7, characterized in that: The step of obtaining an ultrasonic image of the instrument including the water jet cutter head and identifying a plurality of second key feature points in the ultrasonic image comprises: Controlling the water jet to perform a water spraying action, the sprayed high-pressure jet forms a rapidly fluctuating air mass in the liquid immersion environment, and performing jet feature recognition on the contour features of the air mass according to the detected air mass fluctuation area in the ultrasonic image to generate a first auxiliary line; The static features of the water jet cutter head in the ultrasonic image are used to extract the edge and generate the second auxiliary line; The accurate position of the knife hole in the ultrasonic image is determined according to the first auxiliary line generated by the jet feature recognition and the second auxiliary line generated by the edge extraction.

9. The water jet AR display system according to claim 7, characterized in that: The step of acquiring an ultrasonic image of an instrument including a water jet cutter head and identifying a plurality of second key feature points in the ultrasonic image further includes: translating the water jet cutter head for a distance along the main axis of the water jet cutter head and performing a water spraying action to acquire corresponding position points; and / or, acquiring an initial position point of an endoscope included in the instrument, and driving the endoscope to retract to a terminal position to acquire a stop position point of the endoscope; and / or, acquiring a plurality of position points on the tube wall contour of the main axis of the water jet cutter head.

10. The water jet AR display system according to claim 9, characterized in that: The step of acquiring an ultrasonic image of an instrument including a water jet cutter head and identifying a plurality of second key feature points in the ultrasonic image further includes: determining whether a fitting result of the tube wall contour is reliable according to a reliability determination threshold value; when the determination result is considered unreliable, controlling the water jet cutter head to translate a certain distance along the main axis of the water jet cutter head and performing a water spraying action to acquire a corresponding position point; and / or acquiring an initial position point of an endoscope included in the instrument, and driving the endoscope to retract to a termination position to acquire a stop position point of the endoscope.

Images