Image annotation method and system

Abstract

The specification discloses an image labeling method and system. In the embodiment of the specification, the current poses of a medical instrument and a patient are determined according to a positioning marker arranged on the medical instrument and a positioning marker arranged on the patient. Then, the relative poses between a medical instrument model and a body structure image of the patient are determined according to the current poses of the medical instrument and the patient. Finally, the body structure region on which the current surgical operation can be performed is labeled from the body structure image according to the relative poses between the medical instrument model and the body structure image of the patient, and the labeled image is displayed. In the process of performing the surgical operation by the user, the user can take the labeled body structure region as a reference to identify whether the local body structure viewed through the medical instrument is the body structure that needs to be operated.

Description

Technical Field

[0001] This manual relates to the field of medical devices, and in particular to an image annotation method and system. Background Technology

[0002] In the medical field, various endoscopic surgeries, such as spinal endoscopy, percutaneous endoscopic discectomy, and arthroscopy, are increasingly used in clinical practice. Taking percutaneous endoscopic discectomy as an example, it is a procedure in which a percutaneous endoscopic discectomy is performed by inserting a percutaneous endoscope into the intervertebral foramen from the side or posterolateral aspect of the patient's body.

[0003] During percutaneous endoscopic discectomy (PED), due to the small surgical incision, doctors cannot see the entire spinal structure inside the patient's body; they can only see localized spinal structures within the percutaneous endoscopic view. Therefore, movement of the percutaneous endoscopic endoscope during surgery can alter the localized spinal structures seen by the doctor, making it difficult to identify whether the seen structures are the lesions requiring surgery. This can increase the difficulty of the surgery or even lead to surgical failure. Therefore, identifying the specific spinal structures requiring surgery within the percutaneous endoscopic view is a problem that urgently needs to be solved. Summary of the Invention

[0004] This specification provides an image annotation method and system to at least partially solve the problems existing in the prior art.

[0005] The embodiments in this specification adopt the following technical solutions:

[0006] This specification provides an image annotation system, the system comprising: a medical device equipped with positioning markers, a positioning and tracking device, and a processor;

[0007] The positioning and tracking device captures the current first pose of the medical device and the current second pose of the patient using positioning markers located on the medical device and the patient, and reports the first pose and the second pose to the processor;

[0008] The processor receives the first pose and the second pose, and determines the relative pose between the medical device model corresponding to the medical device and the patient's body structure image based on the first pose and the second pose; and based on the relative pose, marks the body structure area in the patient's body structure image where the medical device can currently perform surgical operations, and displays the marked image.

[0009] The medical instrument provided with the positioning marker at least includes a foramen transversarium mirror, a working channel sleeve and a nucleus pulposus forceps, wherein the positioning marker is connected by a plurality of optical spheres in a relative position, and the positioning marker provided by each medical instrument is different in shape and / or size.

[0010] The present specification provides an image labeling method, which comprises:

[0011] Obtaining a body structure image of a patient, and obtaining a first pose of a medical instrument currently captured by a positioning tracking device and a second pose of the patient currently;

[0012] According to the first pose and the second pose, determining the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient;

[0013] According to the relative pose, labeling the body structure region in which the medical instrument can currently perform a surgical operation from the body structure image of the patient, and displaying the labeled image.

[0014] Optionally, according to the first pose and the second pose, determining the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient, specifically comprising:

[0015] According to the first pose of the medical instrument currently and the pose of the medical instrument at the last time, determining the pose difference of the medical instrument at the current time for the pose at the last time as a first pose difference; according to the second pose of the patient currently and the pose of the patient at the last time, determining the pose difference of the patient at the current time for the pose at the last time as a second pose difference;

[0016] According to the first pose difference and the second pose difference, determining the adjustment parameter of the medical instrument model corresponding to the medical instrument which needs to be adjusted at the current time relative to the last time, wherein the adjustment parameter includes a rotation angle of the medical instrument model;

[0017] Based on the adjustment parameter, determining the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient.

[0018] Optionally, based on the adjustment parameter, determining the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient, specifically comprising:

[0019] Judging whether the adjustment parameter is abnormal or not;

[0020] If there is no exception, based on the adjustment parameter, a relative pose between the medical instrument model corresponding to the medical instrument and the structural image of the patient's body is determined.

[0021] Optionally, based on the adjustment parameter, a relative pose between the medical instrument model corresponding to the medical instrument and the structural image of the patient's body is determined, specifically including:

[0022] Based on the adjustment parameter, the pose of the medical instrument model corresponding to the medical instrument at the last time is adjusted to obtain the current pose of the medical instrument model; and based on the second pose difference of the patient, the image position of the structural image of the patient's body at the last time is adjusted to obtain the current image position of the structural image of the patient's body.

[0023] According to the current pose of the medical instrument model and the current image position of the structural image of the patient's body, a relative pose between the medical instrument model and the structural image of the patient's body is determined.

[0024] Optionally, according to the relative pose, a body structure region in which the medical instrument can currently perform a surgical operation is marked from the structural image of the patient's body, specifically including:

[0025] According to the relative pose between the medical instrument model corresponding to the medical instrument and the structural image of the patient's body, a local body structure position of a visual channel bevel orientation involved by the medical instrument model is determined from the structural image of the patient's body.

[0026] According to the local body structure position in the structural image and the opening angle of the visual channel bevel, a body structure region in which the medical instrument can currently perform a surgical operation is marked from the structural image of the patient's body.

[0027] Optionally, the first current pose of the medical instrument includes: an orientation of a visual channel bevel involved by the medical instrument, and a position of the visual channel bevel.

[0028] The image marking device provided in the specification comprises:

[0029] The acquisition module is configured to acquire a structural image of a patient's body, and acquire a first current pose of a medical instrument captured by a positioning and tracking device and a second current pose of the patient.

[0030] The determination module is configured to determine a relative pose between a medical instrument model corresponding to the medical instrument and the structural image of the patient's body according to the first pose and the second pose.

[0031] An annotation module is configured to annotate a region of the body structure on which the medical instrument can perform the operation from the body structure image of the patient according to the relative pose, and display the annotated image.

[0032] The present specification provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the image annotation method.

[0033] The present specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image annotation method when executing the program.

[0034] The above at least one technical solution adopted by the embodiments of the present specification can achieve the following beneficial effects:

[0035] In the embodiments of the present specification, the current pose of the medical instrument and the patient is determined according to the positioning marker arranged on the medical instrument and the positioning marker arranged on the patient. Then, the relative pose between the medical instrument model and the body structure image of the patient is determined according to the current pose of the medical instrument and the patient. Finally, the region of the body structure on which the operation can be performed is annotated from the body structure image according to the relative pose between the medical instrument model and the body structure image of the patient, and the annotated image is displayed. In the process of performing the operation by the user, the real-time pose of the medical instrument in the operation process is tracked through the positioning marker, and the region of the body structure on which the operation can be performed is annotated in the body structure image of the patient in real time. In this way, the user can take the annotated region of the body structure as a reference to identify whether the local body structure viewed through the medical instrument is the body structure that needs to be operated. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are included to provide a further understanding of the present specification, constitute a part of the present specification, and the illustrative embodiments of the present specification and their description serve to explain the present specification, and do not constitute an improper limitation on the present specification. In the drawings:

[0037] Figure 1 A structural schematic diagram of the positioning marker provided for the embodiments of the present specification;

[0038] Figures 2a to 2c A structural schematic diagram of the medical instrument provided with the positioning marker for the embodiments of the present specification;

[0039] Figure 3 A flowchart of the image annotation method provided for the embodiments of the present specification;

[0040] Figure 4A schematic diagram of a local body structure labeled for an embodiment of the present specification;

[0041] Figure 5 A surgical scene diagram of a user performing a surgical operation process for an embodiment of the present specification;

[0042] Figure 6 A device structure schematic diagram of an image standard for an embodiment of the present specification;

[0043] Figure 7 A structure schematic diagram of an electronic device for an embodiment of the present specification. DETAILED DESCRIPTION

[0044] The system and method of image labeling provided by the present specification aims to track the real-time pose of a medical instrument in a surgical process by positioning a marker during the process of a patient undergoing surgery. Then, according to the real-time pose of the medical instrument, the body structure region of the patient that can be operated on by the medical instrument is labeled in real time from the body structure image of the patient. That is, the local body structure region of the patient that can be seen by the visible medical instrument is labeled in real time from the body structure image of the patient, that is, the part of the body structure of the patient that can be seen by the field of view of the visible medical instrument is labeled. Among them, the visible medical instrument at least includes: intervertebral foramen mirror, working channel sleeve, etc.

[0045] In order to make the purpose, technical scheme and advantages of the present specification clearer, the technical scheme of the present specification will be described clearly and completely in combination with the specific embodiments of the present specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present specification, not all. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present specification.

[0046] The technical scheme provided by each embodiment of the present specification will be described in detail below in combination with the drawings.

[0047] In the medical field, the present specification relates to various mirror surgeries such as endoscopic surgery, intervertebral foramen mirror surgery, arthroscopic surgery, etc. Various mirror surgeries have the characteristics of small surgical incision and small mirror field of view. In the process of various mirror surgeries, the medical instruments used by the user during surgery may be different. Among them, the medical instrument referred to in the present specification can refer to a medical instrument that enters the patient's body during surgery to perform surgical operations. The user in the present specification can refer to a surgeon and / or a surgical robot performing surgical operations.

[0048] In the embodiments of the present specification, an image labeling system is provided in the present specification, and the image labeling system at least comprises a medical instrument provided with a positioning marker, a positioning tracking device, a processor, etc. Wherein, the medical instrument and the positioning tracking device are both located in the operating room. The medical instrument can be an instrument that enters the inside of the patient's body to perform surgical operation. The positioning tracking device can be used to capture the pose of each positioning marker, and the positioning tracking device and the processor can communicate through wireless transmission, so that the positioning tracking device can report the pose of each positioning marker captured to the processor, and the processor can process the pose of each positioning marker, determine the relative pose between the medical instrument and the patient, and label the body structure region visible through the visible medical instrument from the body structure image of the patient. Wherein, the visible medical instrument at least includes a foramen transversarium mirror, a working channel sleeve, etc.

[0049] Specifically, the positioning marker can be arranged on the medical instrument and the positioning marker can be arranged on the patient's body, wherein the shape of the positioning marker on the medical instrument is different from the shape of the positioning marker on the patient's body. Then, the positioning tracking device captures the current pose of the medical instrument as a first pose through the positioning marker on the medical instrument and the positioning marker on the patient's body during the operation, and captures the current pose of the patient as a second pose. Then, the first pose and the second pose are reported to the processor through wireless transmission. Wherein, the first pose of the medical instrument can include the position of the medical instrument, the orientation of the visible channel bevel (i.e. the opening direction of the visible channel bevel), the position of the visible channel bevel, etc. The visible channel bevel refers to the channel port that can see the body structure inside the body and has a bevel shape, such as the mirror port in the foramen transversarium mirror and the sleeve port in the working channel sleeve. The second pose of the patient can include the position of the patient, the turning state, etc.

[0050] After the processor receives the first pose and the second pose, the relative pose between the medical instrument model corresponding to the medical instrument and the pre-acquired body structure image of the patient is determined according to the first pose and the second pose. Then, according to the relative pose, the body structure region in which the medical instrument can currently perform surgical operation is labeled from the body structure image of the patient. That is, the part of the body structure that can be seen through the medical instrument is labeled from the body structure image. Finally, the labeled image is displayed to the user through the image display. Wherein, the user can be the doctor performing the surgical operation and / or the surgical robot.

[0051] In the image labeling system, the positioning tracking device is a device matched with the positioning marker. When the positioning marker is an optical positioning marker, the positioning tracking device is also an optical positioning tracking device. In addition, the optical positioning method can be active and passive, which is not limited in the present specification.

[0052] In order to enable the position tracking device to capture the pose change of the position marker on each medical instrument in the surgical procedure, the position tracking device can be placed above the operating table. Specifically, the position tracking device can be hung above the operating table, or the position tracking device can be placed above the operating table through a support of a certain height.

[0053] In a surgical procedure, when there are multiple medical instruments, the shapes and / or sizes of the position markers arranged on each medical instrument are different, that is, in a surgical procedure, different medical instruments can be distinguished according to position markers of different shapes and / or sizes. Among them, the position marker is connected by a plurality of optical spheres in a relative position. That is, a position marker can be rigidly connected by a plurality of connecting rods to a plurality of optical spheres. The shape of a position marker can be determined by the shape of a plurality of optical spheres and a plurality of connecting rods, and the size of a position marker can be determined by the length of the connecting rods connecting the optical spheres. For example: the shape of the position marker connected by four optical spheres can be a parallelogram, a square, a rectangle, a trapezoid, etc. As shown in Figure 1 .

[0054] In Figure 1 , the position marker is connected by four optical spheres, and any two optical spheres are connected by a connecting rod. The shape of the position marker a is a parallelogram, the shape of the position marker b is a square, and the shape of the position marker c is a rectangle.

[0055] For each medical instrument, a position marker can be fixedly arranged on each medical instrument. Specifically, the position marker can be fixedly connected to the medical instrument through a fixing rod. That is, one end of the fixing rod is connected to the position marker, the other end of the fixing rod is connected to the medical instrument, and the connection of the two ends of the fixing rod with the position marker and the medical instrument cannot be moved relatively (i.e., rigid connection). In addition, the position of the position marker arranged on the medical instrument is not limited, as long as it does not affect the user to perform the surgical operation.

[0056] Taking the medical instruments involved in the transforaminal endoscopic surgery as an example, the medical instruments involved in the transforaminal endoscopic surgery at least include a transforaminal endoscope, a working channel sleeve, a nucleus pulposus forceps, etc. Position markers of different shapes can be arranged on each medical instrument. For example: the shape of the position marker on the transforaminal endoscope is a parallelogram, the shape of the position marker on the working channel sleeve is a square, and the shape of the position marker on the nucleus pulposus forceps is a rectangle. As shown in Figures 2a to 2c . Figure 2a is a transforaminal endoscope provided with a position marker, Figure 2b is a working channel sleeve provided with a position marker,Figure 2c The nucleus pulposus forceps provided with the positioning marker.

[0057] The opening of the intervertebral foramen mirror and the working channel sleeve into the patient's body is beveled, such as: duck-billed shape. Therefore, when the bevels of the intervertebral foramen mirror and the working channel sleeve are different, the user can see different ranges of view through the intervertebral foramen mirror and the working channel sleeve. Therefore, in this specification, the pose of the medical instrument includes: the orientation of the visual channel bevel of the medical instrument, the position of the visual channel bevel.

[0058] In addition, in addition to being rigidly connected by a plurality of optical spheres, the positioning marker can also be provided on the outer wall of the medical instrument. The positioning marker of the medical instrument is composed of optical spheres attached to the outer wall of the medical instrument.

[0059] Based on the above description of the structure of the positioning marker and the connection relationship between the medical instrument and the positioning marker, the flowchart of the image labeling method provided by the embodiment of the present specification is as shown in Figure 3 .

[0060] The image labeling method applied to the processor in the image labeling system comprises:

[0061] S300: Obtain the body structure image of the patient, and obtain the first pose of the medical instrument and the second pose of the patient currently captured by the positioning and tracking device.

[0062] In the embodiment of the present specification, the image labeling method shown in Figure 3 is described taking the passive optical positioning and intervertebral foramen mirror surgery as an example.

[0063] In the process of the user performing surgery on the patient, the positioning and tracking device emits infrared light to the surgical area of the patient, and the positioning markers on the patient's body and the medical instrument reflect the infrared light. The positioning and tracking device determines (captures) the pose of the positioning markers on the medical instrument and the patient's body according to the reflected infrared light. Since the positioning markers on the medical instrument are relatively static with the medical instrument, the change in the pose of the positioning markers on the medical instrument is equivalent to the change in the pose of the medical instrument. Similarly, the change in the pose of the positioning markers on the patient's body is equivalent to the change in the pose of the patient's body. For the current time, the positioning and tracking device reports the first pose of the medical instrument and the second pose of the patient captured at the current time to the processor.

[0064] The processor receives (acquires) a first pose of the medical instrument and a second pose of the patient captured by the positioning and tracking device. At the same time, the processor can acquire a body structure image of the patient. The body structure image can be a three-dimensional image of the patient obtained by CT, nuclear magnetic resonance or x-ray before the operation. In addition, the image position change of the body structure image is associated with the pose change of the patient on the operating table, and the image position change of the body structure image can refer to rotation, inversion, translation in the image space of the body structure image.

[0065] S302: According to the first pose and the second pose, the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient is determined.

[0066] In the embodiment of the present application, before the operation, the medical instrument used in the operation can be modeled by modeling software according to the shape and size of the medical instrument, to obtain a medical instrument model corresponding to the medical instrument used in the operation. The pose of the medical instrument model is associated with the pose of the medical instrument, that is, the pose of the medical instrument model changes with the change of the pose of the medical instrument.

[0067] During the operation, after acquiring the first pose of the medical instrument and the second pose of the patient, the processor can determine the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient according to the first pose and the second pose. The relative pose can represent the relative position and relative attitude between the medical instrument model and the body structure in the body structure image of the patient.

[0068] When determining the relative pose between the medical instrument model and the body structure image of the patient, the second pose of the patient can be taken as a reference pose, and the pose difference of the first pose of the medical instrument relative to the second pose of the patient can be calculated, that is, the relative pose.

[0069] In addition, to improve the accuracy of the relative pose between the medical instrument model and the body structure image of the patient, the accuracy of the first pose of the medical instrument and the second pose of the patient captured by the positioning and tracking device at the current time can be verified.

[0070] In addition, since the patient is mostly in a stationary state during the operation, in general, the current first pose of the patient is not much different from the last pose of the patient, and it can be considered that the second pose of the patient is relatively accurate. However, the medical instrument is mostly in a moving state during the operation, so the probability of the inaccuracy of the captured pose of the medical instrument is relatively large. Therefore, it is necessary to verify the accuracy of the first pose of the medical instrument captured by the positioning and tracking device at the current time.

[0071] The verification of the first pose of the medical instrument:

[0072] According to the first pose of the medical instrument at the current time and the pose of the medical instrument at the previous time, the pose difference of the medical instrument at the current time with respect to the pose at the previous time is determined as the first pose difference. At the same time, according to the second pose of the patient at the current time and the pose of the patient at the previous time, the pose difference of the patient at the current time with respect to the pose at the previous time is determined as the second pose difference. Then, according to the first pose difference and the second pose difference, the adjustment parameter of the medical instrument model corresponding to the medical instrument that needs to be adjusted at the current time with respect to the pose at the previous time is determined. Finally, according to the preset threshold, it is judged whether the adjustment parameter is abnormal. If the adjustment parameter is greater than the preset threshold, it is determined that the adjustment parameter is abnormal, and conversely, if the adjustment parameter is not greater than the preset threshold, it is determined that the adjustment parameter is not abnormal.

[0073] When the adjustment parameter is abnormal, it is considered that the current first pose of the medical instrument captured by the positioning and tracking device is inaccurate. When the adjustment parameter is not abnormal, it is considered that the current first pose of the medical instrument captured by the positioning and tracking device is accurate.

[0074] Among them, when the adjustment parameter of the medical instrument model corresponding to the medical instrument with respect to the pose at the previous time is determined according to the first pose difference and the second pose difference, in the surgical process, the reason why the pose of the medical instrument changes from the pose at the previous time to the first pose at the current time may be: first, the patient's body moves and causes the pose of the medical instrument to change. Second, the patient's body does not move, and the medical instrument changes its pose according to the internal changes of the patient's body caused by performing surgical operations. Therefore, in order to exclude the influence of the movement of the patient's body on the pose of the medical instrument, when verifying the first pose, the first pose difference needs to be adjusted according to the second pose difference of the patient to obtain the adjusted first pose difference. Then, according to the adjusted first pose difference, the adjustment parameter of the medical instrument model that needs to be adjusted at the current time with respect to the pose at the previous time is determined. In fact, the adjustment parameter to be adjusted for the medical instrument model to convert from the pose at the previous time to the first pose at the current time is obtained by quantifying the adjusted first pose difference. The adjustment parameter can include the rotation angle of the medical instrument model, the movement distance of the medical instrument model, etc. If the medical instrument has a visible channel bevel, the rotation angle of the medical instrument model corresponding to the medical instrument is equivalent to the rotation angle of the visible channel bevel of the medical instrument model.

[0075] For example, taking the first pose as the rotation angle as an example, the first pose difference of the medical instrument is the rotation angle difference between the current time and the last time of the medical instrument. If the rotation angle difference is 40 degrees, and the rotation angle difference caused by the second pose difference of the patient is 10 degrees, then after excluding the rotation difference caused by the movement of the patient, the rotation angle difference caused by the execution of the surgical operation of the medical instrument is 30 degrees, which is the adjustment parameter required by the medical instrument model.

[0076] After verifying the first pose of the medical instrument, if the adjustment parameter is abnormal, an error prompt information is fed back to the user. The error prompt information is used to prompt the user that the first pose of the medical instrument is incorrect, and prompt the user to check whether the positioning and tracking device is faulty or prompt the user to recalibrate the positioning and tracking device, so as to reposition and track the medical instrument by the positioning and tracking device. If the adjustment parameter is not abnormal, the relative pose between the medical instrument corresponding to the medical instrument model and the structural image of the patient's body can be determined according to the adjustment parameter of the medical instrument model. Specifically, the pose of the medical instrument model at the last time can be updated according to the adjustment parameter of the medical instrument model, and the relative pose between the medical instrument model at the current time and the structural image of the patient's body can be determined again.

[0077] When updating the pose of the medical instrument model at the last time according to the adjustment parameter of the medical instrument model, the pose of the medical instrument model at the last time can be adjusted based on the adjustment parameter of the medical instrument model to obtain the current pose of the medical instrument model.

[0078] When the patient's body moves, the pose of the patient's body structure image can be updated according to the change of the patient's pose at the current time with respect to the last time. Specifically, the image position of the patient's body structure image at the last time can be adjusted according to the second pose difference of the patient to obtain the current image position of the patient's body structure image.

[0079] After updating the pose of the medical instrument model and the pose of the patient's body structure image, the relative pose between the medical instrument model and the patient's body structure image can be determined according to the current pose of the medical instrument model and the current image position of the patient's body structure image.

[0080] S304: According to the relative pose, the body structure region of the patient's body structure image on which the surgical operation can be currently performed by the medical instrument is labeled, and the labeled image is displayed.

[0081] In the embodiments of the present disclosure, after the relative pose between the medical instrument model and the patient's body structure image is determined, the body structure region in which the medical instrument can currently perform the surgical operation can be marked from the patient's body structure image according to the relative pose between the medical instrument model and the patient's body structure image, and the marked image is displayed. The body structure region in which the medical instrument can currently perform the surgical operation can refer to the part of the patient's body structure that can be seen through the visual range of the visible medical instrument at the current time.

[0082] Specifically, according to the relative pose between the medical instrument model corresponding to the medical instrument and the patient's body structure image, the local body structure position of the visual channel bevel orientation involved by the medical instrument model is determined from the patient's body structure image. Then, according to the local body structure position in the body structure image and the opening angle of the visual channel bevel, the body structure region in which the medical instrument can currently perform the surgical operation is marked from the patient's body structure image. Finally, the body structure image in which the body structure region is marked is displayed to the user through the image display.

[0083] When the body structure image in which the body structure region is marked is displayed, the marked three-dimensional image can be displayed to the user, or the marked three-dimensional image can be converted into a plurality of two-dimensional images first, and the plurality of two-dimensional images in which the body structure region is marked are displayed to the user. As shown in the following. Figure 4 Figure 4 The region in which the local body structure of the patient seen through the medical instrument is marked is circular.

[0084] After the body structure image in which the body structure region is marked is displayed to the user through the image display, in the intervertebral foramen mirror surgery, the user can perform the surgical operation for the patient according to the real video under the visual field of the intervertebral foramen mirror, the image in which the body structure region is marked, and the relative pose between the pose of the positioning marker on the nucleus pulposus forceps and the pose of the positioning marker on the intervertebral foramen mirror.

[0085] Through the above Figure 3 ​The method shown can determine the current pose of the medical instrument and the patient according to the positioning marker arranged on the medical instrument and the positioning marker arranged on the patient. Then, the relative pose between the medical instrument model and the body structure image of the patient is determined according to the current pose of the medical instrument and the patient. Finally, the body structure region on which the current surgical operation can be performed is marked from the body structure image according to the relative pose between the medical instrument model and the body structure image of the patient, and the marked image is displayed. In the process of the user performing the surgical operation, the real-time pose of the medical instrument in the surgical process is tracked through the positioning marker, and the body structure region on which the medical instrument can be operated is marked in the body structure image of the patient in real time. In this way, the user can take the marked body structure region as a reference to identify whether the local body structure viewed through the medical instrument needs to be operated.

[0086] Based on the above description of the image marking method and the description of the image marking system, Figure 3 the embodiment of the present specification provides a surgical scene graph of the user performing the surgical operation process, as shown in the description of the image marking method and the image marking system. Figure 5

[0087] In the Figure 5 , taking the performance of intervertebral foramen mirror surgery as an example, the patient lies on the operating table, that is, the front chest of the patient is downward and the back of the patient is upward. Before the operation, the body of the patient is scanned to obtain a body structure image. A positioning tracking device is placed on the upper part of the operating table through a support with a certain height. In order to track the change of the body posture of the patient, a positioning marker is fixed on the body of the patient close to the position needing to be operated, and the positioning marker on the body of the patient is relatively static with the body structure inside the body of the patient. Similarly, different positioning markers (which can be different in shape) are arranged on a plurality of medical instruments. In the process of the operation, the user uses a plurality of medical instruments provided with positioning markers to perform surgical operations for the patient, and in the process, the real image (which can be a real video) under the intervertebral foramen mirror field of view and the body structure image of the patient with the marked body structure region are displayed to the user through the image display. The body structure image with the marked body structure region can be a two-dimensional image with different visual angles such as front view and side view. The body structure region can be part of the body structure that can be viewed by different medical instruments, such as the body structure region corresponding to the working channel sleeve and the body structure region corresponding to the intervertebral foramen mirror. In addition, the opening angle of the visual channel bevel of the intervertebral foramen mirror is also marked in the Figure 5 , and the opening angle and direction of the visual channel bevel of the intervertebral foramen mirror determine the visual range of the body structure that can be viewed through the intervertebral foramen mirror.

[0088] ​The image labeling method provided in the embodiments of the present specification is based on the same idea, and the present specification also provides a corresponding device, a storage medium and an electronic device.

[0089] Figure 6 A structural schematic diagram of an image labeling device provided in the embodiments of the present specification, the device comprising:

[0090] The acquisition module 601 is configured to acquire a body structure image of a patient and acquire a first pose of a medical instrument currently captured by a positioning and tracking device and a second pose of the patient currently;

[0091] The determination module 602 is configured to determine a relative pose between a medical instrument model corresponding to the medical instrument and the body structure image of the patient according to the first pose and the second pose;

[0092] The labeling module 603 is configured to label a body structure region in which the medical instrument can currently perform a surgical operation from the body structure image of the patient according to the relative pose, and display the labeled image.

[0093] Optionally, the determination module 602 is specifically configured to determine a pose difference of the medical instrument at the current time for a previous time as a first pose difference according to the first pose of the medical instrument at the current time and a pose of the medical instrument at the previous time, determine a pose difference of the patient at the current time for the previous time as a second pose difference according to the second pose of the patient at the current time and a pose of the patient at the previous time, and determine an adjustment parameter that needs to be adjusted at the current time for the medical instrument model corresponding to the medical instrument relative to the previous time according to the first pose difference and the second pose difference, wherein the adjustment parameter comprises a rotation angle of the medical instrument model, and determine the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient based on the adjustment parameter.

[0094] Optionally, the determination module 602 is specifically configured to determine whether the adjustment parameter is abnormal, and if not, determine the relative pose between the medical instrument model corresponding to the medical instrument and the body structure image of the patient based on the adjustment parameter.

[0095] Optionally, the determining module 602 is specifically used to: adjust the pose of the medical device model corresponding to the medical device at the previous moment based on the adjustment parameters to obtain the current pose of the medical device model; adjust the image position of the patient's body structure image at the previous moment according to the second pose difference of the patient to obtain the current image position of the patient's body structure image; and determine the relative pose between the medical device model and the patient's body structure image based on the current pose of the medical device model and the current image position of the patient's body structure image.

[0096] Optionally, the annotation module 603 is specifically used to: determine the local body structure position of the visual channel oblique opening of the medical device model in the patient's body structure image based on the relative pose between the medical device model corresponding to the medical device and the patient's body structure image; and annotate the body structure area in the patient's body structure image where the medical device can currently perform surgical operations based on the local body structure position in the body structure image and the opening angle of the visual channel oblique opening.

[0097] Optionally, the current first posture of the medical device includes: the orientation of the oblique opening of the visual channel involved in the medical device and the position of the oblique opening of the visual channel.

[0098] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the above-described actions. Figure 3 The provided image annotation method.

[0099] based on Figure 3 The image annotation method shown in this specification is further provided in the embodiments. Figure 7 The diagram shows the structure of the electronic device. Figure 7 At the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 3 The image annotation method described above.

[0100] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0101] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a piece of PLD by the designer programming it by himself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually fabricating an integrated circuit chip, this programming is now mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing a program, and the original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit that implements the logical method flow can be easily obtained.

[0102] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0103] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0104] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present specification.

[0105] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0107] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0109] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0110] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0111] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0112] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0113] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present specification can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0115] The various embodiments described in this specification are described using a numbering of embodiments approach: these are each individually integrated contributions pertaining to different but related aspects of the description. Each of the various embodiments can stand on its own, and each can be combined with the subject matter of other embodiments to produce further embodiments. Where the same numbers appear in different embodiments, such numbers are used for the sake of ease of understanding only and do not imply that the embodiments in which such numbers appear are the same or similar.

[0116] The above description is embodied in the form of only a description of embodiments of the present specification, and is not intended to limit the present specification. Various changes and modifications can be made by those skilled in the art based on the present specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. An image annotation system, characterized in that, The system includes: a medical device equipped with positioning markers, a positioning and tracking device, and a processor; The positioning and tracking device captures the current first pose of the medical device and the current second pose of the patient using positioning markers located on the medical device and the patient, and reports the first pose and the second pose to the processor; The processor receives the first pose and the second pose, and determines the pose difference of the medical device in the current position relative to the previous position based on the current first pose and the previous pose of the medical device, as the first pose difference; it determines the pose difference of the patient in the current position relative to the previous position based on the current second pose and the previous pose of the patient, as the second pose difference; it determines the adjustment parameters that the medical device model corresponding to the medical device needs to be adjusted relative to the previous position based on the first pose difference and the second pose difference, wherein the adjustment parameters include: the rotation angle of the medical device model; based on the adjustment parameters, it determines the relative pose between the medical device model corresponding to the medical device and the structural image of the patient's body; and based on the relative pose, it marks the body structural area in the patient's body structural image where the medical device can currently perform surgical operations, and displays the marked image; Based on the adjustment parameters, the relative pose between the medical device model corresponding to the medical device and the patient's body structure image is determined, specifically including: Based on the adjustment parameters, the pose of the medical device model corresponding to the medical device at the previous moment is adjusted to obtain the current pose of the medical device model; according to the second pose difference of the patient, the image position of the patient's body structure image at the previous moment is adjusted to obtain the current image position of the patient's body structure image. The relative pose between the medical device model and the patient's body structure image is determined based on the current pose of the medical device model and the current image position of the patient's body structure image. Based on the relative pose, the body structure regions from the patient's body structure image are marked to indicate where the medical device can currently perform surgical operations, specifically including: Based on the relative pose between the medical device model corresponding to the medical device and the patient's body structure image, the local body structure position of the oblique opening of the visual channel involved in the medical device model is determined from the patient's body structure image. Based on the location of local body structures in the body structure image and the opening angle of the oblique opening of the visual channel, the body structure area where the medical device can currently perform surgical operations is marked from the patient's body structure image.

2. The system as described in claim 1, characterized in that, The medical devices equipped with positioning markers include at least: a percutaneous endoscopic discectomy unit, a working channel cannula, and a nucleus pulposus forceps; wherein the positioning markers are composed of a plurality of optical spheres connected in relative positions, and the shape and / or size of the positioning markers are different for each type of medical device.

3. An image annotation method, characterized in that, The method includes: Acquire images of the patient's body structure, as well as the current first pose of the medical device and the current second pose of the patient captured by the positioning and tracking device; Based on the current first pose of the medical device and the pose of the medical device at the previous moment, the pose difference of the medical device at the current moment relative to the previous moment is determined as the first pose difference; based on the current second pose of the patient and the pose of the patient at the previous moment, the pose difference of the patient at the current moment relative to the previous moment is determined as the second pose difference. Based on the first pose difference and the second pose difference, determine the adjustment parameters that the medical device model corresponding to the medical device needs to be adjusted at the current time relative to the previous moment, wherein the adjustment parameters include: the rotation angle of the medical device model; Based on the adjustment parameters, the relative pose between the medical device model corresponding to the medical device and the structural image of the patient's body is determined; Based on the relative pose, the body structure area where the medical device can currently perform surgical operations is marked from the patient's body structure image, and the marked image is displayed; Based on the adjustment parameters, the relative pose between the medical device model corresponding to the medical device and the patient's body structure image is determined, specifically including: Based on the adjustment parameters, the pose of the medical device model corresponding to the medical device at the previous moment is adjusted to obtain the current pose of the medical device model; according to the second pose difference of the patient, the image position of the patient's body structure image at the previous moment is adjusted to obtain the current image position of the patient's body structure image. The relative pose between the medical device model and the patient's body structure image is determined based on the current pose of the medical device model and the current image position of the patient's body structure image. Based on the relative pose, the body structure regions from the patient's body structure image are marked to indicate where the medical device can currently perform surgical operations, specifically including: Based on the relative pose between the medical device model corresponding to the medical device and the patient's body structure image, the local body structure position of the oblique opening of the visual channel involved in the medical device model is determined from the patient's body structure image. Based on the location of local body structures in the body structure image and the opening angle of the oblique opening of the visual channel, the body structure area where the medical device can currently perform surgical operations is marked from the patient's body structure image.

4. The method as described in claim 3, characterized in that, Based on the adjustment parameters, the relative pose between the medical device model corresponding to the medical device and the structural image of the patient's body is determined, specifically including: Determine if the adjustment parameters are abnormal; If no abnormality is found, the relative pose between the medical device model corresponding to the medical device and the structural image of the patient's body is determined based on the adjustment parameters.

5. The method as described in any one of claims 3 to 4, characterized in that, The current first position of the medical device includes: the orientation of the oblique opening of the visual channel involved in the medical device and the position of the oblique opening of the visual channel.

6. An image annotation apparatus, characterized in that, include: The acquisition module is used to acquire images of the patient's body structure, as well as the current first pose of the medical device and the current second pose of the patient captured by the positioning and tracking device. A determining module is configured to: determine the pose difference of the medical device at the current time relative to the previous time, based on the current first pose of the medical device and the pose of the medical device at the previous time, as the first pose difference; determine the pose difference of the patient at the current time relative to the previous time, based on the current second pose of the patient and the pose of the patient at the previous time, as the second pose difference; determine, based on the first pose difference and the second pose difference, the adjustment parameters that the medical device model corresponding to the medical device needs to be adjusted relative to the previous time, wherein the adjustment parameters include: the rotation angle of the medical device model; and determine the relative pose between the medical device model corresponding to the medical device and the structural image of the patient's body based on the adjustment parameters. The annotation module is used to annotate the body structure areas in the patient's body structure image that the medical device can currently perform surgical operations on based on the relative pose, and to display the annotated image; Based on the adjustment parameters, the relative pose between the medical device model corresponding to the medical device and the patient's body structure image is determined, specifically including: Based on the adjustment parameters, the pose of the medical device model corresponding to the medical device at the previous moment is adjusted to obtain the current pose of the medical device model; according to the second pose difference of the patient, the image position of the patient's body structure image at the previous moment is adjusted to obtain the current image position of the patient's body structure image. The relative pose between the medical device model and the patient's body structure image is determined based on the current pose of the medical device model and the current image position of the patient's body structure image. Based on the relative pose, the body structure regions from the patient's body structure image are marked to indicate where the medical device can currently perform surgical operations, specifically including: Based on the relative pose between the medical device model corresponding to the medical device and the patient's body structure image, the local body structure position of the oblique opening of the visual channel involved in the medical device model is determined from the patient's body structure image. Based on the location of local body structures in the body structure image and the opening angle of the oblique opening of the visual channel, the body structure area where the medical device can currently perform surgical operations is marked from the patient's body structure image.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 3-5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 3-5.

Images