Surgical operating system and method

Abstract

The present specification provides a surgical operation system and method, which are applied to the technical field of medical instruments. The system comprises a master device and a slave device; the slave device comprises a surgical instrument; the master device comprises a control device and a display device; the control device is used for acquiring operation instructions of a doctor; the display device corresponds to a first display mode; the first display mode is used for displaying a vascular access image corresponding to a three-dimensional vascular access human model; the vascular access image comprises an augmented reality display image; the vascular access image is used for describing a spatial pose of the surgical instrument in the three-dimensional vascular access human model and a relative spatial position corresponding to a blood vessel; the system accurately and intuitively describes an interventional surgery scene, determines the pose of the surgical instrument in the blood vessel in a three-dimensional manner, ensures effective execution of the surgery, and optimizes the surgical experience of the patient.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of medical device technology, and more particularly to a surgical operating system and method. Background Art

[0002] Interventional surgery is a new surgical procedure that has developed in recent years. Using imaging equipment such as angiography, CT, ultrasound, and MRI, interventional surgery measures the patient's vascular distribution. Then, using interventional devices such as needles, catheters, and guidewires, these devices are inserted into the patient's body through natural holes or tiny incisions on the human body surface. Finally, they are introduced into the affected area for minimally invasive treatment, offering advantages such as minimal trauma and rapid recovery.

[0003] Since the interventional surgical environment has a certain degree of radioactivity, in order to avoid the adverse effects of long-term exposure of doctors to the radioactive environment, it is preferred that the patient's surgical environment and the doctor's operating environment be set to isolated environments, and the surgical process be performed by remote control. At present, when performing interventional surgery, the surgical conditions are generally displayed through captured two-dimensional images. However, since interventional surgery involves surgical operations on the patient's blood vessels, and the human body's vascular pathways are relatively complex, it is difficult to intuitively grasp the current patient's condition and the specific position of the surgical instruments based solely on the captured images, which in turn affects the surgical effect and may even lead to medical accidents in severe cases. Therefore, there is an urgent need for a method that can intuitively and effectively display the surgical conditions to doctors during interventional surgery. Summary of the Invention

[0004] The purpose of the embodiments of this specification is to provide a surgical operating system and an image display method to solve the problem of how to intuitively and effectively display the surgical status of an interventional surgery to a doctor.

[0005] In order to solve the above technical problems, the embodiments of this specification propose a surgical operating system for interventional surgery, including a master device and a slave device; the slave device includes a surgical instrument; the slave device is used to obtain the position information of the surgical instrument; the master device includes a control device and a display device; the control device is used to obtain the doctor's operating instructions so that the slave device controls the movement of the surgical instrument based on the operating instructions; the display device corresponds to a first display mode; the first display mode is used to display a vascular access image corresponding to a three-dimensional vascular access human body model; the vascular access image includes an augmented reality display image; the vascular access image is used to describe the spatial posture of the surgical instrument in the three-dimensional vascular access human body model and the relative spatial position corresponding to the blood vessel.

[0006] In some embodiments, the display device also corresponds to a second display mode; the second display mode is used to display a three-dimensional navigation image; the three-dimensional navigation image includes a first-person display image with the end of the surgical instrument as the viewpoint; the three-dimensional navigation image is used to describe the vascular bifurcation condition on the forward path of the surgical instrument; the display device is also used to receive switching instructions from the doctor to convert different display modes.

[0007] In some embodiments, the three-dimensional vascular access human body model is obtained by: constructing a three-dimensional human body model based on a surgical scene image corresponding to the patient; constructing a vascular access model based on angiography images of the patient; and fusing the three-dimensional human body model and the vascular access model to obtain a three-dimensional vascular access human body model.

[0008] Based on the above embodiment, fusing the three-dimensional human body model and the vascular access model to obtain a three-dimensional vascular access human body model includes: separately registering the three-dimensional human body model and the vascular access model to obtain relative position information of the models; this includes: performing registration using camera calibration, and / or performing registration based on a transfer relationship between a spatial coordinate system and a planar coordinate system, and / or performing registration in combination with an image fusion registration algorithm; and fusing the three-dimensional human body model and the vascular access model based on the relative position information of the models to obtain the three-dimensional vascular access human body model.

[0009] In some embodiments, the position information of the surgical instrument includes a DSA image; the vascular access image is obtained by: identifying the surgical instrument in the DSA image; calibrating registration points for the DSA image and the three-dimensional vascular access human body model respectively; projecting the identified surgical instrument into the three-dimensional vascular access human body model based on the registration points; and generating a vascular access image corresponding to the surgical instrument based on the projected three-dimensional vascular access human body model.

[0010] In some embodiments, the manipulation device includes an operator; the operator is provided with a position sensor; the position sensor is used to obtain motion information of the operator when the operator is moved by manipulation of the doctor, so that the slave device controls the movement of the surgical instrument based on the motion information.

[0011] Based on the above embodiment, the slave device also includes a force sensor; the force sensor is used to obtain the movement resistance of the surgical instrument; the movement resistance includes propulsion force and rotational force; accordingly, the operator is used to feedback the resistance corresponding to the movement resistance when being manipulated by the doctor.

[0012] Based on the above embodiment, the display device is further used to display the moving speed value and / or the propulsion resistance value corresponding to the surgical instrument.

[0013] In some embodiments, the display device is further used to identify the lesion area in the vascular access image and / or the three-dimensional navigation image; correspondingly, it is also used to magnify the lesion area in the vascular access image and / or the three-dimensional navigation image.

[0014] Based on the above embodiment, the control device is used to send a prompt message to the doctor when the distance between the surgical instrument and the lesion area is less than a preset warning distance; the prompt message includes vibration information generated by the vibration unit and / or sound information generated by the speaker.

[0015] In some embodiments, the surgical instrument comprises at least one of a guidewire, a catheter, and an interventional instrument.

[0016] In some embodiments, the vascular access model further includes an organ model constructed using an organ anatomical diagram.

[0017] In some embodiments, the display device is further configured to receive interactive instructions from a physician to perform adjustment operations on the vascular access image in the first display mode; the adjustment operations include at least one of rotation, scaling, perspective, hiding, and layer overlay.

[0018] In some embodiments, the master device further includes a voice analysis module; the voice analysis module is used to obtain the doctor's voice instructions and parse the voice instructions to obtain operation instructions, so that the slave device performs corresponding operations based on the operation instructions.

[0019] This specification also proposes an image display method for interventional surgery, including: using a display device to display a vascular access image; the vascular access image includes an augmented reality display image; the vascular access image is used to describe the spatial posture of the surgical instrument in the three-dimensional vascular access human body model and the relative spatial position corresponding to the blood vessel; after receiving a display mode switching instruction input by a doctor, a three-dimensional navigation image is generated; the three-dimensional navigation image includes a first-person display image with the end of the surgical instrument as the perspective; the three-dimensional navigation image is used to describe the vascular bifurcation condition on the forward path of the surgical instrument; and the three-dimensional navigation image is displayed using a display device.

[0020] In some embodiments, the display of the vascular access image using a display device includes: obtaining an operating instruction from a doctor received by a control device; moving a surgical instrument based on the operating instruction; updating the vascular access image based on the spatial position of the surgical instrument after the movement; correspondingly, generating a three-dimensional navigation image includes: obtaining an operating instruction from a doctor received by a control device; moving the surgical instrument based on the operating instruction; determining a target spatial position of an end of the surgical instrument after the movement; and constructing a corresponding three-dimensional navigation image with the target spatial position as a viewpoint.

[0021] As can be seen from the technical solutions provided by the embodiments of this specification, the above-mentioned surgical operating system for interventional surgery allows doctors to control the slave device using the operator on the master device while also displaying surgical scene images in different display modes through the display device. In the first display mode, a vascular pathway model can be displayed to describe the spatial posture of the surgical instrument within the three-dimensional vascular pathway human body model and the relative spatial position of the corresponding blood vessels. In the second display mode, a three-dimensional navigation image can be displayed to display a first-person perspective image corresponding to the surgical instrument, thereby intuitively describing the vascular bifurcation along the surgical instrument's path. Through the above-mentioned operating system, the display mode can be switched to meet the different application requirements of the surgical scene. For example, when moving the surgical instrument in a large blood vessel, the first display mode is used to determine the overall distribution of the surgical instrument in the blood vessel, while when moving the surgical instrument in a small and complex blood vessel, the second display mode is used to intuitively and effectively control the surgical instrument's movement along the correct path. By switching the display mode, the doctor's different surgical needs can be effectively met, ensuring the effective execution of the surgery and optimizing the patient's surgical experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 This is a structural diagram of a surgical operating system according to an embodiment of this specification;

[0024] Figure 2 This is a schematic diagram of a slave device according to an embodiment of the present specification;

[0025] Figure 3 This is a schematic diagram of a master-end device according to an embodiment of this specification;

[0026] Figure 4This is a schematic diagram of a three-dimensional human body model according to an embodiment of this specification;

[0027] Figure 5 A schematic diagram of an angiography image according to an embodiment of this specification;

[0028] Figure 6 This is a schematic diagram of a vascular access model according to an embodiment of this specification;

[0029] Figure 7 This is a schematic diagram of a scene showing a vascular access image according to an embodiment of this specification;

[0030] Figure 8 This is a schematic diagram of a scenario for operating a vascular access image according to an embodiment of this specification;

[0031] Figure 9 This is a flow chart of a surgical operation method according to an embodiment of this specification. DETAILED DESCRIPTION

[0032] The following will combine the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification. In response to the above problems, the embodiments of this specification propose a surgical operating system 100. Figure 1 As shown, the surgical operating system 100 includes a master device 120 and a slave device 110 .

[0033] The slave device is a device used on the patient side, used to perform surgical operations on the patient. Figure 2 Figure 2 is a schematic diagram of the slave device. The slave device includes a surgical instrument, which is used to perform the interventional surgery. Specifically, the surgical instrument may include a guidewire, a catheter, an interventional device, etc. For vascular interventional surgery, the interventional instrument may be a vascular interventional instrument; for organ interventional surgery, the interventional instrument may be an organ interventional instrument. The specific surgical instrument can be set according to the needs of the interventional surgery in actual application and will not be detailed here.

[0034] The slave device may also be equipped with corresponding sensors for collecting parameters corresponding to the surgical instrument or the surgical environment. For example, the sensors may include a force sensor for collecting the resistance experienced by the surgical instrument during movement, or a position sensor for obtaining the position of the surgical instrument.

[0035] The main device is the device provided to doctors for operation. Figure 3FIG2 is a schematic diagram of the master device, which includes a control device and a display device.

[0036] The manipulation device can be manipulated by the doctor, thereby generating corresponding manipulation instructions for sending to the slave device, so that the slave device can control the robot itself and the corresponding surgical instruments. The manipulation device can be in the form of a manipulator, which the doctor can hold and move in any direction in three-dimensional space. Accordingly, a position sensor is provided in the manipulator, which can obtain the displacement parameters of the manipulator, such as the direction of movement, the distance of movement, and the speed of movement, and send the movement parameters to the slave device, so that the slave device can control the surgical instrument to move accordingly based on the movement parameters of the manipulator. For example, when the doctor pushes the manipulator forward, the surgical instrument is controlled to move forward.

[0037] The slave device and the master device can communicate with each other. For example, the master device can send corresponding control instructions to the slave device, and the slave device can also feed back collected information to the master device. The display device can be used to display images corresponding to the interventional surgical procedure. Specifically, the display device has a first display mode and a second display mode, each for displaying different types of images.

[0038] In the first display mode, the display device is configured to display a vascular access image corresponding to a three-dimensional vascular access human model. The three-dimensional vascular access human model is used to depict the spatial distribution of a patient's blood vessels. The vascular access image, based on this spatial distribution, depicts the spatial position of surgical instruments within the blood vessels. The vascular access image not only displays the patient's overall vascular distribution, but also illustrates the position of surgical instruments within the body, particularly within the blood vessels. Preferably, the vascular access image is an augmented reality display image, i.e., a three-dimensional image obtained by adding markers and other layers to a real-world scene image.

[0039] A 3D vascular access human body model can be a pre-built model corresponding to a patient's vascular access, used to describe the three-dimensional spatial distribution of the patient's blood vessels. In the case of vascular interventional procedures, constructing a 3D vascular access human body model can effectively represent the patient's vascular distribution. Combined with the position of surgical instruments, this model can simulate the relative position of surgical instruments relative to the patient's blood vessels.

[0040] Specifically, when constructing the three-dimensional vascular access human body model, a three-dimensional human body model corresponding to the patient can be constructed first, and then a vascular access model can be constructed based on the patient's angiography image. The three-dimensional human body model and the vascular access model can be integrated to obtain the three-dimensional vascular access human body model.

[0041] The human body three-dimensional model is a model used to characterize the patient's overall external morphology. Specifically, before the operation begins, the patient can be photographed with a camera to obtain patient images and / or surgical scene images, and the patient images and / or surgical scene images can be integrated to complete the construction of the human body three-dimensional model corresponding to the patient. The specific construction method can be, for example, to identify the captured images based on a neural network model to determine the human body three-dimensional model, or to obtain point cloud data corresponding to the patient's body and use the point cloud data to construct the human body three-dimensional model. Figure 4 Schematic diagram of a three-dimensional human body model.

[0042] The specific steps of constructing the three-dimensional human body model can be set based on the needs of actual application and will not be repeated here.

[0043] The vascular pathway model is used to describe the spatial distribution of blood vessels in the patient's body. Specifically, before the operation begins, the patient's angiography image can be obtained. Angiography images are generated by injecting contrast agent into the patient's blood vessels and then photographing the patient using corresponding angiography equipment. Figure 5 Shown is a schematic diagram of angiography image.

[0044] Specifically, the angiography images may include DSA (Digital subtraction angiography) and CTA (CT angiography). The vascular pathway model can be constructed based on the angiography images. Figure 6 Detailed steps for constructing the vascular access model can be set based on actual application requirements and will not be described in detail here.

[0045] Because interventional procedures may also target organs, in some embodiments, the vascular access model also includes an organ model constructed using organ anatomy. For example, the organ could be the patient's heart, effectively reflecting the spatial distribution of the patient's organs. By integrating the organ model, different types of interventional procedures can be effectively and in real time, ensuring a robust application environment.

[0046] Since the three-dimensional human body model cannot reflect the distribution of blood vessels in the patient's body, and the vascular access model can only reflect the spatial distribution of blood vessels, it is difficult to directly associate it with the surgical scene. Therefore, the two models are fused to obtain a three-dimensional vascular access human body model, which is then applied to the surgical process to better understand the surgical conditions.

[0047] Specifically, the human three-dimensional model and the blood vessel passageway model can be calibrated first to calibrate the key points. Then, the two models are registered based on the calibrated points to determine the model relative position information between the models. The two models are fused based on the obtained model relative position information to obtain a three-dimensional blood vessel passageway human model.

[0048] The registration process mainly determines the spatial relative position of the models according to the spatial positions of the calibration points. The specific registration method can be camera calibration, and / or registration based on the transfer relationship between the spatial coordinate system and the plane coordinate system, and / or registration combined with an image fusion registration algorithm. The specific registration process can be set based on the requirements of actual applications, which will not be described here.

[0049] After obtaining the model relative position information, the relative relationship between the spatial positions of the models is determined, and then the two models can be effectively fused to generate a three-dimensional blood vessel passageway human model. The three-dimensional blood vessel passageway human model can not only be used to describe the external three-dimensional state of the human body, but also can determine the blood vessel distribution condition in the human body. In the surgical process, after obtaining the surgical scene image, i.e., the external body surface image of the patient, by using a camera, the current blood vessel distribution condition of the patient can be directly determined by combining the three-dimensional blood vessel passageway human model.

[0050] The above process of constructing a three-dimensional blood vessel passageway human model can be completed in advance before the start of the surgery, so that it can be directly applied in the surgical process. In the surgical process, corresponding sensors can be set on the end device to obtain the position information of the surgical instrument. The position information can be a DSA image corresponding to the surgical instrument, which can be used to determine the spatial position of the surgical instrument in the surgical environment or the patient's blood vessels by identifying the DSA image. It can also be the spatial position of the surgical instrument obtained by using other sensors, such as the spatial position of the surgical instrument determined by infrared, sound wave, etc. Correspondingly, the position information of the surgical instrument can also include the spatial position of the surgical instrument relative to the surrounding environment, for example, when the surgical instrument is located in the patient's body, the position of the surgical instrument relative to the surrounding environment can be determined by the environmental information in the captured image; while determining the position of the surgical instrument by using other sensors, the distribution condition of the surrounding environment can also be obtained.

[0051] In some embodiments, when the position information of the surgical instrument is a DSA image, the process of constructing the blood vessel passage image can be to first identify the surgical instrument in the DSA image, which can be achieved by a corresponding image recognition algorithm, for example, a pre-trained neural network model for identifying the surgical instrument in the DSA image. In addition, the registration points are calibrated for the surgical instrument image and the three-dimensional blood vessel passage human model, and the calibration process can refer to the aforementioned process of constructing the three-dimensional blood vessel passage human model. There is no fixed execution order between the above two steps.

[0052] For the calibrated registration points, the identified surgical instrument in the image can be projected into the three-dimensional blood vessel passage human model, that is, the fusion of the surgical instrument and the three-dimensional blood vessel passage human model is achieved. For example, a three-dimensional model of the surgical instrument can be pre-constructed, and after the spatial distribution position of the surgical instrument is determined, the spatial position of the surgical instrument can also be reflected in the three-dimensional blood vessel passage human model accordingly.

[0053] For the projected three-dimensional blood vessel passage human model, a blood vessel passage image corresponding to the surgical instrument can be generated. The blood vessel passage image can adjust the color, brightness, and gray scale values of the surgical instrument and the blood vessel, so that the blood vessel and the surgical instrument can be clearly distinguished in the displayed image. Since the three-dimensional blood vessel passage human model itself is a three-dimensional model, a three-dimensional image can be effectively generated based on the three-dimensional blood vessel passage human model, and the distribution position information of the surgical instrument in the human body can be intuitively reflected.

[0054] After obtaining the blood vessel passage image, the blood vessel passage image can be sent to the host device to enable the host device to display the blood vessel passage image to the doctor through a display device. As shown in Figure 7 , it is a schematic diagram of displaying the blood vessel passage image based on the display device.

[0055] Correspondingly, when the blood vessel passage image is an augmented reality image, the display device is an AR display device, so that the blood vessel passage image can be effectively displayed, and the spatial distribution condition of the surgical instrument can be more intuitively displayed.

[0056] Although the vascular access image is a three-dimensional augmented reality image, it can generally only display the patient's vascular distribution and the position of surgical instruments from one perspective. The human body's blood vessels are relatively complex. Therefore, in order to better grasp the distribution of blood vessels and the spatial posture of surgical instruments, in some embodiments, the display device may also have interactive features. For example, the screen of the display device may be a touch screen for receiving corresponding instructions input by the user. After the doctor inputs the interactive instruction to the interactive display device, he can adjust the augmented reality display image. The adjustment operation includes at least one of rotation, scaling, perspective, hiding, and layer overlay, so that the displayed image meets the doctor's viewing needs. Figure 8 , which is a schematic diagram of operating an augmented reality display image based on different operation instructions.

[0057] However, when performing interventional surgery on blood vessels, it may be necessary to penetrate deeper into smaller vessels to perform the operation. When surgical instruments move within large vessels, the spatial position of the surgical instruments relative to the vessels can be more clearly understood directly through the above-mentioned vascular access images. However, when surgical instruments penetrate deeper into smaller vessels, the vascular access images may not be able to accurately and timely understand the surgical status. For example, the vascular access images may need to be rotated and magnified, which may interfere with the progress of the operation.

[0058] To address the aforementioned technical issues, when faced with the aforementioned situation, the doctor can switch the display mode of the display device from the first display mode to the second display mode. In the second display mode, the display device can display a three-dimensional navigation image. The three-dimensional navigation image includes a first-person view of the end of the surgical instrument, thereby effectively depicting the vascular bifurcation along the surgical instrument's path.

[0059] Specifically, the process of constructing the 3D navigation image involves determining the spatial position of the surgical instrument, locating that spatial position within a 3D vascular access human model, and thereby determining the surgical instrument's path. Based on the path, the 3D vascular access human model can be sliced ​​to obtain slice images at multiple locations along the path. The 3D navigation image can be constructed from these slice images. Accordingly, the 3D navigation image can also be an enhanced display image, allowing the physician to more intuitively understand the extent of the vascular path.

[0060] It should be noted that when acquiring slice images, the thickness of the blood vessel wall in the three-dimensional vascular pathway human body model can be ignored to ensure that the three-dimensional navigation image can be effectively constructed.

[0061] By displaying three-dimensional navigation images in the second display mode, doctors can view the current vascular extension path from a first-person perspective when facing complex vascular distribution areas or areas with many vascular bifurcations during surgery, and then intuitively select the correct direction of advance to optimize the surgical execution effect.

[0062] In addition, doctors can switch from the first display mode to the second display mode, or from the second display mode to the first display mode at any time according to their needs to adapt to different surgical scenarios.

[0063] It should be noted that the construction of the three-dimensional vascular access human body model and the generation of the vascular access image and the three-dimensional navigation image can be completed mainly by the display device or by other independent computing devices, and there is no limitation on this.

[0064] Since in the current surgical environment, the doctor cannot directly operate the surgical instruments on the slave device, the images displayed alone may be different from the traditional surgical operation process, which may affect the doctor's operation effect. Therefore, in some embodiments, a force sensor can also be set on the slave device. The force sensor can obtain the movement resistance of the surgical instrument during the movement process. Specifically, the movement resistance may include propulsion force and rotational force. Accordingly, the slave device can determine the corresponding resistance type, resistance size, resistance direction and other parameters based on the information collected by the force sensor, and feed these parameters back to the master device.

[0065] In cases where the master device performs interventional surgery based on an operator, meaning the doctor controls the movement of the surgical instrument by moving the operator's hand, the master device can feed back the received information corresponding to the movement resistance to the operator, allowing the doctor to sense the current resistance changes of the surgical instrument based on the operator's hand. For example, when the resistance to the forward movement of the surgical instrument increases, the resistance encountered by the doctor when pushing the operator's hand forward will also increase. By providing real-time feedback of the resistance encountered by the surgical instrument to the doctor, the surgical condition is made closer to the traditional surgical operation process, and the doctor can effectively grasp the current surgical condition, ensuring the execution effect of the surgery.

[0066] Preferably, when the main-end device includes a display device, the display device can also display the current moving speed value and / or propulsion resistance value of the surgical instrument, so that the doctor can not only determine the resistance through the feedback of the operator, but also intuitively determine the resistance change status during the operation based on the displayed numerical changes, thereby further optimizing the surgical effect.

[0067] In some embodiments, the image processing module can also identify a lesion region in the vascular access image or the three-dimensional navigation image. The lesion region can be identified in the model by the doctor or other operators in advance, or the features of the lesion region can be determined in advance, and the lesion region in the vascular access image or the three-dimensional navigation image can be determined based on image recognition technology.

[0068] After detecting the lesion region, in order to ensure the execution effect of the operation of the doctor, the lesion region can be enlarged. When the enhanced reality display image after enlargement is displayed to the doctor, the doctor can more clearly understand the condition of the lesion region of the patient, and the execution effect of the operation is optimized.

[0069] In some embodiments, after identifying the lesion region, the image processing module can also calculate the distance between the surgical instrument and the lesion region. When it is detected that the distance is less than a preset warning distance, it means that the surgical instrument is currently close to the lesion region, and accordingly a prompt information can be sent to the doctor to remind the doctor that the surgical instrument is close to the lesion region. The prompt information can be vibration information generated based on a vibration unit, for example, a vibration unit can be arranged in the operating hand to vibrate the operating hand to remind the doctor; or a speaker can be arranged on the master device to generate sound information based on the speaker to remind the doctor that the surgical instrument is currently close to the lesion region.

[0070] Preferably, in order not to interfere with the operation process of the doctor, the prompt information can only be maintained for a short time to prevent the doctor from being disturbed by the vibration of the operating hand or the voice during the operation process and affecting the normal operation of the operation.

[0071] In some embodiments, in order to optimize the operation process of the doctor, the master device can also include a voice analysis model. The voice analysis module can obtain the voice instruction of the doctor, and analyze the voice instruction based on voice recognition technology to obtain the corresponding operation instruction.

[0072] The operation instruction can be an operation instruction for the display device, for example, the image displayed by the display device can be rotated, enlarged, etc. The operation instruction can also be an operation instruction for the slave device, for example, sent to the slave device to control the surgical instrument to perform the corresponding operation based on the operation instruction.

[0073] By setting the voice instruction collection and analysis, the doctor can directly control through voice when it is inconvenient to control the operating hand or adjust the picture displayed by the display device during the operation process, which improves the operation experience of the doctor.

[0074] Based on the above-described embodiments, the surgical operating system for interventional surgery allows doctors to control the slave device using the operator's hand on the master device while simultaneously displaying surgical scene images in different display modes via the display device. In the first display mode, a vascular pathway model can be displayed to depict the spatial position of surgical instruments within the three-dimensional vascular pathway human model and their relative spatial positions relative to the blood vessels. In the second display mode, a three-dimensional navigation image can be displayed, providing a first-person perspective of the surgical instruments, thereby visually depicting the vascular bifurcations along the instrument's path. Through this operating system, the display mode can be switched to meet the different application requirements of the surgical scenario. For example, when moving surgical instruments within large vessels, the first display mode can be used to determine the overall distribution of the surgical instruments within the vessel, while when moving surgical instruments within small, complex vessels, the second display mode can be used to intuitively and effectively control the surgical instruments along the correct path. By switching display modes, the doctor's diverse surgical needs can be effectively met, ensuring the effective execution of the surgery and optimizing the patient's surgical experience.

[0075] Based on the above surgical operation system for interventional surgery, an image display method for interventional surgery according to an embodiment of this specification is introduced. The execution subject of the image display method for interventional surgery can be the surgical operation system for interventional surgery. Figure 9 As shown, the surgical robot speed limiting method includes the following specific implementation steps.

[0076] S910: Displaying a vascular access image using a display device; the vascular access image includes an augmented reality display image; the vascular access image is used to describe the spatial posture of the surgical instrument in the three-dimensional vascular access human body model and its relative spatial position corresponding to the blood vessel.

[0077] The display device can be used to display images corresponding to the interventional surgery process. Specifically, the display device has a first display mode and a second display mode, each of which is used to display different types of images.

[0078] In the first display mode, the display device is configured to display a vascular access image corresponding to a three-dimensional vascular access human model. The three-dimensional vascular access human model depicts the spatial distribution of a patient's blood vessels. The vascular access image, based on this spatial distribution, depicts the spatial position of surgical instruments within the blood vessels. The vascular access image not only displays the patient's overall vascular distribution, but also illustrates the position of surgical instruments within the body, particularly within the blood vessels.

[0079] Preferably, the vascular access image is an augmented reality display image, that is, a three-dimensional image obtained by adding markers and other layers to a real scene image.

[0080] The three-dimensional vascular access human model can be a model corresponding to the vascular access of the patient, used to describe the three-dimensional spatial distribution state of the blood vessels of the patient. In the case of an interventional surgery being a vascular interventional surgery, constructing the three-dimensional vascular access human model can effectively represent the vascular distribution state of the patient, and in combination with the position of the surgical instrument, the relative position information of the surgical instrument compared with the blood vessels of the patient can be simulated and generated.

[0081] Specifically, when constructing the three-dimensional vascular access human model, a three-dimensional human model corresponding to the patient can be constructed first, and then a vascular access model can be constructed according to the angiogram of the patient. The three-dimensional vascular access human model can be obtained by fusing the three-dimensional human model and the vascular access model.

[0082] The three-dimensional human model is a model used to represent the overall external morphology of the patient. Specifically, before the surgery starts, the patient can be photographed by a camera to obtain patient images and / or surgery scene images, and the construction of the three-dimensional human model corresponding to the patient can be completed by integrating the patient images and / or surgery scene images. The specific construction method may, for example, be based on a neural network model to identify the photographed images to determine the three-dimensional human model, or point cloud data corresponding to the patient's body can be obtained to construct the three-dimensional human model. As shown in Figure 4 is a schematic diagram of a three-dimensional human model.

[0083] Preferably, when constructing the three-dimensional human model, the corresponding angiogram can also be combined to generate the three-dimensional human model. For example, a CT angiogram can be obtained, and the three-dimensional human model can be generated based on the CT angiogram and the patient images and surgery scene images. Since the CT angiogram can clearly show the details of the blood vessels in each part of the body, it can also effectively represent the overall three-dimensional morphology of the patient.

[0084] The specific steps of constructing the three-dimensional human model can be set based on the requirements of actual applications, which will not be described here.

[0085] The vascular access model is used to describe the spatial distribution state of the blood vessels in the patient's body. Specifically, before the surgery starts, the angiogram of the patient can be obtained first. The angiogram is an image generated by injecting a contrast agent into the blood vessels of the patient and then photographing the patient using a corresponding angiographic device. As shown in Figure 5 is a schematic diagram of an angiogram.

[0086] Specifically, the angiography images may include DSA (Digital subtraction angiography) and CTA (CT angiography). The vascular pathway model can be constructed based on the angiography images. Figure 6 Detailed steps for constructing the vascular access model can be set based on actual application requirements and will not be described in detail here.

[0087] Because interventional procedures may also target organs, in some embodiments, the vascular access model also includes an organ model constructed using organ anatomy. For example, the organ could be the patient's heart, effectively reflecting the spatial distribution of the patient's organs. By integrating the organ model, different types of interventional procedures can be effectively and in real time, ensuring a robust application environment.

[0088] Since the three-dimensional human body model cannot reflect the distribution of blood vessels in the patient's body, and the vascular access model can only reflect the spatial distribution of blood vessels, it is difficult to directly associate it with the surgical scene. Therefore, the two models are fused to obtain a three-dimensional vascular access human body model, which is then applied to the surgical process to better understand the surgical conditions.

[0089] Specifically, the 3D human body model and the vascular access model can be calibrated to identify key points. The two models can then be registered based on the calibrated points to determine relative positional information between the models. Based on this relative positional information, the two models are fused to produce a 3D vascular access human body model.

[0090] The registration process primarily determines the relative spatial position of the entire model based on the spatial positions between the calibration points. Specific registration methods can include camera calibration, and / or registration based on the transfer relationship between spatial coordinate systems and planar coordinate systems, and / or registration using an image fusion registration algorithm. The specific registration process can be tailored to the needs of the application and will not be detailed here.

[0091] After obtaining the relative position information of the models, the relative relationship between their spatial positions is determined. The two models can then be effectively fused to generate a 3D vascular access human model. This 3D vascular access human model not only describes the external 3D state of the human body but also identifies the internal vascular distribution. During surgery, the camera captures an image of the surgical scene—that is, an image of the patient's external body surface—and combines it with the 3D vascular access human model to intuitively determine the patient's current vascular distribution.

[0092] The process of constructing the three-dimensional vascular access human model can be completed in advance before the operation starts, so that it can be directly applied during the operation. During the operation, corresponding sensors can be set on the end device to obtain the position information of the surgical instrument. The position information can be a DSA image corresponding to the surgical instrument, which can determine the corresponding spatial position of the surgical instrument in the surgical environment or the patient's blood vessels by identifying the DSA image, or the spatial position of the surgical instrument obtained by other sensors, such as the spatial position of the surgical instrument determined by infrared, sound wave, etc. Correspondingly, the position information of the surgical instrument can also include the spatial position of the surgical instrument relative to the surrounding environment, for example, when the surgical instrument is located in the patient's body, the position of the surgical instrument relative to the surrounding environment can be determined by the environmental information in the captured image; while determining the position of the surgical instrument by other sensors, the distribution of the surrounding environment can also be obtained.

[0093] In some embodiments, when the position information of the surgical instrument is a DSA image, the process of constructing the vascular access image can be to identify the surgical instrument in the DSA image, which can be realized by a corresponding image recognition algorithm, for example, a pre-trained neural network model is used to identify the surgical instrument in the DSA image.

[0094] In addition, the registration points are calibrated for the surgical instrument image and the three-dimensional vascular access human model, respectively. The calibration process can refer to the process of constructing the three-dimensional vascular access human model described above. There is no fixed execution order between the above two steps.

[0095] For the calibrated registration points, the identified surgical instrument in the image can be projected into the three-dimensional vascular access human model, that is, the fusion of the surgical instrument and the three-dimensional vascular access human model is realized. For example, a three-dimensional model of the surgical instrument can be constructed in advance, and after the spatial distribution position of the surgical instrument is determined, the spatial position of the surgical instrument can also be reflected in the three-dimensional vascular access human model accordingly.

[0096] For the projected three-dimensional vascular access human model, a vascular access image corresponding to the surgical instrument can be generated. The vascular access image can adjust the color, brightness, gray scale, etc. of the surgical instrument and the blood vessels, so that the blood vessels and the surgical instrument can be clearly distinguished in the displayed image. Since the three-dimensional vascular access human model itself is a three-dimensional model, a three-dimensional image can be effectively generated based on the three-dimensional vascular access human model, and the distribution position information of the surgical instrument in the human body can be intuitively reflected.

[0097] After obtaining the vascular access image, the vascular access image can be sent to the host device to enable the host device to display the vascular access image to the doctor through the display device. For example, the doctor can adjust the color, brightness, etc. of the vascular access image through the display device, so that the doctor can clearly observe the distribution of the surgical instrument in the patient's body. Figure 7 , which is a schematic diagram of displaying a vascular access image based on a display device.

[0098] Correspondingly, when the vascular access image is an augmented reality image, the display device is an AR display device, so that the vascular access image can be effectively displayed, and the spatial distribution of surgical instruments can be more intuitively displayed.

[0099] Furthermore, while the vascular access image is displayed, if the doctor inputs corresponding operating instructions using the control device on the master device and moves the surgical instrument based on the operating instructions, the spatial position of the surgical instrument will change. Therefore, the vascular access image can be updated based on the spatial position of the surgical instrument after the movement. The specific construction method can refer to the process of constructing the vascular access image described above, thereby ensuring that the displayed vascular access image corresponds to the actual application situation and ensuring the effective execution of the surgery.

[0100] S920: After receiving the display mode switching instruction input by the doctor, a three-dimensional navigation image is generated; the three-dimensional navigation image includes a first-person display image with the end of the surgical instrument as the perspective; the three-dimensional navigation image is used to describe the vascular bifurcation condition on the forward path of the surgical instrument.

[0101] Although the vascular access image is a three-dimensional augmented reality image, it can generally only display the patient's vascular distribution and the position of surgical instruments from one perspective. The human body's blood vessels are relatively complex. Therefore, in order to better grasp the distribution of blood vessels and the spatial posture of surgical instruments, in some embodiments, the display device may also have interactive features. For example, the screen of the display device may be a touch screen for receiving corresponding instructions input by the user. After the doctor inputs the interactive instruction to the interactive display device, he can adjust the augmented reality display image. The adjustment operation includes at least one of rotation, scaling, perspective, hiding, and layer overlay, so that the displayed image meets the doctor's viewing needs. Figure 8 , which is a schematic diagram of operating an augmented reality display image based on different operation instructions.

[0102] However, when performing interventional surgery on blood vessels, it may be necessary to penetrate deeper into smaller vessels to perform the operation. When surgical instruments move within large vessels, the spatial position of the surgical instruments relative to the vessels can be more clearly understood directly through the above-mentioned vascular access images. However, when surgical instruments and / or robots penetrate deeper into smaller vessels, the vascular access images may not be able to accurately and timely understand the surgical status. For example, the vascular access images may need to be rotated or magnified, which may interfere with the progress of the operation.

[0103] To address the aforementioned technical issues, when faced with the aforementioned situation, the doctor can switch the display mode of the display device from the first display mode to the second display mode. In the second display mode, the display device can display a three-dimensional navigation image. The three-dimensional navigation image includes a first-person view of the end of the surgical instrument, thereby effectively depicting the vascular bifurcation along the surgical instrument's path.

[0104] Specifically, the process of constructing the 3D navigation image involves determining the spatial position of the surgical instrument, locating that spatial position within a 3D vascular access human model, and thereby determining the surgical instrument's path. Based on the path, the 3D vascular access human model can be sliced ​​to obtain slice images at multiple locations along the path. The 3D navigation image can be constructed from these slice images. Accordingly, the 3D navigation image can also be an enhanced display image, allowing the physician to more intuitively understand the extent of the vascular path.

[0105] It should be noted that when acquiring slice images, the thickness of the blood vessel wall in the three-dimensional vascular pathway human body model can be ignored to ensure that the three-dimensional navigation image can be effectively constructed.

[0106] By displaying three-dimensional navigation images in the second display mode, doctors can view the current vascular extension path from a first-person perspective when facing complex vascular distribution areas or areas with many vascular bifurcations during surgery, and then intuitively select the correct direction of advance to optimize the surgical execution effect.

[0107] S930: Display the three-dimensional navigation image using a display device.

[0108] After acquiring the three-dimensional navigation image, the display device can display it so that the doctor can intuitively grasp the distribution of blood vessels in the direction of advance of the surgical instrument and ensure the effective implementation of the operation.

[0109] Accordingly, while displaying the 3D navigation image, if the doctor inputs corresponding operation instructions using the control device on the master device and moves the surgical instrument based on the operation instructions, the spatial position of the surgical instrument will change. Therefore, the 3D navigation image can also be updated based on the spatial position of the surgical instrument after the movement.

[0110] The specific construction method can refer to the aforementioned process of constructing a three-dimensional navigation image, thereby ensuring that the displayed three-dimensional navigation image corresponds to the current position of the surgical instrument, thereby ensuring the effective conduct of the operation.

[0111] Although the process flows described above include multiple operations occurring in a particular order, it should be understood that these processes may include more or fewer operations, which may be performed sequentially or in parallel (eg, using parallel processors or a multi-threaded environment).

[0112] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0115] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0116] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0117] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0118] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] Embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In distributed computing environments, program modules may be located in local and remote computer storage media, including storage devices.

[0120] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the embodiments in this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0121] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A surgical operating system, characterized in that: Including master end device and slave end device; The slave device includes a surgical instrument; the slave device is used to obtain position information of the surgical instrument; The master-end device includes a control device and a display device; the control device is used to obtain the doctor's operating instructions so that the slave-end device controls the movement of the surgical instrument based on the operating instructions; the display device corresponds to a first display mode; the first display mode is used to display a vascular access image corresponding to a three-dimensional vascular access human body model; the vascular access image includes an augmented reality display image; the vascular access image is used to describe the spatial posture of the surgical instrument in the three-dimensional vascular access human body model and the relative spatial position corresponding to the blood vessel; the display device also corresponds to a second display mode; the second display mode is used to display a three-dimensional navigation image; the three-dimensional navigation image includes a first-person display image with the end of the surgical instrument as the viewpoint; the three-dimensional navigation image is used to describe the vascular bifurcation condition on the forward path of the surgical instrument; the display device is also used to receive the doctor's switching instructions to convert different display modes.

2. The surgical operating system according to claim 1, wherein: The three-dimensional vascular access human body model is obtained by the following method: constructing a three-dimensional human body model based on the surgical scene image corresponding to the patient; constructing a vascular pathway model based on the angiographic image of the patient; The three-dimensional human body model and the vascular access model are fused to obtain the three-dimensional vascular access human body model.

3. The surgical operating system according to claim 2, wherein: The fusing of the three-dimensional human body model and the vascular access model to obtain the three-dimensional vascular access human body model includes: Registering the three-dimensional human body model and the vascular pathway model to obtain relative position information of the models; wherein the registration includes: performing registration using camera calibration, and / or performing registration based on a transfer relationship between a spatial coordinate system and a planar coordinate system, and / or performing registration in combination with an image fusion registration algorithm; The three-dimensional human body model and the vascular access model are fused based on the relative position information of the models to obtain the three-dimensional vascular access human body model.

4. The surgical operating system according to claim 1, wherein: The position information of the surgical instrument includes a DSA image; the vascular access image is obtained by: identifying a surgical instrument in the DSA image; calibrating registration points for the DSA image and the three-dimensional vascular access human body model respectively; projecting the identified surgical instrument into the three-dimensional vascular access human body model based on the registration points; A vascular access image corresponding to the surgical instrument is generated based on the projected three-dimensional vascular access human body model.

5. The surgical operating system according to claim 1, wherein: The manipulation device includes an operator; the operator is provided with a position sensor; the position sensor is used to obtain motion information of the operator when the operator is moved by manipulation of the doctor, so that the slave device controls the movement of the surgical instrument based on the motion information.

6. The surgical operating system according to claim 5, wherein: The slave device further includes a force sensor; the force sensor is used to obtain the movement resistance of the surgical instrument; the movement resistance includes propulsion force and rotational force; Accordingly, the operating hand is used to feedback a resistance corresponding to the movement resistance when being manipulated by a doctor.

7. The surgical operating system according to claim 6, wherein: The display device is further used to display the moving speed value and / or the propulsion resistance value corresponding to the surgical instrument.

8. The surgical operating system according to claim 1, wherein: The display device is further used to identify a lesion area in the vascular access image and / or the three-dimensional navigation image; correspondingly, it is also used to magnify the lesion area in the vascular access image and / or the three-dimensional navigation image.

9. The surgical operating system according to claim 8, wherein: The control device is used to send a prompt message to the doctor when the distance between the surgical instrument and the lesion area is less than a preset warning distance; the prompt message includes vibration information generated by the vibration unit and / or sound information generated by the speaker.

10. The surgical operating system according to claim 1, wherein: The display device is further configured to receive interactive instructions from a doctor to perform adjustment operations on the vascular access image in the first display mode; the adjustment operations include at least one of rotation, scaling, perspective, hiding, and layer overlay.

11. The surgical operating system according to claim 1, wherein: It also includes a voice analysis module; the voice analysis module is used to obtain the doctor's voice instructions and parse the voice instructions to obtain operation instructions, so that the slave device performs corresponding operations based on the operation instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the following steps are implemented: Displaying a vascular access image corresponding to the three-dimensional vascular access human body model using a first display mode corresponding to the display device; the vascular access image includes an augmented reality display image; the vascular access image is used to describe the spatial position of the surgical instrument in the three-dimensional vascular access human body model and its relative spatial position with respect to the blood vessel; After receiving a display mode switching instruction input by a doctor, a three-dimensional navigation image is displayed using the corresponding second display mode of the display device; the three-dimensional navigation image includes a first-person display image with the end of the surgical instrument as the viewpoint; the three-dimensional navigation image is used to describe the vascular bifurcation condition along the advancement path of the surgical instrument; The three-dimensional navigation image is displayed using a display device.

13. The computer-readable storage medium of claim 12, wherein: Displaying a vascular access image corresponding to a three-dimensional vascular access human body model using a first display mode corresponding to a display device includes: Obtaining the doctor's operating instructions received by the control device; moving the surgical instrument based on the operating instructions; updating the vascular access image based on the spatial posture of the surgical instrument after the movement; Correspondingly, the display device displays the three-dimensional navigation image in the corresponding second display mode, including: Obtaining the doctor's operating instructions received by the control device; moving the surgical instrument based on the operating instructions; determining a target spatial position of the end of the moved surgical instrument; A corresponding three-dimensional navigation image is constructed with the target spatial position as a viewpoint.

Images