Intraoperative augmented reality system

By using augmented reality system to project artificially generated reference illustrations onto the body, the problem of low data utilization efficiency in surgery in the prior art is solved, the accuracy and safety of the surgery are improved, and the implant is ensured. Correct position and angle.

CN114795464BActive Publication Date: 2024-12-06MATECAN CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210458404.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-04-29
Filing Date
2022-04-28
Publication Date
2024-12-06
Estimated Expiration
2042-04-28

AI Technical Summary

Technical Problem

The prior art is difficult to effectively utilize two-dimensional and three-dimensional data during surgery, which makes it difficult for surgeons to accurately determine whether the implant’s position and fixation device are correct during surgery, and it is difficult to evaluate whether the implant enters bone or adjacent tissue, affecting the Accuracy and safety of the surgery.

Method used

Using an augmented reality system, the artificially generated reference illustrations are projected onto the body's natural perception through visualization devices, providing optical presentation of surgical areas, including displays of bones, implants and drilling channels, ensuring that surgeons can Operation is performed at the correct angle and position during the operation and projection deviation is corrected with optical or acoustic warning messages.

Benefits of technology

Improves the accuracy and safety of the surgery, ensures the correct position and angle of the implant, reduces errors during the surgery, and enhances the surgeon's operational confidence and reliability of surgical results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114795464B_ABST
    Figure CN114795464B_ABST
Patent Text Reader

Abstract

The present invention relates to augmented reality, and in particular to an intraoperative augmented reality system. A system is provided, wherein data for generating an artificial diagram is provided to a visualization device for assisting surgery on a human or animal body. The visualization device is used to combine the natural perception of at least one area of ​​the user's body with at least one artificially generated diagram. The artificial diagram is a reference diagram of the body area, and for the purpose of performing surgery, the artificial diagram is optically projected onto the natural perception of the body area with the aid of the visualization device, or, in the absence of projection, is placed next to the body for intraoperative comparison.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to augmented reality, and in particular to an intraoperative augmented reality system.

[0002] The invention relates to an intraoperative augmented reality method for assisting surgery on the human or animal body according to claim 1 .

[0003] The invention also relates to the use of a visualization device according to claim 20 and to a system for carrying out the method according to claim 21 . Background Art

[0004] German patent DE 10 2016 218 364 A1 "A method for performing a procedure on a patient with the aid of a medical system". The invention includes the following steps: determining a target value for each sub-step of the procedure; detecting an actual value for each sub-step of the procedure. Comparing the target value with the actual value for each sub-step of the procedure to determine the deviation between the target value and the actual value.

[0005] It would be desirable to be able to provide a surgeon with an approach for performing surgery on an animal and / or human body such that data and / or information generated during the surgical planning phase can be provided to the surgeon in two-dimensional and three-dimensional form during surgery.

[0006] The invention aims to achieve, through a method and a system, providing data for generating an artificial diagram to a visualization device for combining natural perception of at least one area of ​​the user's body with at least one artificially generated diagram during surgery on the human or animal body. Summary of the invention

[0007] The present invention aims to solve the defects in the prior art and provides an intraoperative augmented reality system.

[0008] Among them, artificial illustration: the present invention understands the term "artificial illustration" as a "reference illustration" that can be compared with the natural perception of the corresponding area of ​​the human or animal body.

[0009] Artificial diagrams are depicted as reference diagrams for corresponding areas of the body.

[0010] For the purpose of performing the surgery, the artificial representation is optically presented with the aid of a visualization device or is projected onto the natural perception of the body region.

[0011] The reference representation preferably shows an artificial representation of at least one section of the region of the body to be operated on.

[0012] The reference illustration can show vertebrae and / or bones of the human skeleton and can be supplemented with other elements, such as implants. The reference illustration can show a single region of the body (e.g. a vertebra) in combination with at least one other region of the body (e.g. a hip joint). It follows that each region of the human body can be shown individually or together with at least one other region of the human body in the form of an artificial illustration of the reference illustration.

[0013] The reference representation is optically projected for the surgeon by means of a visualization device onto the natural perception of the corresponding body region or, alternatively, is placed next to the corresponding body region.

[0014] The artificial representation can be projected as a reference representation with the aid of a visualization device relative to the natural perception of the corresponding area of ​​the body above or below or on at least one side. The artificial representation includes the positioning and / or the dimensions of at least one inserted implant and / or provided endoprosthesis, Kirschner wire (K-wire), planning report and / or information about the implant.

[0015] If the optical projection of the artificial representation deviates from the user's natural perception of the body area, the visualization device triggers an optical or acoustic warning message.

[0016] The deviation of the optical projection can relate to the length and / or width and / or thickness of the object, its natural perception compared to its artificial representation.Alternatively, the deviation can also relate to the positioning of the attached markings and their ratio to each other.

[0017] The reference representation can be displayed at a 1:1 ratio relative to the corresponding natural perception of the body region. As a reference representation, the artificial representation can be presented reduced or enlarged relative to the natural perception of the corresponding body region and its positioning in space can be changed.

[0018] In order to prepare and / or to carry out and / or to control an operation for a corresponding surgeon, a reference diagram, for example, can show or include a drilling channel. The drilling channel describes the length and / or the corresponding diameter of the drill hole. It shows the entry angle of the drill hole in the corresponding area of ​​the body to be operated on. The reference diagram of the drilling channel serves as an aid to the surgeon. In particular, the surgeon can ensure with the help of the reference diagram that the screw he wants to use for the operation has the correct length or the correct strength or the correct entry angle into the bone. For this purpose, the reference diagram can be adapted, for example, in terms of transparency, decomposed into previously defined sub-areas and derive specific areas.

[0019] Obviously, the reference illustrations may show any area of ​​the human or animal body. The reference illustrations may show at least one area of ​​a preferred implant.

[0020] In the context of the present invention, the term "artificially generated diagram" has the same meaning as the term "reference diagram".

[0021] The data for generating the artificial representation are provided with the aid of a graphical aid.

[0022] The artificially generated diagram can be a scale. Obviously, the artificially generated diagram can also be another measurement system.

[0023] The artificially generated representation can also be a two-dimensional or multi-dimensional image of the body region to be treated. This can also be a representation of other planning contents related to the planned operation.

[0024] The manually generated representation may be a cutting line that serves as a reference representation for the surgeon to guide an incision in an area of ​​the body.

[0025] The manually generated diagram can also be a drilling line. The drilling line must indicate at which entry depth a hole is to be drilled, preferably in a bone, for example in order to be able to insert an implant in a specific area of ​​the body. The manually generated diagram provides the entry angle at which a preferred screw and / or implant should be arranged in a body area.

[0026] The artificially generated representation displays, for example, an at least two-dimensional image of an implant. The artificial representation may also include more than two dimensions.

[0027] By means of the artificially generated representation, at least one implant may preferably be selected from a plurality of implants, but not limited thereto. The implants may be of different types, sizes and / or materials.

[0028] The reference diagram shows the state of the body region corresponding to the corresponding area in the preoperative stage. The reference diagram shows the implant before implantation in the body. The implant can also be shown in its position in the body after implantation.

[0029] The artificial representation is projected onto the natural perception of the body region, in particular with the aid of at least one reference mark, preferably congruently. The reference mark may be an optical reference mark, which serves to attach an optical reference point to the body region. The at least one optical reference point serves for the surgeon to orientate the artificial representation when projecting it onto / on the body. Alternatively, the artificial representation can also be used without projection, in that it is freely displayed in space and used for comparison with the corresponding body region.

[0030] Compared with the prior art, the present invention has beneficial effects.

[0031] The invention provides a method in which data for generating an artificial representation are provided to a visualization device for assisting surgery on the human or animal body.

[0032] The visualization device is used to combine natural perception of at least one area of ​​the user's body with at least one artificially generated illustration.

[0033] The artificial representation is a reference representation of the body region and, for the purpose of surgery, is optically projected onto the natural perception of the body region with the aid of the visualization device or, without projection, is placed next to the body for intraoperative comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. The protection scope of the present invention is not limited to the following description.

[0035] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0036] Surgery and procedures to perform surgery:

[0037] The surgeon plans the operation with the help of preoperative planning. According to the invention, at least one artificial representation of at least one area of ​​the body is generated during preoperative planning. During preoperative planning, at least one representation of an implant is generated.

[0038] The object of the operation can be one or more surgical procedures on the body. For example, but not exclusively, the surgical procedure includes bone removal, insertion of an implant, and selection of the implant according to specific parameters. Parameters are, for example, the size and type of the implant. Parameters can include different geometric requirements. Geometric requirements can be dimensional measurements, such as the height, width, range, and orientation of a skeletal part and / or a bone part. Parameters can relate to bones, soft parts, tissues, and / or organs.

[0039] In the case of surgical treatment by the introduction of implants, 3D patient images, which are usually X-ray images, are superior to 2D patient images for preoperative planning. When generating 3D patient images, a lower radiation dose is applied to the patient than with X-rays.

[0040] In the case of 2D patient images, the zoom and spatial depth in the viewing direction of the photo cannot be displayed, or can only be displayed in a limited manner. It is difficult to evaluate whether the implant is correctly attached to the patient's bone or on the bone. It is difficult to judge the fixture of the implant, such as whether the screw is correctly engaged in the bone, or whether it protrudes beyond the bone using traditional methods. In addition, it is usually unclear whether the screw will enter the adjacent bone and / or adjacent tissue and / or adjacent soft part. It is not always possible to take a 2D photo of the patient from the required viewing direction.

[0041] When a coordinate relationship is established between the 2D patient image and the 3D patient image, the corresponding processing steps of the planner / surgeon can be transferred to other representations in a coordinate-dependent manner. Thus, the processing steps can be planned and checked in all selected representations. The representations include in particular the loaded and unloaded positions of the corresponding body segments. The representations include the relevant dimensions, preferably in the comparison of target values ​​and actual values.

[0042] Method according to the present invention also relates to work and planning steps, and described work and planning steps are carried out in the work diagram by planner / surgeon, and especially need to be carried out in association with determined possible implant.Work and planning steps preferably include but are not limited to possible implants based on shown implant model positioning.Location on vertebra or spine can indicate the shape and / or size variation of implant model.Can show the distortion or displacement of implant model.Similarly, can show the variation of the entry angle of implant model, for example with respect to the variation of vertebra.Size or measurement result can be displayed on one or more screen windows simultaneously, and are proportional to spatial coordinates.

[0043] Preoperative planning and / or surgical simulations include one or more artificial illustrations of the body segments of the patient undergoing the operation. These illustrations are constructed as working illustrations, in which at least one model of an implant that is movable relative to the body segment can be arranged. Simulations will be discussed separately below.

[0044] The term “surgery on the human body” is understood by the present invention to mean a preferably instrumented surgical procedure which is performed on and / or in the body of a patient.

[0045] Surgery is preferably performed for therapeutic purposes. It may also be performed to perform further diagnostics.

[0046] The surgery may preferably, but not exclusively, be a gynecological surgery. It may be obstetric, but the surgery may also be performed in urology or otolaryngology.

[0047] Human surgery is referred to as surgery hereafter, and includes the preoperative stage, intraoperative stage, and postoperative stage.

[0048] Human or animal body.

[0049] The method according to the invention and the associated system can be applied to all areas of the human and / or animal body. For reasons of simplicity only the human body or the simplified patient will be mentioned below.

[0050] The method and the system of the present invention are related to all aspects and components of the human body, exemplarily and not exclusively, the human head, skin, limbs, organs and bones.

[0051] In the following, the method and the associated system will be described using bones of a skeleton as an example.

[0052] A facility for performing surgery on the human body.

[0053] The facility used to perform surgery on the patient can be a hospital or a clinic. In addition, it can refer to a rehabilitation clinic or a polyclinic. The facility can be a clinic or other medical facility.

[0054] Obviously, it may also refer to any other facility used to perform surgery on the human body.

[0055] Medical and nursing assistance is provided to patients in this facility.

[0056] In the hospital, once a patient's illness and / or affliction and / or physical impairment is identified, a plan is developed to treat the illness, affliction or physical impairment and surgery is performed.

[0057] Facilities that perform surgery on the human body may also refer to obstetrics or end-of-life care.

[0058] A facility for performing surgery on a human body may refer to emergency treatment measures. In addition, there may be fully fixed or partially fixed facilities. In a facility for performing surgery on a human body, pre-hospitalization and / or post-hospitalization treatment may be performed.

[0059] Likewise, outpatient therapeutic measures or rehabilitation therapeutic measures may be performed.

[0060] The foregoing and all subsequent lists are exemplary only and are by no means exhaustive.

[0061] Visualization device:

[0062] The present invention understands the term "visualization" as the visualization of preferably abstract data, texts and / or relationships in a graphical and / or visually detectable form.

[0063] A visualization device is provided for displaying data, content, text and / or relationships in a graphical or visually detectable form.

[0064] The visualization device is configured as an optical device. The visualization device can be a smartphone or can be configured as glasses.

[0065] By way of example, but not exclusively, the visualization device for combining a user's natural perception of a body region with at least one artificially generated illustration is designed as glasses, and the glasses operate based on augmented reality.

[0066] The visualization device can also be configured as an optical device and arranged on the body of the person participating in the operation. Preferably, the optical device is arranged on the head of the surgeon.

[0067] The optical device is designed to be pivotable. It can be pivoted from a rest position in the head area into a working position, in which the device is positioned at least in front of the doctor's eyes.

[0068] The facility is arranged at a distance in front of at least one eye of the physician so that abstract data, texts and other relationships can be displayed to the physician in a graphical and / or visually detectable form. The term "physician" is used synonymously with the term "surgeon".

[0069] The visualization device comprises a storage medium for storing data, values ​​and information.

[0070] The data are preferably transmitted wirelessly and / or via a WLAN connection from the hospital's information system to the visualization device.

[0071] The visualization device also comprises a computer unit, with which data, values ​​and information can be processed.

[0072] Obviously, the values, information and data can also be processed in the computer unit in order to be able to be displayed optically in the visualization device.

[0073] The computer unit of the visualization device will be discussed in detail below.

[0074] In order to establish communication between the visualization device and the information system of the hospital, the visualization device comprises at least one interface.

[0075] Communication can also be established between the visualization device and the device of the information system via the interface. And the device is preferably a device located in the ward, which is used to monitor the patient or to carry out further treatment measures. Obviously, the device can also be used for other tasks and functions.

[0076] The visualization device is assigned to at least one person, preferably a surgeon.

[0077] Obviously, the visualization device can also work with the aid of an application, which is preferably, but not exclusively, already installed in a PC, a mobile phone, a tablet or another electronic device.

[0078] The visualization device can be distributed to the medical staff participating in the operation, such as the surgeon, by means of a barcode. The visualization device can also be distributed to the medical staff participating in the operation, such as the surgeon, by means of a QR code. Of course, the visualization device can be distributed to the doctor or another person participating in the operation by scanning a fingerprint. The visualization device is preferably distributed to the surgeon and can be implemented by means of iris recognition. In this case, the iris of the eye, preferably the iris of the surgeon performing the treatment, is recognized by a camera. In this way, the visualization device is distributed to the corresponding doctor.

[0079] The visualization device further comprises a camera for combining the user's natural perception of the body region with at least one artificially generated representation of the body region. The same applies to the natural perception of the implant and the corresponding artificially generated representation.

[0080] Obviously, the visualization device can also be assigned to a specific person by any other technical method.

[0081] A work list can be entered into the visualization device, which visually displays to the respective operator of the visualization device (surgeon) the name and / or location and number of the operating room of the patient, whom the doctor must visit on the respective date for the operation.

[0082] In the work list, other tasks and appointments that the doctor has to carry out on the corresponding day are also displayed visually.

[0083] The work list can predetermine the order in which the respective operations are to be performed for the treating physician. The work list may contain additional points that need to be taken into account when performing the operation on the respective patient.

[0084] Worklists are created using data, information, and values ​​provided by the hospital's information system.

[0085] Computer unit of the visualization device:

[0086] The visualization device comprises a computer unit. The computer unit is used to process and / or edit data, values ​​and information stored in a storage medium of the visualization device.

[0087] The visualization device therefore comprises a computer unit in order to combine the user's natural perception of the body region with at least one associated artificially generated representation.

[0088] All interfaces of the visualization device to the hospital's information system, via which data can be transferred from the hospital's information system to the storage medium of the visualization device, are stored in the computer unit.

[0089] All interfaces of the visualization device to the devices to which the patient's health status is connected during surgery and for further treatment are unified and managed in the computer unit. Also unified and managed in the computer unit is the at least one interface for transferring data, values ​​and information from the hospital's information system to the storage medium of the visualization device.

[0090] In this way, all data, values ​​and information of the devices connected to the patient can be assigned and / or evaluated and / or compared with one another for the purpose of performing the operation.

[0091] In this way, the computer unit of the visualization device communicates with at least one device in the hospital to which the patient is connected for the operation.

[0092] The computer unit of the visualization device is designed to transfer data, values ​​and information back from the hospital's information system during the operation. The transfer can be done manually or automatically.

[0093] The automatic transmission of data, values ​​and / or information depends on the respective operating room in which the treating surgeon and his / her visualization device are currently located.

[0094] The automatic transmission of data from the hospital's information system to the computer unit of the visualization device can also take place according to corresponding times.

[0095] This can also depend on the extent to which the treating physician and / or the carrier of the visualization device has processed the work list assigned to him for the respective operation to be performed.

[0096] Obviously, the owner of the visualization device can call up the data manually and / or by other technical means via the visualization device.

[0097] The data are transferred manually or automatically to the storage medium of the visualization device, and this process can be performed before, during and after the operation, preferably in the operating room when the treating physician approaches the patient with his visualization device and prepares for the operation.

[0098] When the operation on the previous patient is finished and the treating physician leaves the previous operating room, data, values ​​and information about the patient whose operation was accompanied in the work list can be automatically recorded.

[0099] The automatic transmission of data, values ​​and information can involve both data from the hospital's information system and data emitted by at least one device to which the patient is connected in the hospital for monitoring during surgery and / or for further treatment after surgery.

[0100] Natural perception:

[0101] The present invention understands the term "natural perception" as: preferably the surgeon experiences all properties of his environment with his available senses. Natural perception preferably, but not exclusively, involves the surgeon's vision and / or hearing and / or smell and / or taste and / or touch.

[0102] The surgeon is able to identify and categorize all events and things related to the area of ​​the body being operated on. Identification and categorization is also relevant to implants inserted during surgery.

[0103] Before, during and after surgery, the surgeon focuses his vision and / or natural sense on the patient. The surgeon also directs his vision and / or natural sense to the area of ​​the patient's body to be operated on.

[0104] Preferably, natural and / or visual perception by the surgeon and based on light stimulation incident on the retina of the surgeon's eye.

[0105] Generating artificial illustrations with the aid of visualization devices based on “virtual reality” and / or “mixed reality” and / or “augmented reality”:

[0106] The generation of the artificial illustration is performed with the aid of a visualization device, preferably but not limited to, based on “virtual reality” and / or “mixed reality” and / or “augmented reality”.

[0107] By importing data, values ​​and information into the visualization device in the form of augmented reality, the carrier of the visualization device perceives the data, values ​​and information it calls up as a "projection floating in the space in front of him."

[0108] Exemplarily, but not exclusively, it is further assumed that the artificial illustration is generated based on “mixed reality”.

[0109] By importing data, values ​​and information into the visualization device in the form of "mixed reality", the carrier of the visualization device perceives the data, values ​​and information it calls as "projections floating in the space in front of him", and if necessary, these data, values ​​and information can interact with the space or be associated with actual objects.

[0110] The present invention understands the term "virtual reality" as the representation and simultaneous perception of a person in a computer-generated interactive virtual environment generated in real time, for example.

[0111] The term "mixed reality" refers to a person's perception of their natural environment. Computer-generated virtual reality is preferably introduced into the perception of a person's natural environment.

[0112] The present invention defines the term "augmented reality" as an extended reality achieved by computer-assisted expansion of the user's perception. In other words, "augmented reality" preferably refers to a visual representation of information, whereby an image or video is supplemented with additional computer-generated information.

[0113] Furthermore, for simplicity, it is assumed that the visualization device functions based on “mixed reality”.

[0114] In the present invention, data, values ​​and information can be displayed in the form of "mixed reality" in the visualization device.

[0115] By importing data, values ​​and information into the visualization device in the form of "mixed reality", the carrier of the visualization device, preferably the surgeon, perceives the data, values ​​and information he calls up as a "projection floating in the space in front of him".

[0116] Because it is "mixed reality", the wearer of the visualization device can move, expand and / or change the virtual information "floating in the space in front of him".

[0117] With the help of “mixed reality”, the carrier of the visualization device can turn pages in the “book floating in front of him” that is introduced in his visualization device with the help of “mixed reality”. In addition, other objects can also be introduced into the visualization device.

[0118] With the aid of the visualization device, data, values ​​and information can be virtually visualized in the visualization device as an optical representation and can be stored in a storage medium of the visualization device.

[0119] With the aid of "mixed reality", the operator of the visualization device, preferably the surgeon, can import data, values ​​and information into his visualization device. The imported data, values ​​and information are made available to the operator as additional information and / or as comparison values ​​during surgery and / or for evaluating the surgical process and results of the patient.

[0120] To create documentation about the operation, the treating physician can import his oral report into his visualization device in the form of a "mixed reality" illustration.

[0121] In this case, the treating surgeon “sees” the textual form he has created “floating in the space in front of his visualization device.”

[0122] The surgeon can compare the photographs and images that he creates as artificial illustrations while preparing for surgery using a visualization device with the natural perception of the patient or implant during surgery and include them in the surgical documentation.

[0123] The “mixed reality”-based capabilities of the visualization unit enable the treating physician to record the procedure he is performing at the patient’s bedside.

[0124] Since the visualization device works on the basis of "mixed reality", the surgeon sees the surgical report converted into text form "floating in the space in front of the visualization device."

[0125] Thus, the surgeon can check the correctness and completeness of the report at a time closely related to the operation or during the operation.For example, the record of the operation performed by the surgeon can be forwarded to other participants of his surgical team and / or preferably to the information system of the hospital.

[0126] Image aids:

[0127] In the present invention, the image assistance facility is able to visualize, scale and / or read out the data accessible to the surgeon.At least two or more dimensional representations of at least one part of the patient's body may be projected onto the natural perception of the part of the patient's body.

[0128] Preferably, the image-assisted facility is a facility that works with electronic devices and related software and has at least one interface. The image-assisted facility requires calling image data, anatomical data and geometric data in at least one data storage device. The data storage device works with imaging algorithms / methods and other facilities, including unconnected facilities. Data can be received or processed by manual input. The data can be processed in software-related algorithms so that the output of preoperative planning and / or planning data can be transmitted to the display, assistance, measurement or imaging shooting facilities used during the operation. The data can be called or processed in real time during the operation and can be returned to the connected facility.

[0129] Imaging method / algorithm means any method used preoperatively or intraoperatively by which information about the structure or properties of the human or animal body or parts thereof can be obtained and by which the images taken by the image-assisted facility can be received, evaluated, edited and transmitted via an interface to a connected display, assistance, measurement or imaging facility for the purpose of preoperative planning and / or intraoperative control.

[0130] Interface means a device capable of transmitting data generated by image-aiding facilities to display, assist, measure, image capture or navigation facilities, and / or receiving information from display, assist, measure or image capture facilities, by using electrical, radio, optical or magnetic transmission (including program algorithms).

[0131] A navigation system is a device for supporting surgery on the human or animal body. The device is preferably used to adjust the required geometric and spatial proportions. The device can detect the actual geometric proportions and can receive information from the image-assisted device via an interface of the image-assisted device and provide information to the image-assisted device.

[0132] Control means calling up the image information generated by the image-assisted means on the basis of manual input, gestures or information from a connected imaging means or information from a connected means for collecting geometric data, which goes beyond the preoperative planning method.

[0133] Furthermore, the present invention aims to enable scaling of information in patient images.

[0134] According to the invention, the preoperative situation can be analyzed and serve as the basis for the method of the invention by automatically, semi-automatically or manually calculating all relevant angles and other anatomical or surgical parameters, such as material properties, dimensions, etc. of a possible implant to be inserted, as well as patient data.

[0135] The method for planning, preparing, monitoring and tracking a surgical operation is performed on an image-assisted facility which enables the planner / surgeon to perform an archivable, storable and reproducible preoperative plan, thereby simplifying his work.

[0136] An image aid in the sense of the invention enables visualization, scaling and reading of data accessible to a planner / surgeon, showing at least two multidimensional views of at least one section of the patient's body, wherein at least two views differ at least in the number of dimensions.

[0137] In particular, the image processing unit recognizes the provided patient image with regard to its format, for example as a 3D image and / or 3D image data or as a 2D image and / or 2D image data.

[0138] For example, the information can be made accessible to image-supporting facilities via the Internet using suitable transmission protocols and / or in another way, in particular for calling up, loading, downloading and imaging.

[0139] The data transmission can take place directly from the hard disk of the computer or by means of a hard disk, CD, DVD, flash memory device (eg memory pen, compact flash, secure digital card), other mobile or fixed data carrier of a cloud system.

[0140] Additionally, encrypted or unencrypted transmission may be performed via email or other digital transmission device to any compatible computer device, smart phone, PDA, tablet computer, or other device capable of transmitting electronic information.

[0141] It is conceivable that a 2D layer obtained from a 3D patient image or an already existing external 2D layer of a 3D patient image can be accessed by an image-assisted facility and can be used for further processing and / or planning of a surgical procedure, as well as for the insertion of an orthopedic implant. Accordingly, the same applies to 3D images obtained from 2D patient images.

[0142] The image-assisted facility can also derive and calculate 2D layers from 3D patient images and vice versa. To represent a 2D layer, a plane is provided in a three-dimensional body, such as a bone, tissue, joint or soft part, in particular a vertebra of the spine, which plane can extend horizontally or vertically or at an angle to the axis of the spine. The coordinates of the plane are used to determine the coordinates of the 2D layer and allow a perspective view of a specific two-dimensional layer of the three-dimensional body to be displayed. The calculation of the 2D data can be performed automatically.

[0143] Furthermore, 2D and 3D patient images available for the method can be presented in high-resolution MIP or MPR representations with the aid of image-aided facilities.

[0144] Furthermore, according to the invention, the 2D patient image can be processed, in particular scaled, before, during or after it is accessible by means of the image aid. Here, the dimensions of the imaged body segment of the patient, in particular the dimensions of the spine and / or the scale of the image are determined automatically, semi-automatically or manually by means of the image aid, for example by using a known reference object, for example a sphere made of a non-radiation-transmissive material.

[0145] The scaling process may be performed and repeated individually for each individual overlay 2D patient image.

[0146] Image-assisted facilities also allow surgeons to segment or zoom the accessible 3D patient images in different views, parts and / or perspectives. This enables volume recognition, rather than just identifying a single point of the patient image. Segmentation can also be performed and stored in a network or cloud system.

[0147] Segmentation allows identification and medical / anatomical naming and / or numbering of individual bones, vertebrae, joints, soft parts, tissues, etc. Segmentation also enables identification and medical / anatomical naming and / or numbering of the vertebrae of the spine according to reference models and target values. It allows comparison with medical and anatomical ideals and standard values ​​of human or animal anatomy so that the optimal values ​​of the patient's anatomy sought by surgery can be performed as ideally as possible, and the patient's anatomy information can be provided to the image-assisted facility. In addition, individual bones, soft parts and tissue parts can be imported or exported or highlighted in the display by means of the image-assisted facility.

[0148] This process can be performed automatically and checked manually with the aid of image-assisted segmentation, for example, identifying and naming individual bones and / or bone parts, vertebrae, joints, soft parts, tissues, etc.

[0149] In particular, the vertebrae can be shown, described and / or examined based on the accessible sketch.

[0150] The invention, with the aid of image-assisted facilities, can simultaneously display screen windows with different display images on the screen, such as different patient images, sketches of body segments relevant to the operation (such as the spine or vertebrae).

[0151] In another aspect of the present invention, a spatial coordinate relationship can be established between a 2D patient image and a 3D patient image by means of an image-assisted facility. In particular, a correlation can be established between preferred spatial information of data and information in a 2D patient image and a 3D patient image, and vice versa. In this way, information can be transferred, presented and processed from one view to another.

[0152] The invention further provides the possibility of processing the patient image displayed in the at least one screen window by the surgeon using image-aiding tools.

[0153] In particular, for example, the surgeon can activate one of the representations in the at least one screen window by clicking with the cursor in order to carry out processing in the working representation.

[0154] The term "cursor" in this sense is to be understood as any display, input, command input and / or processing device, for example a mouse, trackball, keyboard, touch screen or display of a graphics tablet, in particular a display with a pointer.

[0155] In this case, at least one of the multidimensional representations, in particular at least one 2D representation or 3D representation, can be activated as a working representation by means of a graphical aid, wherein spatial coordinate relationships can be established between the individual representations.

[0156] The 2D diagram corresponding to the 3D diagram can also be processed with the help of a cursor. Here, the 2D diagram and / or the 2D layer can be operated, for example, with the help of a scroll wheel, a mouse or a similar computer-related pointing mechanism. The corresponding working position is transmitted to each other screen window simultaneously and in the corresponding spatial coordinates on the corresponding diagram. Therefore, the method according to the invention enables the precise orientation and transmission of the coordinates in the 2D diagram of the displayed section of the patient's body and the corresponding 3D diagram. The cursor indicates which area of ​​the 3D diagram and / or 2D diagram the planner / surgeon marks and / or plans and / or processes at the corresponding time.

[0157] For example, such a treatment measure in a 3D patient image can preferably also automatically reproduce its effect in a 2D patient image, preferably by establishing a coordinate relationship, and vice versa if the treatment measure is carried out in a 2D patient image.

[0158] The external device can be an image-aiding device, a display, an auxiliary device, a measuring device or preferably a mobile or stationary (surgical) navigation system. The navigation system is used in particular for controlling and monitoring the surgical operation, in particular for determining the local orientation and / or for determining the coordinates of the operation and for introducing and carrying out surgical measures. The planning data can be transmitted wirelessly or wired, for example, via a suitable protocol.

[0159] The interface may further be configured to provide information to a remotely arranged display device, such as a screen arranged in an operating room.

[0160] The interface can be designed in such a way that it can not only receive data from an external source or can only provide control data to the display facility. In the case of a suitably equipped interface, data can be permanently loaded from an external source, compared with stored data and provided to the surgeon via the display facility. In addition to the above-mentioned image data, possible data are also geometric data determined by external measurement facilities, for example data on the position, orientation or progress of the surgical operation.

[0161] In this context, it is particularly possible to establish an "electronic fence", for example for limiting the surgical field and / or for marking specific areas within the patient's body for the precise positioning of implants in multidimensional, preferably three-dimensional planning, and to implement the electronic fence to enclose and / or exclude these areas and for local coordination of the surgery and / or the implant, and to transmit the underlying coordinates wirelessly or by wire to a mobile or fixed surgical navigation system.

[0162] It is also possible to convert the data determined during the preoperative planning into a 3D model of the implant and to send these data via an interface to a machine connected thereto. For example, this can be a cutting machine that either manufactures a new implant or individually reprocesses an existing implant. Similarly, a 3D printer can be operated that either manufactures an adapted implant or generates a pattern that is not suitable as an implant, which is used for research purposes during the preoperative planning as needed, is suitable as a model for teaching purposes, or can also be a model of an implant that needs to be manufactured by molding.

[0163] Another possibility is to transmit the results of the preoperative planning to the navigation system via an interface, wherein the control data are either transmitted directly or the navigation system receives the control data from the acquired planning data or image data itself.

[0164] It is also possible that the image-assisted system not only supplies data to the connected device via its interface. Data from the external device can also be transmitted back to the image-assisted system via the interface. Thus, changes that occur during the operation can be determined by continuous comparison of target, planned and actual data, so that the operation being performed can be effectively monitored.

[0165] In addition to the automatic collection (detection) of data and automatic comparison (matching), an additional visual monitoring is provided, i.e. the part of the body to be operated on or the process within the body may not be recognizable to the surgeon.

[0166] If the image-aided facility or the interface is capable of carrying out or enabling monitoring, it can be ensured that data or commands are also transmitted to the connected external device via the interface. In particular when using navigation systems, this provides the possibility of drawing attention to deviations from the planned measures and planned results and thus avoiding irregularities in the performance of the operation. The type of information given may depend on the capabilities of the navigation device. For example, optical and / or acoustic information may be output.

[0167] The image-aiding device can output the image data to a monitor in the operating room via an interface. In cases where the surgeon must be aware of details or current changes in the preoperative planning, gesture control can be coupled to the image-aiding device via an interface. Thus, despite the hygiene requirements in the operating room, it is still possible to check in time whether the operation and its results meet the requirements of the planned results.

[0168] This planning result can be associated with patient image, especially 2D and / or 3D patient image through the interface of image-aided facility.In this case, one or more X-ray, CT, MRT or other pictures produced in imaging method of patient are generated on operating table.

[0169] Furthermore, the photographs can be calibrated using known methods. By means of an interface of the image-aided system, a correlation is established between the coordinates of the plurality of patient images so that the patient image on the operating table is aligned, oriented and represented in relation to the representation of the planning result in the image-aided system, in particular the navigation view. Figure 1 To.

[0170] The interface between the image-aiding system and the navigation system can also provide planning results and / or parts of planning results and / or at least patient images created by the patient on the operating table to an imaging device, such as an X-ray device, an MRT device or a CT device. In addition, the interface between the image-aiding system and the navigation system can provide, store and display graphical data and information for the image-aiding system and the navigation system.

[0171] According to the invention, planning results and / or parts of planning results and / or patient images generated by the patient on the operating table can be transmitted by wire or wirelessly to an image monitoring device, which virtually determines the position and orientation of the patient and the position, orientation and progress of the individual steps of the planned operation according to known methods.

[0172] Based on the calibration, the actual size and dimensions of the patient are stored with the aid of image aids and can be transferred to the navigation system via an interface.

[0173] For example, the instrument or implant is also stored according to the planning of the method with the aid of an image-aided device or can be accessed with the aid of an image-aided facility, for example by accessing already stored information or by imaging and calibrated photographs of the instrument or implant.

[0174] Image-assisted facilities and navigation systems allow automatic correlation of the patient's position, extension and location with the position, orientation and location of the implant or instrument.

[0175] Additionally, the instrument or implant can be connected to at least one positioning device, which can transmit the relative position of the instrument or implant as well as changes in the position, extension and location of the instrument or device to an image-assisted facility and a navigation system.

[0176] With the help of image aids and navigation systems, the position of all tools, implants and patients relative to each other can be displayed as they actually are in the room.

[0177] In particular, the planned surgery, the guide lines of the planned implants and the electronic fence are displayed by means of a navigation system or an image-assisted facility. Therefore, the surgeon can adapt each surgical step to the surgical step planned before the operation in real-time comparison. According to the present invention, the surgeon can especially adapt the orientation, position, entry angle, depth, positioning and execution of the steps planned before the operation to the planning results. Thus, the surgeon can prevent, for example, the surgical step from being performed too deeply and / or at a wrong angle or in a wrong position.

[0178] According to the present invention, there is also provided a device for performing a surgical operation based on a surgical operation planning result and / or a partial planning result obtained according to the present invention, and for performing a surgical operation planned according to the present invention.

[0179] The data and / or the representations fed into the image processing system are preferably processed and / or edited.

[0180] The data and / or illustrations fed into the image-assisted facility contain the types and sizes of implants of different manufactures. In addition, the image-assisted facility also includes implant variants available on the market that are manufactured by different manufacturers.

[0181] Figures and / or data:

[0182] Patient images are in particular X-rays, which are 2D images or pictures, and volume-based images, such as CT or MRT, which are 3D images or pictures.

[0183] Volume-based photographs, ie 3D photographs or images, usually consist of multiple layers of 2D images (so-called 2D layers). These volume-based photographs can be divided into individual 2D image layers.

[0184] Unlike 2D photos or images, volume-based images are already scaled when they are taken. This means that the size of the object being photographed and the scale of the photo are known. The corresponding data information used to scale the image is stored as a dataset together with the corresponding volume-based photo.

[0185] In contrast, 2D images usually have to be scaled by the planner / surgeon before planning in order to plan the surgical procedure in order to be able to determine the scale and possible distortions in the depth of the depicted section of the patient's body.

[0186] The planning data can be present in the form of 2D image data or can be obtained from the 2D image data, for example, by image observation or image measurement.

[0187] In addition to displaying images of the patient and possible implants, known computer-aided systems for preoperative planning can also make suggestions to the planner / surgeon regarding the selection of specific implants and associated instruments to be used during the surgical procedure. The same applies to similar implants or instruments.

[0188] However, in the known cases of using only 2D patient images or only 3D patient images for planning surgical operations, there are disadvantages such as displacements of joints, loading, unloading or deformations of bones and joints, in particular pathological deformations of the spine such as kyphosis, lordosis or due to different load distributions cannot be imaged in the recording system and therefore cannot be used for preoperative planning.

[0189] Thus, for example, a patient volume image is recorded with the patient lying down; in this case, the patient is usually recorded in the unloaded state.

[0190] In contrast, 2D patient images of the spine, such as x-rays, are usually recorded with the patient standing up, since the recording system can more easily display this position and the relevant body part can be reproduced in its actual loaded state.

[0191] However, especially when planning surgeries on the human or animal spine and its joints or bone parts, it must be possible to include different recording situations and positions of the patient, especially standing and lying situations, for example in order to determine the sagittal balance, especially because surgical operations are performed on lying patients.

[0192] Therefore, the information obtained from 2D and 3D patient images - in particular about the loaded and unloaded states of skeletal parts, bones, joints, vertebrae, tissue and soft tissue parts - must be made available to the surgical planner / surgeon in a simple manner so that he can preferably view and further process this information simultaneously.

[0193] Patient images are, for example, two-dimensional or three-dimensional photographs generated by an imaging method, which are obtained by known methods. Preferably, these photographs are 2D photographs from an X-ray application or 3D photographs from a CT or MRT application. When 2D and / or 3D patient images are mentioned below, in addition to the actual imaging, basic data sets are also to be understood.

[0194] Unless otherwise stated, the terms information and data are used synonymously below.

[0195] Patient images and patient information can be generated at different points in time, including 2D and / or 3D image data sets of the patient and / or other additional data, in particular data about the patient's height, weight, age, and including data from previous surgeries, data from previous examinations and / or data created directly in connection with the surgery.

[0196] In addition, in order to implement the method provided by the present invention, the image-assisted facility can access other information, such as patient images, dimensional data, functional data and / or other reference data, target values, standard values ​​and / or general patient-non-specific data and / or information about the implant, such as 2D and 3D models of the implant, manufacturer information of the implant, article number, size, material, instructions for use, etc., and / or sketches of the human or animal anatomical structure and standard values, ideal values ​​or target values.

[0197] Patient-specific and general information can be displayed, mapped and / or stored, in particular, using a plurality of multi-dimensional patient images.

[0198] The representations and associated data preferably include multi-dimensional images.

[0199] The multi-dimensional image is preferably a 2D representation or a 3D representation.

[0200] It can be understood that the above specific description of the present invention is only used to illustrate the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those skilled in the art should understand that the present invention can still be modified or replaced by equivalents to achieve the same technical effects; as long as the use requirements are met, they are within the protection scope of the present invention.

Claims

1. Intraoperative augmented reality system, characterized by: For use in assisting surgery on the human or animal body, including: a visualization device for combining natural perception with at least one artificially generated illustration based on an augmented reality system; To generate the artificial illustration, data from the memory is provided; calling the artificial diagram as a reference diagram for a specific area of ​​the body; wherein the artificial illustration displays a two-dimensional or multi-dimensional mapping; The artificial diagram includes one, several or all of the following items: at least one implant inserted, The endoprosthesis provided, Positioning of Kirschner wires, The dimensions carried out, Planning reports, Information about implants; wherein the artificial representation can be optically projected onto and / or next to the natural perception of the body area, and is called up by scanning fingerprints and / or iris recognition and by means of a QR code and / or a manually entered code, wherein the artificial representation may be broken down into its components and reassembled, and / or adapted in terms of transparency, size, direction and orientation, and otherwise processed, and The artificial representation enables the surgeon to access the 3D patient image in different views, parts and / or perspectives in sections or zooms, and Furthermore, positioning at one or more bone regions can indicate changes in the shape and / or size of the implant model, In this case, the dimensions or measurement results can be displayed simultaneously and proportionally to the spatial coordinates in one or more screen windows. In this case, by importing data, values ​​and information into the visualization device in the form of "augmented reality", the carrier of the visualization device perceives the data, values ​​and information it calls as "projections floating in the space in front of it", and when necessary, these projections can interact with the space. wherein a treating surgeon is able to create a text dictation and display the text dictation as text floating in space in front of a visualization device and / or play the text via audio playback, and This allows changes that occur during surgery to be identified by continuously comparing target, planned and actual data, so that the surgery being performed can be effectively monitored. wherein an optical or acoustic warning message is triggered in the event of a deviation of the optical projection of the artificial representation, In addition to displaying icons, it can also transmit images and videos, and In particular, images and videos can be supplemented with additional computer-generated information. enabling communication via interfaces between visualization devices at different locations, This allows synchronization to be achieved via multiple visualization devices in the space and thus access to the same and / or different scenes, ie holograms, Thus, multiple planners / surgeons can view one or more holograms in the same position. wherein the view of the hologram on the synchronized visualization device is oriented according to the respective viewing angle of the user, and This reproduces the real appearance and in a further step The hologram is modified by the planner / surgeon according to the processing function and adapted on all visualization devices by synchronization and in the final step Integration into preoperative planning data can be achieved by exporting the preoperative planning data into the corresponding application.

2. The intraoperative augmented reality system according to claim 1, characterized in that: The visualization means for combining the user's natural perception of the body region with at least one artificially generated representation is an “augmented reality” and / or “virtual reality” and / or “mixed reality” device.

3. The intraoperative augmented reality system according to claim 1, characterized in that: The reference diagrams include one, several or all of the following items: at least one implant inserted, The endoprosthesis provided, Used for positioning of Kirschner wires. The dimensions carried out, Planning reports, Information about implants.

4. The intraoperative augmented reality system according to claim 1, characterized in that: The storage is delivered by a connected Picture Archiving and Communication System (PACS) and / or by local storage of the hospital / clinic.

5. The intraoperative augmented reality system according to claim 1, characterized in that: The visualization device for combining a user's natural perception of a body region with at least one artificially generated representation comprises a computer unit.

6. The intraoperative augmented reality system according to claim 1, characterized in that: The visualization device for combining a user's natural perception of a body region with at least one artificially generated representation comprises a camera.

7. The intraoperative augmented reality system according to claim 1, characterized in that: The data for generating the artificial illustration are provided by an image-assisted facility.

8. The intraoperative augmented reality system according to claim 1, characterized in that: The artificially generated illustration is a scale.

9. The intraoperative augmented reality system according to claim 1, characterized in that: The artificially generated representation can predetermine a cutting line on the body region and / or a drilling line in the body region.

10. The intraoperative augmented reality system according to claim 1, characterized in that: The artificially generated representation is a two-dimensional or multi-dimensional image of the area of ​​the body to be treated.

11. The intraoperative augmented reality system according to claim 1, characterized in that: The artificially generated representation may display at least a two-dimensional image of at least one implant.

12. The intraoperative augmented reality system according to claim 1, characterized in that: The surgeon can adapt individual surgical steps to the preoperatively planned surgical steps in real-time comparison.

13. The intraoperative augmented reality system according to claim 1, characterized in that: The artificially generated representation predetermines an entry angle and / or an entry depth of the implant relative to the body region.

14. The intraoperative augmented reality system according to claim 1, characterized in that: With the aid of the artificially generated representation of the implant, a selection is made from at least two implants, wherein the implants differ in type, size and / or material.

15. The intraoperative augmented reality system according to claim 1, characterized in that: The reference representation reproduces a state of the body region which corresponds to the state of the region in a pre-operative phase or in a planned post-operative phase.

16. The intraoperative augmented reality system according to claim 1, characterized in that: It is also possible to project the artificial representation onto a body part, determine reference points on the body part and determine their coordinates, compare the reference points with reference points of the artificial representation and subsequently project the artificially generated representation onto the corresponding body part.

17. The intraoperative augmented reality system according to claim 1, characterized in that: The artificial representation is congruently projected onto the natural perception of the body region by means of at least one reference marker and / or is placed next to the body without projection for intraoperative comparison.

18. The intraoperative augmented reality system according to claim 1, characterized in that: The artificial representation shows the ratio of the identified markers to each other.

19. The intraoperative augmented reality system according to claim 1, characterized in that: If the optical projection of the artificial representation deviates from the user's natural perception of the body region, the visualization device triggers an optical or acoustic warning message.

20. The intraoperative augmented reality system according to any one of claims 1 to 19, characterized in that: The visualization device has its use for assisting surgery on the human or animal body.

Citation Information

Patent Citations

  • Optimizing the performance of a procedure on a patient using a medical system

    DE102016218364A1

  • Mixed reality-aided education using virtual models or virtual representations for orthopedic surgical procedures

    CN112566578A

  • Surgical Systems and Methods for Facilitating Tissue Treatment

    US20200253666A1

  • Systems and methods for augmented reality based surgical navigation

    US20200375666A1