Integrated optical device and related diagnostic imaging method

Abstract

An integrated optical device comprising an optical instrument for generating a real image and an instrument for generating a virtual image, the optical instrument for generating the real image comprising, in order, an eyepiece, an optical body, a binocular body and a support arm, where the instrument for generating the virtual image comprises at least: a display (1) for projecting the virtual image; a lens (2) for adjusting a focus; a plurality of prisms (3) for inserting, guiding and mixing the virtual images; and a plurality of mirrors (4) and a first camera (5) for guiding and positioning the virtual image in order to superimpose the virtual image on the real image in the eyepiece of the optical instrument.

Description

[0001] The present invention relates to an integrated optical device and related method for creating augmented reality.

[0002] The present invention belongs to the technical field of machinery and technology existing in hospital and non-hospital operating rooms, in particular to the technical field of surgical microscopes.

[0003] Electronics and digital technologies have become an integral part of many fields and sectors, whether it is industry or mechanics or healthcare and / or a typical part of everyone's daily life.

[0004] The importance of interconnecting electronic tools and software, which has been recognized by the industry, represents the so-called Medicine 4.0, that is, the transfer of Industry 4.0 to the medical field.

[0005] The use of these digital technologies includes 3D printers and robotics to enable the rapid exchange of information not only between doctors and patients, but also between doctors in different departments and hospitals, with the main goal of increasingly improving patient-centered care.

[0006] In particular, the technology sector related to virtual reality is becoming quite important, and VR is being incorporated into many different applications, starting with video games.

[0007] In recent decades, the surgical world has increasingly turned to the use of vision systems that allow for magnification and improvement of the surgical field in order to better observe the anatomical structures in which the type of surgery requires it. This development has allowed for a reduction in iatrogenic injuries, postoperative complications, and healthcare costs.

[0008] New technologies for assisted surgery include the possibility of using virtual reality and augmented reality as additional visual aids to improve surgery, for example by using a virtual helmet or mask.

[0009] The possibility to integrate images from the computer with images from the surgical field opens up new research frontiers in surgery.

[0010] Imaging has also undergone a profound transformation by making the transition from analog to digital and from two-dimensional to three-dimensional.

[0011] New technologies have supplemented traditional optical instruments to implement and improve, for example, radiological images, thereby increasing information useful for surgical planning and different case needs.

[0012] Specifically, in the field of maxillofacial surgery, the prior art basically includes two different techniques.

[0013] In more detail, these technologies include head-mounted systems (HMS) and camera-display based systems.

[0014] The second technique aims to achieve a better correlation of the virtual image with the real image, since, unlike the HMS, the camera is not on the surgeon's head, but is placed elsewhere in the operating room. However, this better correlation comes at the expense of increased virtual reality, since the camera no longer has the same perspective as the surgeon, but rather that of the surgical light, for example.

[0015] Another disadvantage is that the surgeon will see the augmented reality content projected on a screen instead of looking directly at the patient.

[0016] The most recent technology that is an evolution of HMS is the Video and Optical See-Through Augmented Reality Surgery System (VOSTARS).

[0017] This technology has improved functionality compared to HMS, but has some key issues due to system stability and image projection.

[0018] In this case, the system is integrated into a viewer worn by the surgeon so it can move according to the surgeon's movements, which can be frequent and unstable during long or demanding surgeries due to fatigue or other factors.

[0019] As for image projection, HMS and VOSTARS project onto a screen also worn by the surgeon, rather than directly onto the viewer or lens itself, causing additional discomfort and inconvenience to the surgeon.

[0020] Therefore, an object of the present invention is to provide an integrated optical device and related methods that can solve the above-mentioned inconveniences and key issues.

[0021] In particular, it is an object of the present invention to provide an integrated optical device and a related method capable of combining the advantages of using a microscope with the typical advantages of augmented reality.

[0022] It is another object of the present invention to provide integrated optical devices and related methods capable of introducing new and enhanced levels of virtual reality into surgical microscopes.

[0023] It is another object of the present invention to provide an integrated optical device and related methods that enable an operator located anywhere in the world to connect remotely, thereby enabling rapid and efficient consultation with an operator in the operating room.

[0024] Another object of the present invention is to provide integrated optical devices and related methods having excellent stability, which is necessary for cameras to achieve good image correlation.

[0025] It is another object of the present invention to provide an integrated optical device and associated methods that are capable of projecting images from two displays directly into the surgeon's eyes, allowing high resolution details present in a CT scan or X-ray to be superimposed on the surgical field.

[0026] It is another object of the present invention to achieve integrated optical devices and related methods that can be integrated with existing systems and utilize and enhance their capabilities.

[0027] Another object of the invention is to provide integrated optical devices and related methods that enable the operator to navigate anatomical parts that are not visible but are always superimposed on the real anatomical parts, thereby increasing his visual perception or observing the whole in a part of the field of view.

[0028] Another object of the present invention is to provide integrated optical devices and related methods for simulating surgery to test its difficulty, using the integrated digital vision capabilities of augmented reality and virtual reality for practice or research.

[0029] Finally, it is a final object of the present invention to provide an integrated optical device and related methods that allow even an external operator far from the operating room to digitally insert himself into the operating field by providing his own contribution simply using a cursor or specific tools.

[0030] These and other objects are achieved by an integrated optical device and related methods according to the appended independent claims.

[0031] Further detailed features are given in the attached dependent claims.

[0032] The present invention will now be described in an illustrative and non-limiting manner according to some preferred implementation forms of the present embodiment and with the aid of the accompanying drawings, in which:

[0033] - Figure 1 An integrated optical device for creating augmented reality according to the present invention is shown;

[0034] - Figure 2 The optical path of the camera used to track the operator's pupil is shown;

[0035] - Figure 3A and 3B are two details of the internal components of the integrated optical device according to the invention;

[0036] - Figures 4A-4D is a representation of a galvanometer included in an integrated optical device according to the present invention for projecting a laser beam;

[0037] - Figure 5 shows the optical path from the display to the surgeon or operator's eye within an integrated optical device according to the present invention;

[0038] - Figure 6 An assembly of lenses, prisms and supports included in a display and an integrated optical device according to the present invention is shown;

[0039] - Figure 7 shows the optical path of the integrated optical device according to the invention from the surgical field to two cameras for 3D reconstruction of objects in the field;

[0040] - Figure 8 is an example of what the camera sees (rectangular surgical field) and what the surgeon or operator sees (full black circle) when the surgeon or operator uses the integrated optical device according to the present invention;

[0041] - Fig. 9A and 9B are two examples of possible views that the surgeon or operator may see, in which the X-ray is placed laterally ( Fig. 9A ) or zoom in to the center of the field of view for better visualization ( Fig. 9B );

[0042] - Fig. 10A , 10B and 10C shows the projection of patient X-rays in a negative focal plane below the surgical field, a positive focal plane visible above the surgical field, and a focal plane obstructed by the surgical field;

[0043] - Fig.11 shows light reflection performed by an instrument including a beam splitter;

[0044] - Fig. 12A and 12B represents an example of mixed reality, where a 3D model ( Fig. 12B ) by placing it on the dental element ( Fig. 12A ) is identified and correlated with a previously performed CT scan in which the mandibular nerve and dental elements are visible. Specifically, Fig. 12A represents a possible example of a view that a surgeon or operator may see, wherein a 3D TC reconstruction is superimposed on the patient's anatomy represented by a 3D model;

[0045] - Fig.13 Possible remote actions via a smartphone on a surgical field visible from an integrated optical device and the associated method according to the invention are shown.

[0046] With reference to the above-mentioned drawings, there is shown in the same drawings preferred forms of embodiment of an integrated optical device and associated method according to the present invention.

[0047] With reference to the above-mentioned figures, the integrated optical device object of the invention is indicated by reference numeral 100 and, in a first embodiment, comprises, according to the invention, an instrument for creating and viewing augmented reality integrated with a traditional optical instrument belonging to the category of surgical microscopes.

[0048] Exemplarily, the optical instrument used may be a surgical stereomicroscope commonly used in dental microsurgery.

[0049] Structurally, a stereo microscope is mainly composed of an eyepiece, an optical body, a binocular body, and a mechanical support arm.

[0050] The surgeon or operator places his eye on the eyepiece, which allows the same surgeon or operator to see the area where the optical instrument is oriented and pointed.

[0051] The integrated instrumentation of device 100 includes multiple elements and techniques to enable a surgeon or operator to utilize additional images and additional tools to facilitate manipulation of a patient or other delicate or precise actions.

[0052] To achieve augmented reality, digital or virtual images must be superimposed on real images ( Figures 9A-9B and Figures 10A-10C ). The real image comes from known optical equipment, such as an optical or electron microscope commonly used in medical and surgical environments, or, as mentioned earlier, from a stereo microscope.

[0053] In particular, the real image ( Figure 8 ) comes from the objective of a microscope or stereomicroscope which must be constantly illuminated. Therefore, the microscope must be equipped with a coaxial illumination system (not shown) that illuminates the surgical field.

[0054] On the other hand, the digital image comes from some screen or display 1, which projects the image by itself as a light source and color. The fusion of the real image and the digital image is done by a special semi-reflective prism 3.

[0055] In particular, the device 100 comprises a series of displays 1 designed to project images that will reach directly into the eyepiece of the optical instrument so that the surgeon or operator does not have to move his or her eyes to obtain the required information.

[0056] For each display 1 , inside the device 100 , there is a first holder for supporting a lens 2 which projects an image through some prisms 3 contained in a second holder.

[0057] In particular, the lens 2 is intended to adjust the focus of the image and allow the same image to reach the prism 3 made of optical glass.

[0058] The above-mentioned prism 3 is an element necessary for the correct direction of the image from the display 1 or other image source.

[0059] The image from the surgical field passes through the prism 3 and reaches the mirror 4, then is directed to the lateral prism 3 and finally collected by the camera 5.

[0060] Advantageously, a number of auxiliary accessories may be inserted between the optical body of the optical instrument and the binocular body, such as,

[0061] – additional eyepiece for a second observer or operator,

[0062] – Camcorder or camera,

[0063] -Surgical lasers.

[0064] The magnification of a stereo microscope is variable, usually ranging from 4x to 20x.

[0065] An illumination system is necessary and may be already included in the optical body or present as an external infrared (laser) with an illuminator, preferably connected by an optical fiber. Furthermore, such an illumination system is substantially coaxial with the optical path.

[0066] Advantageously, the stereomicroscope has two optical paths parallel to each other; a first one for the right eye and a second one for the left eye.

[0067] In particular, although there are two optical paths, the objective lens remains single. In this way, unlike other technologies used in this field, the two optical paths have a single focal plane, making image viewing better and more efficient and allowing simultaneous visual acquisition of both eyes, even in particularly deep and narrow cavities, while still maintaining excellent depth perception during delicate surgical procedures.

[0068] The distance of the optical path is programmed to be proportional to the stereo effect, which is strongly affected by the magnification factor.

[0069] Advantageously, the displays 1 can be of any type, preferably they are of AMOLED type, since in this technology each pixel can emit light and change color independently of the others. This feature results in a greater black depth and a very high contrast.

[0070] Alternatively, for example, an LCD display or a DLP projector may also be used.

[0071] The image is projected into the optical path through the eyepiece and then onto the eyes of the surgeon or operator without the latter moving their eyes away from the surgical field.

[0072] Still advantageously, the display 1 can insert images from an external source (such as a computer or diagnostic equipment) through a special prism 3 in the optical glass, as well as the graphic interface of the selected medical device.

[0073] This allows the surgeon to review the required data without having to take his eyes away from the surgical field, thus avoiding distraction and wasted time. Figures 9A-9B and Fig. 12A ).

[0074] When the system is working, the two cameras 5 capture images of the surgical field in real time from the various optical paths, intercepting them along the paths.

[0075] Specific software will then analyze the images from both optical paths to perform real-time three-dimensional reconstruction of structures and objects in the surgical field by identifying them and correlating them with structures and objects contained in previous data or images collected in a database.

[0076] In detail, such data or images in the database may consist of two-dimensional or three-dimensional radiological information, anatomical scans, mathematical models of surgical instruments, etc.

[0077] Advantageously, an additional camera allows other operators or collaborators to see a real-time image of the surgical procedure on a monitor, or it may allow another operator acting as a teacher or expert to interact with them using a pointer 31 visible within the microscope and moved via a computer or smartphone 32 .

[0078] In detail, with this additional camera, more operators can be involved even in geographically remote locations.

[0079] Another advantage is that a link can be established between the real image and additional images previously acquired and processed during surgical planning, making it possible to interact with them.

[0080] In particular, it is possible to interact with real and virtual images simultaneously in order to facilitate the surgeon's operation or provide him with more information in real time.

[0081] To do this requires that the image seen through the microscope and the previously acquired image have a relevant number of fixed or undeformable points in common.

[0082] This common fixed point is necessary for the correct positioning and association of the real image with the virtual image.

[0083] For example, in addition to fixed points, object targets or planar figures recognized by the software by comparison with data in an archive or database may also be used.

[0084] Alternatively, fixed points may also be derived from a 3D model 21 generated using specific hardware and software instrumentation.

[0085] The acquired images may be obtained from, for example, X-ray, MRI or other diagnostic image acquisition methods.

[0086] Through the matching process, the software will be able to recognize the image and match it with the real image obtained from the optical instrument used, thus sending the desired result directly to the eyepiece ( Fig. 12A ).

[0087] In detail, the task of this process is to match the fixed points of the real image with the corresponding fixed points of the virtual image.

[0088] Advantageously, the operator or surgeon is able to see the real image and superimpose a virtual image which adapts with a delay of a few milliseconds to the movements of the surgical field or the microscope itself.

[0089] This delay depends on the performance of the video card used and inserted in the device 100 .

[0090] Further advantageously, the operator or surgeon will be able to see areas or anatomical features that are not visible due to overlap with instruments or other anatomical parts of the patient.

[0091] In particular, surgeons will also be able to obtain real-time internal images of the anatomical area being treated externally.

[0092] For example, a surgeon who is treating the outside of a tooth will also be able to simultaneously see the location of a nerve located inside the dental arch in the overlay.

[0093] Optionally, the virtual image can be reproduced in an area outside the surgical field, but remain within the eyepiece and do not interfere with or obstruct the surgeon's or operator's view ( Fig. 9A ).

[0094] The previously acquired image is also projected using a specific type of instrument comprising a prism 3 or a beam splitter 10 .

[0095] In particular, these instruments or beam splitters 10 comprise, in addition to a prism 3, mirrors 4 for forming the optical path and complex control electronics.

[0096] The beam splitter 10 is advantageously formed by two prisms 3 with a triangular base, which allow a single light beam to be split into two different beams, a transmitted beam and a reflected beam, wherein the percentage of incident light is predefined to correspond to the light flux coming from the display 1 and the surgical field ( Fig.11 As an illustrative but not binding example, a beam splitter with a transmission and reflection ratio of 30 / 70% may be used.

[0097] Through these special prisms 3, images are passed, mixed, crossed, and sometimes even flow in opposite directions, and these images are assigned to four shooting sources, such as:

[0098] - sources from both displays 1,

[0099] - a source from an illuminator, preferably infrared (laser),

[0100] - Sources from the surgical area,

[0101] -Source from the observer's eye.

[0102] Advantageously, the instrument (beam splitter) 10 is designed to maintain the necessary compactness so as not to reduce the flexibility of the stereomicroscope or optical instrument on which the instrument is to be mounted.

[0103] Advantageously, the multi-stage structure can be designed so that the beam splitter 10 and the mirror 4 are placed on the first stage, intercepting the image from the main optical path to be shared with two cameras 5 (left and right) for photogrammetry and with a third camera for use in a shared operating room monitor and optionally in a monitor outside the operating room.

[0104] The same two beam splitters 10 and mirrors 4 send the additional image from the display 1 to the two eyepieces, mixing the additional image with the real image from the main optical channel.

[0105] On a second stage, another beam splitter 10 intercepts the pupil image of the operator's right eye from the eyepiece to deliver it to a fourth camera capable of detecting its movement.

[0106] In addition to controlling depth of field, the dual aperture placed on the lower attachment plate also allows dimming or brightness enhancement of the real image.

[0107] Advantageously, two 500nm blue filters are located in front of the camera for photogrammetry in order to optimize its target shooting.

[0108] It is also advantageous to provide a cooling fan for the heat generated by the display 1 and the control electronics.

[0109] Additionally, the volume of data flowing in and out of the facility necessitated the installation of fiber optics to enable rapid data and image transfer.

[0110] Advantageously, a foot pedal or cursor or other instrument is connected to the device 100 which can be operated by the foot or by technology that does not distract the surgeon or operator from the surgical field, and which has the functionality of a cursor to allow the surgeon to interact with additional functions in the eyepiece desktop and integrated optical device without distracting the view from the surgical field.

[0111] Advantageously, all image creation, correct positioning and management of the images are delegated to software and libraries appropriately programmed for the correct operation of the device 100 .

[0112] For example, the library can be used for object recognition in industrial robots, for augmented reality in video games, or for virtual simulations of indoor or architectural spaces.

[0113] We can still use an example to illustrate this point, Microscope and Microscope Software has been implemented to create the graphical environment and properly interconnect users and devices.

[0114] As mentioned previously, advantageously, the introduction of augmented reality within the microscope provides an extraordinary tool for two-way communication by opening up new roles and opportunities in the world of surgery. In particular, the assistant will be able to follow events on the surgical field more directly by interacting with it, not only by positioning instruments, but also by practicing by selecting and inserting the most appropriate visual input required by the surgeon or operator.

[0115] In particular, additional surgeons, operators or assistants will be able to provide assistance in real time via a fixed or mobile device 32 (such as a PC or smartphone).

[0116] An additional laser galvanometer device consisting of two mirrors 4 at an angle to each other inserted into the optical system allows the laser beam to be projected as a pointer and surgical laser ( Figures 4A-4D ).

[0117] A prism 3 and specific filters are inserted into the optical path of the microscope to deflect the laser beam toward the surgical field and prevent it from being deflected toward the operator's eyepieces.

[0118] Through the Internal Microscope Application and Remote Microscope Applications include guiding laser sources for pointing and drawing (laser show) and also for surgical applications.

[0119] The laser source projection path may also be placed outside the optical system of the microscope, for example, below the microscope eyepiece itself.

[0120] From the description made, the characteristics and advantages of the integrated optical device and method thereof which are the object of the present invention are clear.

[0121] Finally, as long as the components used are compatible with the specific use and dimensions, they may be any according to requirements and the state of the art.

[0122] If features and techniques mentioned in any claim are followed by reference signs, the sole purpose of adding these reference signs is to improve the intelligibility of the claims and therefore these reference signs have no limiting effect on the interpretation of each element identified by way of example with these reference signs.

Claims

1. An integrated optical device, comprising an optical instrument for generating a real image and an instrument for generating a virtual image, wherein the optical instrument for generating a real image comprises, in sequence, an eyepiece, an optical body, a binocular body, and a support arm usable by a first operator, and the instrument for generating a virtual image comprises at least: A display (1) for projecting the virtual image; a lens (2) for adjusting the focus; A plurality of prisms (3) for inserting, directing and blending the virtual images and a plurality of mirrors (4) and a first camera (5) for directing and positioning the virtual images so as to superimpose the virtual images on the real images in the eyepiece of the optical instrument, the integrated optical device being characterized in that it comprises a pointer (31) which is visible in the eyepiece of the optical instrument of the first operator and which is moved by means of a computer or smartphone (32) which can be used by other operators, enabling the other operators to interact in real time with the superimposed virtual and real images.

2. The integrated optical device according to claim 1, characterized in that: include: A plurality of beam splitters (10) for projecting, transmitting and mixing the virtual image and the real image, each of the beam splitters (10) comprising two prisms (3) with a triangular base positioned in contact so as to separate a light beam into a transmitted light beam and a reflected light beam, control electronics and a series of mirrors (4) for forming an optical path.

3. The integrated optical device according to claim 2, characterized in that: The beam splitter (10) is positioned in a multi-stage structure, wherein: - positioning the beam splitter (10) and the first mirror (4) in a first region of the first stage to intercept the virtual image from the primary optical path and to share the virtual image with the first camera (5) and a third camera for a shared monitor; - in the second region of the first stage, the virtual image from the display (1) is mixed with the real image of the primary optical channel by the beam splitter (10) and the second reflector (4) and then sent to the eyepiece; - in the second stage, another beam splitter (10) is positioned to intercept the image of the right eye pupil coming from the eyepiece and deliver it to a fourth camera capable of detecting its movement; 4. The integrated optical device according to claim 1, characterized in that: A dual aperture is included to allow depth of field and dimming or brightness enhancement of the real image.

5. Integrated optical device according to at least one of the preceding claims, characterized in that A foot pedal or cursor or other instrument operable by the first operator is included to allow the first operator to interact with the integrated optical device without distracting from the surgical field.

6. Integrated optical device according to at least one of the preceding claims, characterized in that A laser galvanometer device comprising a projected laser beam, said laser beam being deflected towards said surgical field by an optical system comprising two mirrors (4) of said set of mirrors (4) which are angled with respect to each other, said prism (3) and a specific filter which prevents said laser beam from being deflected towards said eyepiece of said first operator.

7. The integrated optical device according to claim 6, characterized in that: The laser beam is transmitted through a microscope such as Internal software applications and through The remote software application is guided to point and draw and perform the surgical application.

8. Integrated optical device according to at least one of the preceding claims, characterized in that It comprises two blue filters located in front of the first camera (5).

9. An imaging diagnostic method implemented by the integrated optical device according to claim 1, characterized in that: Includes the following steps which are programmed, executed and managed by application software: - Acquiring the virtual image or the corresponding 3D model; - generating fixed points on the virtual image or the 3D model; - generating fixed points on the real image; - Matching is performed between the fixed points of the virtual image and the fixed points of the real image.

Images