Image processing device, image processing method, and surgical microscope system

Abstract

The image processing device (13) according to an embodiment of the present disclosure comprises: an image input unit (13b) for receiving an operation field image for a patient's eye; an eyeball tracking unit (13e) for tracking the eyeball in the operation field image; and a display image generator (13f) for setting a plurality of regions having different display modes for the operation field image, and generating a display image in which the boundary between the plurality of regions indicates at least one of a specific position, a specific direction, or a specific size with respect to the eye. On the basis of the result of tracking the eyeball, the display image generator (13f) changes the display mode for any or all of the plurality of regions, and changes at least one of the position, the direction, or the size of the boundary.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and a surgical microscope system.

Background Art

[0002] As a method of refractive correction in ophthalmology, an artificial lens called an intraocular lens (IOL) is widely inserted into the eye to eliminate refractive abnormalities such as the crystalline lens and improve visual functions such as visual acuity. As an intraocular lens, an intraocular lens inserted into the capsular bag is most widely used as a substitute for the crystalline lens removed by cataract surgery. In addition to inside the capsular bag, there are various intraocular lenses, such as those fixed (retained) in the ciliary sulcus (Phakic IOL).

[0003] When performing ophthalmic surgery such as cataract surgery, in order to improve the postoperative visual function, the surgeon performs the surgery so that the incision position, incision shape, and the posture of the implant such as the intraocular lens to be inserted with respect to the eye are appropriate in light of the preoperative plan. At this time, it is desired to present information regarding an appropriate incision position, incision shape, implant posture, etc. in a form that facilitates the surgery for the surgeon. Therefore, in Patent Document 1, a technique for changing the position of a mark (pattern) indicating a preoperative plan according to the result of eye tracking has been proposed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, because the marks based on the preoperative plan are superimposed on the surgical field image, occlusion occurs, where parts of the surgical field image become obscured by the marks. As a result, it becomes difficult for the surgeon to see the surgical field image, making it challenging to perform the surgery precisely according to the preoperative plan.

[0006] Therefore, this disclosure proposes an image processing device, an image processing method, and a surgical microscope system that can accurately perform surgery in accordance with the preoperative plan. [Means for solving the problem]

[0007] An image processing apparatus according to an embodiment of the present disclosure includes an image input unit for receiving a surgical field image of a patient's eye, an eyeball tracking unit for tracking the eyeball in the surgical field image, and a display image generation unit for setting a plurality of regions with different display modes relative to the surgical field image and generating a display image in which the boundaries of the plurality of regions indicate at least one of a specific position, a specific direction, and a specific size relative to the eye, wherein the display image generation unit changes the display mode of any or all of the plurality of regions and changes at least one of the position, direction, and size of the boundary.

[0008] An image processing method according to an embodiment of the present disclosure includes an image processing device receiving a surgical field image of a patient's eye, tracking the eyeball in the surgical field image, setting a plurality of regions with different display modes relative to the surgical field image, and generating a display image in which the boundaries of the plurality of regions show at least one of a specific position, a specific direction and a specific size relative to the eye, wherein the image processing device changes the display mode of any or all of the plurality of regions and changes at least one of the position, direction and size of the boundaries.

[0009] A surgical microscope system according to an embodiment of the present disclosure comprises a surgical microscope for obtaining a surgical field image of a patient's eye, an image processing device for generating a display image, and a display device for displaying the display image, wherein the image processing device comprises an image input unit for receiving the surgical field image, an eye tracking unit for tracking the eyeball in the surgical field image, and a display image generation unit for setting a plurality of regions with different display modes relative to the surgical field image and generating a display image in which the boundaries of the plurality of regions indicate at least one of a specific position, a specific direction, and a specific size relative to the eye, wherein the display image generation unit changes the display mode of any or all of the plurality of regions and changes at least one of the position, direction, and size of the boundaries. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of a schematic configuration of a surgical microscope system according to the embodiments of this disclosure. [Figure 2] This figure shows an example of a schematic configuration of a surgical microscope according to the embodiment of this disclosure. [Figure 3] This figure shows an example of a schematic configuration of an image processing apparatus according to an embodiment of the present disclosure. [Figure 4] This figure shows an example of a display image according to the embodiment of this disclosure. [Figure 5] This is a first figure illustrating the display image generation according to the embodiment of this disclosure. [Figure 6] This is a second figure illustrating the display image generation according to the embodiment of this disclosure. [Figure 7] This figure shows an example of a display image 2 according to the embodiment of this disclosure. [Figure 8] This figure shows an example of a display image 3 according to the embodiment of this disclosure. [Figure 9] This figure shows an example of a display image according to the embodiments of this disclosure. [Figure 10] This figure shows an example of a display image 5 according to the embodiment of this disclosure. [Figure 11] This figure shows an example 6 of a display image according to the embodiment of this disclosure. [Figure 12] FIG. 7 showing an example of a display image according to an embodiment of the present disclosure. [Figure 13] FIG. 8 showing an example of a display image according to an embodiment of the present disclosure. [Figure 14] FIG. 9 showing an example of a display image according to an embodiment of the present disclosure. [Figure 15] FIG. 10 showing an example of a display image according to an embodiment of the present disclosure. [Figure 16] FIG. 11 showing an example of a display image according to an embodiment of the present disclosure. [Figure 17] FIG. explaining the change in boundary presentation according to the tracking status in an embodiment of the present disclosure. [Figure 18] The first figure for explaining the change in boundary presentation according to time in an embodiment of the present disclosure. [Figure 19] The second figure for explaining the change in boundary presentation according to time in an embodiment of the present disclosure. [Figure 20] The third figure for explaining the change in boundary presentation according to time in an embodiment of the present disclosure. [Figure 21] FIG. explaining the change in boundary presentation according to the tracking result in an embodiment of the present disclosure. [Figure 22] FIG. 12 showing an example of a display image according to an embodiment of the present disclosure. [Figure 23] FIG. 13 showing an example of a display image according to an embodiment of the present disclosure. [Figure 24] FIG. 14 showing an example of a display image according to an embodiment of the present disclosure. [Figure 25] The fourth figure for explaining the change in boundary presentation according to time in an embodiment of the present disclosure. [Figure 26] FIG. 15 showing an example of a display image according to an embodiment of the present disclosure. [Figure 27] The first figure showing Example 16 of the display image according to an embodiment of the present disclosure. [Figure 28] The second figure showing Example 16 of the display image according to an embodiment of the present disclosure. [Figure 29]This figure shows an example 17 of a display image according to the embodiment of this disclosure. [Figure 30] This figure shows an example of a schematic configuration of a computer according to the embodiments of this disclosure. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described in detail below with reference to the drawings. Note that these embodiments do not limit the apparatus, methods, systems, etc., related to this disclosure. Furthermore, in the following embodiments, the same reference numerals are used for essentially the same parts to avoid redundant explanations.

[0012] The one or more embodiments (including examples and modifications) described below can each be implemented independently. On the other hand, at least some of the embodiments described below may be implemented in appropriate combination with at least some of the other embodiments. These embodiments may contain novel features that differ from each other. Therefore, these embodiments may contribute to solving different objectives or problems and may produce different effects.

[0013] This disclosure will be explained in the order of the items shown below. 1. Embodiment 1-1. An example of a schematic configuration of a surgical microscope system 1-2. An example of a schematic configuration of a surgical microscope 1-3. Outline configuration of an image processing device and an example of image processing. 1-4. Action and Effects 2. An example of a general computer configuration 3. Addendum

[0014] <1. Embodiments> <1-1. An example of a schematic configuration of a surgical microscope system> An example of the schematic configuration of the surgical microscope system 1 according to this embodiment will be described with reference to Figure 1. Figure 1 is a diagram showing an example of the schematic configuration of the surgical microscope system 1 according to this embodiment.

[0015] As shown in Figure 1, the surgical microscope system 1 comprises a surgical microscope 10 and a patient bed 20. This surgical microscope system 1 is used for eye surgery. The patient lies on the patient bed 20 while undergoing eye surgery. The surgeon, a physician, performs the surgery while observing the patient's eye with the surgical microscope 10.

[0016] The surgical microscope 10 includes an objective lens 11, an eyepiece lens 12, an image processing device 13, and a monitor 14.

[0017] The objective lens 11 and the eyepiece lens 12 are lenses used for magnified observation of the patient's eye during surgery.

[0018] The image processing device 13 outputs various images and various information by performing predetermined image processing on the image captured through the objective lens 11.

[0019] The monitor 14 displays images captured through the objective lens 11, as well as various images and information generated by the image processing device 13. This monitor 14 may be provided separately from the surgical microscope 10.

[0020] In this surgical microscope system 1, for example, the surgeon looks through the eyepiece 12 and performs surgery while observing the patient's eye through the objective lens 11. The surgeon also performs surgery while checking the surgical field image, various images (e.g., images before and after image processing), and various information displayed on the monitor 14. It is also possible to perform surgery using only the image on the monitor 14.

[0021] <1-2. An example of a general configuration of a surgical microscope> An example of the schematic configuration of the surgical microscope 10 according to this embodiment will be described with reference to Figure 2. Figure 2 is a diagram showing an example of the schematic configuration of the surgical microscope 10 according to this embodiment.

[0022] As shown in Figure 2, the surgical microscope 10 includes, in addition to the objective lens 11, eyepiece lens 12, image processing device 13, and monitor 14, a light source 51, an observation optical system 52, a front image acquisition unit 53, a tomographic image acquisition unit 54, a display unit 55, an interface unit 56, and a speaker 57. The monitor 14 and the display unit 55 correspond to display devices.

[0023] The light source 51 emits illumination light according to the control of the control unit 13A of the image processing device 13, illuminating the patient's eyes.

[0024] The observation optical system 52 is composed of optical elements such as an objective lens 11, a half mirror 52a, and lenses (not shown). This observation optical system 52 guides the light reflected from the patient's eye (observation light) to the eyepiece lens 12 and the front image capture unit 53.

[0025] More specifically, light reflected from the patient's eye enters the half mirror 52a as observation light via the objective lens 11 and other lenses (not shown). Approximately half of the observation light that enters the half mirror 52a passes directly through the half mirror 52a and enters the eyepiece lens 12 via the transmissive display unit 55. On the other hand, the remaining half of the observation light that enters the half mirror 52a is reflected by the half mirror 52a and enters the front image capture unit 53.

[0026] The frontal image acquisition unit 53 is composed of, for example, a video camera. This frontal image acquisition unit 53 receives observation light incident from the observation optical system 52 and converts it into photoelectric energy to capture a frontal image, which is an image of the patient's eye observed from the front, that is, an image of the patient's eye taken from approximately the direction of the eye axis. The frontal image acquisition unit 53 captures (images) a frontal image according to the control of the image processing device 13 and supplies the obtained frontal image to the image processing device 13.

[0027] The tomographic imaging unit 54 is composed of, for example, an optical coherence tomography (OCT) or a Scheinpulk camera. This tomographic imaging unit 54 captures (imports) a cross-sectional image of the patient's eye according to the control of the image processing device 13, and supplies the obtained tomographic image to the image processing device 13. Here, a tomographic image is a cross-sectional image of the patient's eye in a direction approximately parallel to the ocular axis.

[0028] Furthermore, the tomographic image acquisition unit 54 acquires tomographic images using the interference principle with respect to infrared light, and the optical path of the infrared light and a portion of the optical path of the observation light in the observation optical system 52 may be made a common optical path.

[0029] The eyepiece 12 focuses the observation light incident from the observation optical system 52 via the display section 55 to form an optical image of the patient's eye. As a result, the optical image of the patient's eye is observed by the operator looking through the eyepiece 12.

[0030] The display unit 55 is composed of a transmissive or opaque display device and is positioned between the eyepiece 12 and the observation optical system 52. This display unit 55 transmits observation light incident from the observation optical system 52 to the eyepiece 12 and also displays various images (e.g., frontal images, tomographic images, etc.) and various information supplied from the image processing device 13 as needed. The various images and information may be displayed superimposed on the optical image of the patient's eye, or they may be displayed at the periphery of the optical image so as not to interfere with the optical image. Furthermore, the transmissive and opaque display devices are configured to be switchable and can be switched as needed. For example, there are transmissive and opaque modes, which can be changed by the operator, etc., to switch between the transmissive and opaque display devices.

[0031] The image processing device 13 has a control unit 13A that controls the operation of the entire surgical microscope 10. For example, the control unit 13A changes the illumination conditions of the light source 51 or changes the zoom magnification of the observation optical system 52. The control unit 13A also controls the acquisition of images from the frontal image acquisition unit 53 and the tomographic image acquisition unit 54 based on operation information from the surgeon, etc., supplied from the interface unit 56.

[0032] The interface unit 56 is composed of, for example, a communication unit. The communication unit receives commands from operation units such as a touch panel superimposed on the monitor 14, a foot switch, a controller, and a remote controller, and communicates with external devices. This interface unit 56 supplies information such as the operator's actions to the image processing device 13. The interface unit 56 also outputs device control information such as the information supplied from the image processing device 13 to the external device for controlling the external device.

[0033] The monitor 14 displays various images, such as a front view image, and various information on its display screen in accordance with the control unit 13A of the image processing device 13.

[0034] The speaker 57, in response to control by the control unit 13A of the image processing device 13, outputs sounds such as a buzzer or melody, or a message (voice), to notify the surgeon or others of a dangerous situation, for example, if a dangerous situation is detected during surgery. The surgical microscope 10 may also be equipped with a rotating light or indicator light (lamp) to notify the surgeon or others of a dangerous situation.

[0035] In the surgical microscope system 1 configured as described above, by using a display screen that presents one or both of a specific position and a specific size (a specific position and a specific size relative to the eye) based on the preoperative plan, occlusion is prevented, the surgeon can easily view the surgical field image, and the aforementioned specific position or size can also be grasped, enabling highly accurate ophthalmic surgery in accordance with the preoperative plan. The differences in display modes are differences in parameters related to display conditions, such as differences in brightness, saturation, color temperature, color, contrast, and sharpness.

[0036] <1-3. Outline configuration of an image processing device and an example of image processing> The schematic configuration of the image processing apparatus 13 according to the embodiment and an example of image processing will be described with reference to Figure 3. Figure 3 is a diagram showing an example of the schematic configuration (configuration and processing flow) of the image processing apparatus 13 according to the embodiment.

[0037] As shown in Figure 3, the image processing device 13 includes a pre-operative planning receiving unit 13a, an image input unit 13b, a registration unit 13c, an information storage unit 13d, an eyeball tracking unit (eyeball tracking unit) 13e, and a display image generation unit 13f.

[0038] The preoperative planning receiving unit 13a receives preoperative planning information for the patient's eye (e.g., preoperative images and posture information of the preoperative plan). The posture information of the preoperative plan includes information (size information, position information, orientation information, etc.) regarding the size of indicators (indicators related to the procedure) based on parts such as the limbus of the cornea in the preoperative image, the position of the indicators, and the orientation of the indicators around the axial length of the eye (position in the rotational direction around the axial length of the eye). For example, the size, position, and orientation of the indicators around the axial length of the eye may include the position, shape, and size of the incision, as well as the position and orientation of implants such as intraocular lenses to be inserted. The orientation around the axial length of the eye is defined by the angle in the rotational direction around the axial length of the eye with respect to a reference line perpendicular to the axial length of the eye. However, both the position in the coordinate system of the indicator and the position in the rotational direction around the axial length of the eye correspond to the position information of the indicator (position information of a specific position).

[0039] The image input unit 13b receives a surgical field image (frontal image) from the frontal image acquisition unit 53 (see Figure 2) and supplies the received surgical field images (for example, the surgical field image at the start of surgery or the surgical field image in real time during surgery) to the registration unit 13c, the eye tracking unit 13e, the display image generation unit 13f, etc.

[0040] The registration unit 13c compares the preoperative image from the preoperative plan with the surgical field image at the start of the surgery to determine the correspondence between the preoperative image from the preoperative plan and the surgical field image at the start of the surgery, for example, the conversion parameters from the preoperative image to the surgical field image at the start of the surgery (for example, the conversion parameters for coordinate transformation). The registration unit 13c then supplies the relationship information regarding the obtained conversion parameters, along with the surgical field image at the start of the surgery, to the information storage unit 13d.

[0041] The information storage unit 13d converts (modifies) the preoperative plan posture information to match the preoperative plan posture information based on the relevant information supplied from the registration unit 13c and the preoperative field image at the start of the surgery, and stores the preoperative field image at the start of the surgery and the preoperative plan posture information converted to match that preoperative field image.

[0042] The eyeball tracking unit 13e tracks the eyeball in the real-time surgical field image by comparing the surgical field image at the start of the surgery with the real-time surgical field image. The eyeball tracking unit 13e also supplies relationship information (e.g., conversion parameters, etc.) showing the relationship between the eyeball's posture information in the real-time surgical field image and the preoperative plan posture information stored by the information storage unit 13d to the display image generation unit 13f as the tracking result (tracking result). The eyeball's posture information, like the preoperative plan posture information, includes information (size information, position information, orientation information, etc.) regarding the size of the eyeball, the position of the eyeball, and the orientation of the eyeball around the axial length (position in the rotational direction around the axial length). However, both the position in the coordinate system of the eyeball and the position in the rotational direction around the axial length correspond to the eyeball's position information.

[0043] The display image generation unit 13f processes the real-time surgical field image to set multiple regions with different display characteristics relative to the real-time surgical field image, and generates a display image in which the boundary of each region indicates a specific position or size relative to the patient's eye. At this time, the display image generation unit 13f processes the real-time surgical field image, i.e., each region, based on the converted preoperative plan posture information, so that the boundary of each region indicates a specific position or size. Furthermore, the display image generation unit 13f processes the real-time surgical field image to change the posture (position, orientation, size, etc.) of the boundary of each region based on the eyeball tracking results of the real-time surgical field image, and generates a display image. At this time, the display image generation unit 13f processes the position and size of the boundary in the real-time surgical field image so that the relationship between the position and size of the boundary relative to the eyeball in the surgical field image at the start of the surgery does not change, based on the relationship information supplied from the eyeball tracking unit 13e, and generates a display image.

[0044] (Example image 1) An example of a display image according to the embodiment will be described. Figure 4 is a diagram showing an example of a display image according to the embodiment.

[0045] As shown in Figure 4, the display image presents a boundary K1 between two regions (left and right regions) with different display modes. This boundary K1 indicates a specific location based on the preoperative plan or other plans, i.e., a specific location related to the procedure. Depending on the direction and amount of eye movement, the boundary K1 is transformed (modified) to move by the aforementioned amount in the direction of eye movement so that there is no change in the posture of the boundary K1 relative to the eye. A display image having such a boundary K1 is displayed on the display screen by either or both the monitor 14 and the display unit 55. When the display unit 55 displays a display image having boundary K1, it switches the transparent display device to an opaque display device and uses the opaque display device.

[0046] In the example in Figure 4, boundary K1 is a line-shaped boundary that passes through the central position where the intraocular lens B1, such as a toric IOL for astigmatism correction, is to be placed. This boundary K1 indicates the boundary line for positioning the intraocular lens B1 (the target position for placement of the intraocular lens B1). Two marks B1a on the intraocular lens B1 (for example, three points arranged in a straight line) are aligned with this boundary K1. For example, if the intraocular lens B1 is a toric IOL, the toric axis of the intraocular lens B1 must be aligned with the patient's astigmatism axis, and if there is a misalignment in the orientation around the axial length of the eye (position in the rotational direction around the axial length of the eye), sufficient astigmatism correction cannot be obtained. Therefore, the toric IOL has two marks B1a engraved on its endpoints that indicate the toric axis, making it possible to understand the orientation of the toric IOL around the axial length of the eye. During surgery, the marks B1a on the toric IOL are aligned with boundary K1 in the real-time surgical field image, and the toric IOL is placed inside the eye.

[0047] In many ophthalmic surgical guidance systems, preoperative images and images taken at the start of surgery (surgical field images at the start of surgery) are registered, and then marks based on the preoperative plan are mapped onto the real-time image by comparing (tracking) the images taken at the start of surgery with the real-time image (surgical field images in real time). However, with this method, since the marks based on the preoperative plan are superimposed on the surgical field image, occlusion occurs, where parts of the surgical field image become invisible due to the marks.

[0048] Therefore, it becomes difficult for the surgeon to see the surgical field image, making it difficult to perform tasks such as aligning the axis of the toric IOL (intraocular lens for astigmatism correction), centering the IOL, creating the incision, and performing anterior capsulotomy, thus making it difficult to perform surgery accurately according to the preoperative plan. To address this, by indicating specific locations or sizes related to the procedure not with marks, but with boundaries of each region that have different display characteristics (e.g., boundary K1), it is possible to prevent occlusion and enable surgery to be performed accurately according to the preoperative plan.

[0049] (Specific processing flow) The specific processing is carried out according to the following flow: registration, tracking, image generation, and image presentation are performed sequentially. Figures 5 and 6 are diagrams illustrating the display image generation (conversion process) according to the embodiment.

[0050] During registration, preoperative planning information, including preoperative images (e.g., eyeball images), and an image taken at the start of surgery (surgical field image at the start of surgery) are received to align the reference points for the orientation and position of the eyeball during surgery with those during preoperative planning. For example, the alignment may be performed automatically by image processing using the preoperative and start-of-surgery images, or the user may manually adjust the position and orientation (e.g., rotation angle). Upon successful registration, surgical planning information based on the coordinates of the start-of-surgery image is retained.

[0051] In tracking, eye movements are tracked from the start of surgery, and transformation parameters (coordinate transformation parameters) are determined from the image at the start of surgery to the coordinates at the tracking time. For example, the tracking method may involve extracting feature points from both the pre-operative and start-of-surgery images and determining the transformation parameters from their correspondence; or, feature points may be extracted from one of the images, the movement of those feature points may be searched, and then the transformation parameters determined; or, the images may be input into a machine learning model, and the transformation parameters may be determined through learning.

[0052] In image generation, based on the transformation parameters extracted by tracking, the control point coordinates P1 of the region that generates the boundary K1 indicating a specific position (e.g., a modulation region) are transformed from the surgical plan information obtained by registration, as shown in Figure 5, or the control point coordinates P1 of the boundary line indicating boundary K1 are transformed, as shown in Figure 6. Subsequently, if the target of transformation is the control point coordinates P1 of the region, image processing is performed on the region; if the target of transformation is the control point coordinates P1 of the boundary line, image processing is performed on the region based on the boundary line to generate a display image. This display image will contain multiple regions with different display characteristics.

[0053] In the example in Figure 5, there are four control point coordinates P1 that indicate the region, but any three or more control point coordinates P1 are acceptable. The region enclosed by these control point coordinates P1 is the target of processing. In the example in Figure 6, there are two control point coordinates P1 that indicate the boundary line, but any two or more control point coordinates P1 are acceptable. If the boundary line has a complex shape, the number of control point coordinates, i.e., control points, may be increased. Once the boundary line is defined, the region on either the left or right (or top or bottom) side of that boundary line is the target of processing. Note that each control point coordinate P1 indicating the region or boundary line may be connected by a straight line, smoothly connected by spline interpolation, or connected by a specific shape such as a semicircle passing through two points. Furthermore, the region to be processed is not limited to one; there may be multiple regions.

[0054] In image presentation, the display image generated by the image generation process is presented as a surgical field image. This display image with boundary K1 (see Figure 4) is displayed on the display screen by either or both the monitor 14 and the display unit 55.

[0055] Here, coordinate transformations such as affine transformations and homography transformations can be used. Image processing is achieved by adjusting parameters such as luminance (brightness), contrast (intensity), saturation, color temperature, sharpness, grayscale conversion, and changing specific colors to other specific colors, i.e., by changing the pixel values ​​of the image. Specifically, processing based on calculation formulas (e.g., nonlinear calculations such as gain adjustment, offset processing, and gamma processing), processing using lookup tables (e.g., changing from one specific color to another, or converting from one specific luminance value to another to change the contrast), and processing using spatial filters can be used individually or in combination. In this case, the display image generation unit 13f may automatically select and execute processing that makes the boundaries more prominent on the original surgical field image (original image). An example of conversion from one specific luminance value to another is the modification of the S-curve of the contrast curve.

[0056] In image processing, brightness is adjusted by, for example, modifying a specific channel. Contrast is adjusted by, for example, adding gain according to the value of a specific channel. Saturation is adjusted by, for example, adding a uniform gain to a specific channel. Color temperature is adjusted by, for example, adding a uniform gain that differs for each channel. Grayscale conversion is adjusted by, for example, changing the value of a specific channel. Color changes are adjusted by, for example, performing a conversion according to the pixel value.

[0057] Images contain color information in the form of channels, for example. RGB images have three channels: Red, Green, and Blue. HSL images have three channels: Hue, Saturation, and Lightness / Luminance (or Intensity). CMYK images have four channels: Cyan, Magenta, Yellow, and Black.

[0058] Furthermore, in image generation, the information patterns and processing methods presented may be changed based on instructions from the user, such as the surgeon. Examples of information patterns to present include various information patterns (various display patterns) corresponding to wound creation, anterior capsulotomy, alignment of toric IOLs (intraocular lenses for astigmatism correction), and centering of IOLs. For example, the user can operate the control unit to select the information pattern to present or the processing method.

[0059] Furthermore, because tracking takes processing time, waiting for tracking information to be generated before creating the surgical field image results in a large delay between the image and reality. Therefore, it may be possible to ignore the delay in tracking information and use the latest calculated historical tracking information for image generation to minimize the delay in image creation.

[0060] (Examples of displayed images 2-6) Examples 2 to 6 of the display images according to the embodiment will be described with reference to Figures 7 to 11. Figures 7 to 11 are diagrams showing examples 2 to 6 of the display images according to the embodiment. Examples 2 to 6 will describe variations of the display images.

[0061] As shown in Figure 7, two boundaries K2 and K3 are presented in the displayed image, indicating the central position. In the example in Figure 7, the intersection of boundaries K2 and K3 indicates, for example, the central position for IOL placement (e.g., the axial position of the eye). Also, in the example in Figure 7, the area to the right of boundary K2 in the surgical field image is processed, and the area below boundary K3 is processed.

[0062] As shown in Figure 8, a boundary K4 indicating the incision site is presented in the displayed image. In the example in Figure 8, boundary K4 is two sides of a triangle, and the vertices of the triangle indicate the incision site (e.g., the incision start site). In addition, in the example in Figure 8, the area below boundary K4 (the triangular area) of the surgical field image is processed.

[0063] As shown in Figure 9, two boundaries K5 and K6 are presented in the displayed image to indicate the incision location. In the example in Figure 9, the intersection of each boundary K5 and K6 indicates the incision location (e.g., the incision start position). Also, in the example in Figure 9, the area to the right of boundary K5 in the surgical field image is processed, and the area below boundary K6 is processed.

[0064] As shown in Figure 10, a boundary K7 indicating the incision size and position is presented in the displayed image. In the example in Figure 10, boundary K7 indicates the incision size and position (e.g., consecutive incision positions) for, for example, CCC (anterior capsulotomy). This boundary K7 functions as, for example, a boundary with a semicircular shape, i.e., a semicircular boundary (a semicircle for forming the target circle for anterior capsulotomy). In the example in Figure 10, the area to the right of boundary K7 in the surgical field image is processed. In addition to the axial length of the eye, the center of the ring shape such as the target circle can also be the center of the limbus, the center of the pupil, the center of the preoperative pupil, the visual axis, the center of the anterior capsulotomy margin, etc.

[0065] As shown in Figure 11, a boundary K8 is presented in the displayed image, indicating a specific region, i.e., the size and location of the specific region. In the example in Figure 11, boundary K8 is the six sides (or four sides) of a hexagon, indicating the size and location of the specific region. This boundary K8 indicates to the surgeon, for example, to bring the eyeball (eye) to the center of the surgical field image if the eyeball is lost during tracking. In the example in Figure 11, the region outside boundary K8 of the surgical field image is processed.

[0066] (Examples of displayed images 7-9) Examples 7 to 9 of the display images according to the embodiment will be described with reference to Figures 12 to 14. Figures 12 to 14 show examples 7 to 9 of the display images according to the embodiment. In examples 8 and 9, additional points are described to reduce the difference between the pre-processed image and the post-processed image while maintaining the clarity of the boundary K1.

[0067] As shown in Figure 12, the region to which image processing is applied may be one side of the boundary K1. When the region to which image processing is applied is one side of the boundary K1, the amount of change in the processed region is large, while the unprocessed region remains. In other words, when processing one side of the region, the modulation is performed at a level where the boundary K1 is visible, so the processed region has a large difference from the original image, while the unprocessed region also exists, which is an advantage.

[0068] As shown in Figure 13, the areas to which image processing is applied may be the areas on both sides of boundary K1. In the example in Figure 13, one side of the area (the upper area of ​​boundary K1) is made 10% brighter than the original image (the surgical field image before processing), and the other side of the area (the lower area of ​​boundary K1) is made 10% darker than the original image. By applying different processing to the areas on both sides in this way, it is possible to maintain the clarity of the boundary gap while reducing the amount of change from the original image. In other words, when processing both sides of the area, there is no area that is not processed, and since the same boundary gap is achieved by processing both sides of the area, there is an advantage in that the difference between the original image and the processed surgical field image can be kept to a small range.

[0069] As shown in Figure 14, information transmission is possible if there is a difference (difference in display mode) between the regions on both sides of boundary K1, so the intensity of processing on the region (e.g., the intensity of modulation) may be weakened as you move away from boundary K1. In the example in Figure 14, in the region to the left of boundary K1, the intensity of processing on the region weakens as you move away from boundary K1. For example, if the processing on the region is a process to increase brightness, then as you move away from boundary K1, the intensity of the brightness-increasing process on the region weakens, and the brightness of the region weakens as you move away from boundary K1. By weakening the intensity of processing on the region as you move away from boundary K1 in this way, the difference between the original image and the processed surgical field image can be reduced in the part away from boundary K1. In other words, the clarity of the gap at boundary K1 can be maintained, and the region away from boundary K1 can be brought closer to the original image.

[0070] (Examples of displayed images 10 and 11) Examples 10 and 11 of the display images according to the embodiment will be described with reference to Figures 15 and 16. Figures 15 and 16 show examples 10 and 11 of the display images according to the embodiment. Examples 10 and 11 will explain additional points regarding the presentation of 3D images (three-dimensional surgical field images). Typically, 3D images are often used in ophthalmic surgery. In this case, there are stereoscopic images for the left eye and stereoscopic images for the right eye so that the sense of depth can be presented as a difference in parallax. For this reason, additional points regarding the presentation of boundaries for the stereoscopic left eye and stereoscopic right eye images will be explained. In the following, "stereoscopic left eye" will simply be referred to as "left eye," and "stereoscopic right eye" will simply be referred to as "right eye."

[0071] As shown in Figure 15, regarding boundary presentation for 3D images, boundaries may be presented for both the left-eye and right-eye images, i.e., boundary K2 may be presented for the right-eye image and boundary K3 for the left-eye image, or boundaries K2 and K3 may be presented for only one of the left-eye or right-eye images (see Figure 7). Furthermore, presenting boundaries K2 and K3 only for one eye image results in minimal image change from the original image, thus having little effect on 3D perception, and the operator has the advantage of being able to visually perceive boundaries K2 and K3. Therefore, boundaries K2 and K3 may be presented only for one eye image, or different boundaries (e.g., boundary K2 and boundary K3) may be presented for the left-eye and right-eye images.

[0072] In the example shown in Figure 15, different boundaries K2 and K3 are presented to the left-eye image and the right-eye image (boundary K2 is presented to the right-eye image and boundary K3 to the left-eye image). By fusing these boundaries K2 and K3 in the brain, the intersection of each boundary K2 and K3 is presented. In this way, for example, the centering position or the wound position can be indicated by the crossing position of the two boundaries K2 and K3. Furthermore, the amount of information that can be presented can be increased by combining various boundaries (e.g., boundaries K2 and K3).

[0073] Furthermore, when presenting different boundaries K2 and K3 for the left eye image and the right eye image, the different boundaries K2 and K3 may be presented based on the information tracked in the monocular image, or the different boundaries K2 and K3 may be presented based on the information tracked in the binocular image, which consists of the left eye image and the right eye image.

[0074] In addition to presenting different boundaries K2 and K3 in both the left-eye and right-eye images, the same boundary (e.g., boundary K1) may also be presented in both images. When the same boundary is presented in both the left-eye and right-eye images, depth perception will occur relative to the boundary, so it may be possible to control where the boundary is localized. For example, if the ophthalmic surgery is CCC (anterior capsulotomy), the boundary may be localized to the position of the anterior capsule.

[0075] As shown in Figure 16, boundary K1 may be processed to create a depth gap in the 3D image. In the example in Figure 16, the image processing involves shifting the pixels of the left-eye image to the right and the pixels of the right-eye image to the left. After processing (after parallax modulation), the parallax changes, and the fixed position in depth changes. The surgeon perceives boundary K1 as protruding forward. This makes it easier for the surgeon to grasp the position of boundary K1, allowing for more precise ophthalmic surgery.

[0076] Furthermore, when the intensity of image processing (e.g., the degree of modulation) is small, applying processing to only one eye's image or different processing to individual images of both eyes does not cause flickering, unlike the superposition of marks, and also does not cause parallax, thus preventing conflict between the surgical field and depth. In addition, if the same processing is applied to individual images of both eyes, it becomes possible to localize to a desired specific depth position by creating parallax, and it is also possible to localize to the position where the user will perform the procedure according to the guide.

[0077] (Changes in boundary presentation) This section will explain additional points regarding the changes in boundary presentation according to the tracking status and time according to the embodiment. First, the changes in boundary presentation according to the tracking status will be explained with reference to Figure 17, then the changes in boundary presentation according to time will be explained with reference to Figures 18 to 20, and finally the changes in boundary presentation according to the tracking result (change in eye size) will be explained with reference to Figure 21. Figure 17 is a diagram illustrating the changes in boundary presentation according to the tracking status according to the embodiment. Figures 18 to 20 are the first to third diagrams illustrating the changes in boundary presentation according to time according to the embodiment. Figure 21 is a diagram illustrating the changes in boundary presentation according to the tracking result according to the embodiment.

[0078] Due to the principles of eye tracking, tracking accuracy deteriorates as the amount of the eye visible in the surgical field decreases. Furthermore, to maintain tracking accuracy, a tracking detection limit may be set. That is, there may be situations where tracking accuracy is confident, or situations where it is approaching the tracking detection limit. In such cases, as shown in Figure 17, the processing strength of a predetermined area (e.g., the degree of modulation) may be reduced to bring the image in that area closer to the original image, or the processing strength of a predetermined area may be increased to make the image in that area stand out as a warning. Of course, in conjunction with these measures, a message may also be overlaid and displayed at a fixed position on the display screen. By presenting the aforementioned tracking status to the user, such as the surgeon, the user can grasp the reliability of the tracking.

[0079] Furthermore, while presenting information using boundaries K1-K7 for each region does not, in principle, cause occlusion (obstruction) of the surgical field, some surgeons may perceive a slight decrease in visibility along boundaries K1-K7. Therefore, as shown in Figures 18 and 19, the positions of boundaries K1-K7, which present the same information (specific location), may be changed.

[0080] In the example shown in Figure 18, boundary K7 rotates 360 degrees around the axial length of the eye, etc. For example, in a surgery such as anterior capsulotomy, boundary K7 rotates at a predetermined speed (for example, faster than the speed at which the surgeon moves the tip of the surgical instrument) from the start of the surgery. This rotation is repeated during the procedure. As a result, boundary K7 forms a target circle for anterior capsulotomy. In this way, visibility can be improved by changing the position of boundary K7, which presents the same information.

[0081] Here, the aforementioned predetermined speed is set in advance and is faster than a general value, such as the average speed at which the surgeon moves the tip of the surgical instrument. However, the rotation speed of boundary K7 does not have to be the predetermined speed; for example, the tip of the surgical instrument being moved by the surgeon or the endpoint of the anterior capsulotomy margin may be detected, and boundary K7 may be rotated according to the movement of the surgical instrument or the movement of the endpoint of the anterior capsulotomy margin. Also, the rotation angle of boundary K7 may be other angles, such as 180 degrees.

[0082] In the example shown in Figure 19, boundary K1 moves in parallel at a predetermined speed. Boundary K1 moves a predetermined distance to the left from the reference position, returns to the reference position from that predetermined position to the left, then moves a predetermined distance to the right from the reference position, and returns to the reference position from that predetermined position to the right. This movement is repeated during the procedure (periodic movement). This allows for improved visibility by changing the position of boundary K1, which presents the same information.

[0083] Furthermore, as shown in Figure 20, visibility may be improved by periodically weakening the processing intensity (e.g., the degree of modulation) in a predetermined area. In the example in Figure 20, the processing intensity is gradually weakened from the initial state, then gradually strengthened, and then returned to the initial state. By periodically weakening the processing intensity (the degree of image change from the original image) in a predetermined area in this way, visibility can be improved.

[0084] In this way, by periodically weakening the processing strength of a predetermined area (the degree of image change from the original image), the visibility of the surgical field can be further improved. Also, by periodically shifting the presentation position of boundary K1, such as the toric axis, it is possible to avoid the boundary K1 overlapping with the toric IOL mark B1a (see Figure 4), which would make the mark B1a difficult to see. Furthermore, since tracking accuracy tends to decrease when the cornea, which is the tracking target, is located at the edge, weakening the processing strength of a predetermined area can improve the visibility of the surgical field or encourage the user to keep the cornea in the central area where tracking accuracy is higher.

[0085] As mentioned above, the positions of boundaries K1 and K7 may be changed at predetermined intervals, or they may be switched according to user instructions. For example, the user can switch the positions of boundaries K1 and K7 by operating an operating unit such as a touch panel, foot switch, or controller.

[0086] Furthermore, as described above, image processing may be performed on a predetermined area, or the operating part of a surgical instrument or the like (for example, the tip of a surgical instrument) operated by the surgeon may be detected, and image processing (for example, brightness modulation processing or color modulation processing) may be performed on the area that does not include that operating part.

[0087] Furthermore, as shown in Figure 21, the position and size of boundary K7 may be changed in accordance with changes in eye size. Since a portion of boundary K7 is semicircular and indicates the size of the eye, the position of boundary K7 will change in accordance with changes in eye size. For example, if the size of the eye in the surgical field image decreases, the semicircular size of boundary K7 also decreases, and the position of boundary K7 moves closer to the center. Conversely, if the size of the eye in the surgical field image increases, the semicircular size of boundary K7 also increases, and the position of boundary K7 moves away from the center. Examples of size include the diameter of the CCC, the incision width of the wound, and centering.

[0088] (Examples of displayed images 12-14) Examples 12 to 14 of the display images according to the embodiment will be described with reference to Figures 22 to 24. Figures 22 to 24 are diagrams showing examples 12 to 14 of the display images according to the embodiment.

[0089] As shown in Figure 22, boundary K7 presents two specific locations in the displayed image. In the example in Figure 22, the two specific locations each indicate wound information (e.g., wound creation location). For example, a part of boundary K7 is formed in a triangular shape, and the area near its vertex is the wound creation location. In the example in Figure 22, the region to the right of boundary K7 in the surgical field image is processed.

[0090] As shown in Figure 23, in the displayed image, boundary K11 indicates the width and position of the main incision. In the example in Figure 23, boundary K11 indicates the incision width, which is the width of the triangle formed from the center of the cornea over the limbus. The position of the main incision, i.e., the position of the incision, is the virtual perpendicular bisector of the triangle. In the example in Figure 23, the left region of boundary K11 in the surgical field image is processed.

[0091] As shown in Figure 24, in the displayed image, boundary K11 indicates the width of the main incision, and boundary K12 indicates the position of the main incision. In the example in Figure 24, boundary K11 indicates the incision width, which is the width of the triangle formed from the center of the cornea on the limbus. Boundary K12 indicates the position of the main incision, i.e., the virtual perpendicular bisector of the triangle. In the example in Figure 24, the left region of boundary K11 in the surgical field image is processed, and the lower region of boundary K12 is processed. Alternatively, a left-eye image including boundary K11 and a right-eye image including boundary K12 may be presented, and their boundaries K11 and K12 may be fused in the brain to realize a 3D image (see Figure 16).

[0092] (Changes in boundary presentation) The changes in boundary presentation over time according to this embodiment will be explained with reference to Figure 25. Figure 25 is a fourth diagram illustrating the changes in boundary presentation over time according to this embodiment.

[0093] As shown in Figure 25, the width and position of boundary K11 may be periodically changed in the displayed image. For example, the desired incision width is set to the point where the width of boundary K11 (the size of the area within boundary K11) is at its maximum, and the width of boundary K11 is narrowed to indicate the incision position. In other words, if the incision width is only indicated by the width at the limbus, it may be difficult to determine the incision position. For this reason, the width of boundary K11 may be periodically changed to narrow the width of boundary K11 and indicate the incision position.

[0094] <Brightness Area> Here, we will explain examples 15-17 of display images that show the boundary M3-M5 using multiple luminance regions with different brightness levels.

[0095] (Example image 15) Figure 26 shows an example of a display image 15 according to the embodiment. As shown in Figure 26, two luminance regions with different luminances are set, and the boundary M3 between these luminance regions is presented. This boundary M3 functions as a line-shaped boundary, i.e., a line boundary (target line for intraocular lens B1 placement). In the example in Figure 26, the luminance of the right luminance region (the shaded area in Figure 26) is set lower than the luminance of the left luminance region. The toric axis is aligned with this boundary M3, and the toric IOL is placed. Note that the number of luminance regions is not limited to two; there may be two or more.

[0096] (Example image 16) Figures 27 and 28 show an example 16 of a display image according to the embodiment. As shown in Figures 27 and 28, two brightness regions with different brightness levels are set and presented as the boundary M4 between these brightness regions. This boundary M4 functions as a boundary with a semicircular shape, i.e., a semicircular boundary (a semicircle for forming a target circle for anterior capsulotomy). In the example of Figures 27 and 28, the brightness region boundary M4 is rotated 90 degrees around the axial length of the eye, etc.

[0097] For example, in surgeries such as anterior capsulotomy, the boundary M4 of the brightness region rotates 360 degrees around the axial length of the eye at a predetermined speed (for example, the speed at which the surgeon moves the tip of the surgical instrument) from the start of the surgery. This causes boundary M4 to form a target circle for anterior capsulotomy. The predetermined speed is set in advance and is a general value such as the average speed at which the surgeon moves the tip of the surgical instrument. However, the rotation speed of boundary M4 does not have to be the predetermined speed; for example, the tip of the surgical instrument being moved by the surgeon or the endpoint of the anterior capsulotomy margin may be detected, and boundary M4 may be rotated according to the movement of the surgical instrument or the movement of the endpoint of the anterior capsulotomy margin. The processing start unit 13g, which will be described later, can be used to detect the tip of the surgical instrument or the endpoint of the anterior capsulotomy margin. Also, the rotation angle of boundary M4 may be other angles, such as 180 degrees.

[0098] (Example image 17) Figure 29 shows an example of a display image 17 according to the embodiment. As shown in Figure 29, in addition to the boundary M4 shown in Figures 27 and 28, several (two in the example of Figure 29) boundaries M5 are presented. These boundaries M4 and M5 are formed by the boundary between two luminance regions with different luminances, similar to example 10 of the display image. Boundary M5 is a boundary indicating the incision position.

[0099] As is clear from the above explanation, boundaries K1-K12 and M3-M5 are not marks superimposed on the surgical field image, but rather boundaries that allow for visual indication of orientation (position, orientation, size, etc.). Unlike superimposed marks, boundaries K1-K12 and M3-M5 do not obscure the surgical field image at the mark's location, thus improving the visibility of the surgical field compared to using superimposed marks.

[0100] Furthermore, as mentioned above, the surgical field image may be fixed, and the orientation of boundaries K1-K12 and M3-M5 may be changed so that they are in an appropriate position (position, orientation, etc.) relative to the fixed surgical field image (for example, so that there is no displacement of boundaries K1-K12 and M3-M5 relative to the eyeball in the fixed surgical field image). Changing the orientation of boundaries K1-K12 and M3-M5 will change the range of each region (for example, size, shape, etc.).

[0101] For example, when changing the orientation of boundaries K1-K12 and M3-M5, the display image generation unit 13f generates a display image while changing the orientation of boundaries K1-K12 and M3-M5 according to the displacement of the eyeball, based on the orientation information of the eyeball. For example, the display image generation unit 13f moves boundaries K1-K12 and M3-M5 in the direction of eyeball movement by the aforementioned amount relative to the real-time surgical field image, according to the direction and amount of eyeball movement, and changes the orientation of boundaries K1-K12 and M3-M5 (for example, the range of each region). In other words, by fixing the surgical field image and changing the orientation of boundaries K1-K12 and M3-M5, the positional relationship between the eyeball and boundaries K1-K12 and M3-M5 remains unchanged.

[0102] Furthermore, tracking may fail, resulting in times when the eye's posture (position, orientation, etc.) in the real-time image cannot be estimated. In such cases, the display image generation unit 13f may continue displaying by maintaining the posture of the image from the time when the posture was last estimated (the last displayed image). In addition to maintaining the posture of the surgical field image from the time when the posture was last estimated, the posture of the surgical field image from the time when the posture was estimated may also be maintained using constant velocity, constant angular velocity, constant acceleration motion, or constant angular acceleration motion. Also, if tracking fails, the display characteristics of a predetermined area (e.g., brightness, color, etc.) may be changed to indicate that the failure has occurred.

[0103] While various display images as described above are used, these display images may be selectable by the operator or staff. The selection of a display image is achieved through input operations on the control unit by the operator or staff. For example, the operator or staff operates the control unit to select a display mode that displays the desired display image. In response to this selection, the display image generation unit 13f generates a display image based on the selected display mode. Similarly, the size, position, etc., of each image may be changed by the operator or staff. The display image generation unit 13f changes the size, position, etc., of the image in response to input operations on the control unit by the operator or staff and generates a display image.

[0104] <1-4. Action and Effects> As described above, according to the embodiment, the image input unit 13b receives a surgical field image of the patient's eye, the eyeball tracking unit 13e tracks the eyeball in the surgical field image, and the display image generation unit 13f sets multiple regions with different display modes relative to the surgical field image and generates a display image in which the boundary of each region (e.g., boundaries K1-K12, M3-M5) indicates at least one of a specific position, specific direction, and specific size relative to the eye. Furthermore, based on the eyeball tracking result (tracking result), the display mode of one or all of the regions is changed, and at least one of the position, direction, and size of the boundary is changed. This prevents occlusion by presenting the specific position of the eye not with a mark, but with the boundary of each region with a different display mode. Therefore, the surgeon can grasp the specific position or specific size, making it easier to view the surgical field image and perform surgery accurately, thus enabling surgery to be performed with high precision according to the preoperative plan. Displacement includes arbitrary changes to the subject such as the eyeball, such as translation, rotation, enlargement / reduction, deformation, and combinations thereof.

[0105] Furthermore, the display image generation unit 13f changes the display mode of one or all of the multiple regions based on the eyeball tracking results, and changes at least one of the position, direction, and size of the boundary. For example, when changing the display mode of one of the multiple regions, the processing speed can be improved compared to when changing the display mode of all of the multiple regions. Also, when changing the display mode of all of the multiple regions, the operator can more easily see the boundaries of each region compared to when changing the display mode of one of the multiple regions, thus enabling surgery to be performed with high accuracy in accordance with the preoperative plan.

[0106] Furthermore, the preoperative planning receiving unit 13a receives a preoperative image and at least one piece of information (e.g., position, orientation, and size on a coordinate system) based on the preoperative plan for the patient's eye. The information storage unit 13d compares the preoperative image with the surgical field image at the start of surgery and modifies at least one piece of information (position, orientation, and size) of the indicators to match the surgical field image at the start of surgery. The system then stores the surgical field image at the start of surgery and at least one piece of information (position, orientation, and size) of the modified indicators. This allows the surgical field image at the start of surgery and at least one piece of information (position, orientation, and size) of the modified indicators to be used in post-processing.

[0107] The eyeball tracking unit 13e tracks the eyeball in the real-time surgical field image by comparing the surgical field image at the start of surgery with the real-time surgical field image, and outputs relational information showing the relationship between at least one piece of information about the position, direction, and size of the eyeball in the real-time surgical field image (e.g., position, orientation, and size on a coordinate system) and at least one piece of information about the position, direction, and size of the changed indicator. The display image generation unit 13f places the boundary based on at least one piece of information about the position, direction, and size of the changed indicator, and based on the relational information, changes at least one of the position, direction, and size of the boundary in the real-time surgical field image to eliminate at least one change in the position, direction, and size of the boundary with respect to the eyeball in the surgical field image at the start of surgery, and generates a display image. As a result, by changing at least one of the position, direction, and size of the boundary K1 to K7 with respect to the eyeball in the surgical field image at the start of surgery, at least one relationship between the position, direction, and size of the eyeball and the boundary K1 to K7 remains unchanged. Therefore, the surgeon can grasp at least one specific position, direction, and size in detail, and can perform surgery in accordance with the preoperative plan with greater accuracy.

[0108] Furthermore, the display image generation unit 13f modifies multiple regions by coordinate transformation so that at least one of the position, direction, and size of the boundary is changed based on the eyeball tracking results, and generates a display image. This ensures that at least one of the position, direction, and size of the boundary is changed based on the eyeball tracking results, and a display image is generated.

[0109] Furthermore, the display image generation unit 13f modifies the boundary line based on the eye tracking results, changing at least one of the boundary's position, direction, and size. This improves processing speed compared to modifying the region. For example, a boundary line only requires two control points (conversion points), while a region requires three or more control points.

[0110] Furthermore, the display image generation unit 13f performs processing to adjust the same type of parameter (for example, brightness, color, etc.) for two or more of the multiple regions. This simplified processing makes it easy to highlight the boundaries of each region. As a result, the surgeon can more easily visualize the boundaries of each region, enabling them to perform surgery with high accuracy in accordance with the preoperative plan.

[0111] Furthermore, the display image generation unit 13f performs processing to adjust different types of parameters (e.g., brightness, color, etc.) for two or more regions. This simplified processing makes it easy to highlight the boundaries of each region. As a result, the surgeon can more easily visualize the boundaries of each region, enabling them to perform surgery with high accuracy in accordance with the preoperative plan.

[0112] Furthermore, the display image generation unit 13f weakens the processing strength for any or all of the multiple regions as it moves away from the boundary. This makes it possible to maintain the clarity of the boundary while bringing the regions further away from the boundary closer to the original image, thereby enabling highly accurate surgery in accordance with the preoperative plan.

[0113] Furthermore, the display image generation unit 13f generates a display image including the boundary as either a display image for the left eye (for stereoscopic viewing of the left eye) or a display image for the right eye (for stereoscopic viewing of the right eye), or it generates a display image including the boundary as a display image for the left eye and a display image for the right eye, respectively. As a result, when generating a display image including the boundary as either a display image for the left eye or a display image for the right eye, the processing speed can be improved compared to when the display image including the boundary is generated as both a display image for the left eye and a display image for the right eye, respectively. Also, when generating a display image including the boundary as both a display image for the left eye and a display image for the right eye, respectively, the positional accuracy of the boundary can be improved compared to when the display image including the boundary is generated as either a display image for the left eye or a display image for the right eye, respectively.

[0114] Furthermore, the eyeball tracking unit 13e tracks the eyeballs in one or both of the left and right eye field images, and the display image generation unit 13f changes at least one of the boundary position, direction, and size based on the tracking results of one or both of the left and right eyeballs to generate a display image for the left eye and a display image for the right eye. As a result, when using one of the left or right eye field images, the processing speed can be improved compared to when using both field images. When using both field images, the positional accuracy of the boundary can be improved compared to when using one of the field images.

[0115] Furthermore, the display image generation unit 13f ensures that the boundary is in the same position for both the left eye display image and the right eye display image. This makes it easier for the surgeon to understand the boundary position, enabling highly accurate surgery in accordance with the preoperative plan.

[0116] Furthermore, the display image generation unit 13f generates display images for the left eye and the right eye, in which at least one of the boundary positions, directions, and sizes differs, so as to indicate at least one of a specific position, direction, and size. This makes it possible to indicate at least one of a specific position, direction, and size at multiple boundaries, thereby enabling highly accurate surgery in accordance with the preoperative plan.

[0117] Furthermore, the display image generation unit 13f shifts the display images for the left eye and the right eye based on the depth information of the desired localization of the three-dimensional image, which includes the display images for the left eye and the display images for the right eye, to generate a three-dimensional image. This makes it easier for the surgeon to grasp the position of the boundary, enabling highly accurate surgery in accordance with the preoperative plan.

[0118] Furthermore, the display image generation unit 13f maintains the last displayed image before the eyeball moved out of the tracking range of the eyeball by the eyeball tracking unit 13e. This makes it possible to avoid interruptions to the surgery due to the loss of the displayed image, thereby enabling highly accurate surgery in accordance with the preoperative plan.

[0119] Furthermore, the display image generation unit 13f changes the display mode of one or all of the multiple regions according to the time period. This allows the display image to periodically be brought closer to the original image, or the boundaries of the display image to be made more prominent. Therefore, the surgeon can reliably see the original image and boundaries, enabling highly accurate surgery in accordance with the preoperative plan.

[0120] Furthermore, the display image generation unit 13f generates a display image in which the boundary indicates multiple specific locations. This makes it possible to present more specific locations, thereby enabling highly accurate surgery in accordance with the preoperative plan.

[0121] Furthermore, the display image generation unit 13f changes the size of one or all of the multiple regions in response to changes in the size of the eyeball. This makes it possible to change the size of one or more regions to match the size of the eyeball, and to change the position and size of the boundaries, thereby enabling highly accurate surgery in accordance with the preoperative plan.

[0122] Furthermore, the display image generation unit 13f changes the size of one or all of the multiple regions according to the time period. This makes it possible to periodically change the size of one or more regions and alter the position and size of the boundaries, thereby enabling highly accurate surgery in accordance with the preoperative plan. Also, for example, by making the region smaller, the displayed image can be made closer to the original image, allowing the surgeon to clearly see the original image, thus enabling highly accurate surgery in accordance with the preoperative plan.

[0123] Furthermore, the display characteristics of each of the multiple regions differ due to the differences in brightness between them. This makes it easy to highlight the boundaries of each region. Consequently, the surgeon can more easily visualize the boundaries of each region, enabling them to perform surgery with high precision according to the preoperative plan.

[0124] Furthermore, the specific location is the toric axis position of the intraocular lens, and the display image generation unit 13f changes the brightness of each of the multiple regions so that the boundary indicates the toric axis position. This makes it possible to easily highlight the toric axis position. Therefore, the surgeon can more easily visualize the toric axis position, enabling them to perform surgery with high accuracy in accordance with the preoperative plan.

[0125] Furthermore, the specific locations are one of the following: the toric axis position of the intraocular lens, the incision position for forceps insertion, the incision position for anterior capsulotomy, the axial position, the center of the limbus, the center of the pupil, the center of the preoperative pupil, the visual axis position, and the center of the anterior capsulotomy margin. This makes it possible to highlight one of the following at the boundary of each region: the toric axis position of the intraocular lens, the incision position for forceps insertion, the incision position for anterior capsulotomy, the axial position, the center of the limbus, the center of the pupil, the center of the preoperative pupil, the visual axis position, and the center of the anterior capsulotomy margin. Therefore, the surgeon can more easily visualize these specific locations, enabling them to perform surgery with high precision according to the preoperative plan.

[0126] Furthermore, the display image generation unit 13f changes the display mode of one or all of the multiple regions according to the eye tracking status (tracking status) of the eyeball by the eyeball tracking unit 13e. This makes it possible to bring the displayed image closer to the original image or to make the boundaries of the displayed image more prominent according to the eyeball tracking status, so that the operator can grasp the eyeball tracking status (for example, the reliability of the tracking).

[0127] <2. An example of a general computer configuration> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0128] Figure 30 shows an example of a schematic configuration of a computer 500 that executes the series of processes described above by a program.

[0129] As shown in Figure 30, the computer 500 has a CPU (Central Processing Unit) 510, a ROM (Read Only Memory) 520, and a RAM (Random Access Memory) 530.

[0130] The CPU 510, ROM 520, and RAM 530 are interconnected by a bus 540. An input / output interface 550 is further connected to this bus 540. An input / output interface 550 is connected to an input unit 560, an output unit 570, a recording unit 580, a communication unit 590, and a drive 600.

[0131] The input unit 560 consists of a keyboard, mouse, microphone, image sensor, etc. The output unit 570 consists of a display, speaker, etc. The recording unit 580 consists of a hard disk, non-volatile memory, etc. The communication unit 590 consists of a network interface, etc. The drive 600 drives a removable recording medium 610 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0132] In the computer 500 configured as described above, the CPU 510 loads, for example, a program stored in the recording unit 580 into the RAM 530 via the input / output interface 550 and the bus 540, and executes it, thereby performing the series of processes described above.

[0133] The program executed by the computer 500, i.e., the CPU 510, can be provided by recording it on a removable recording medium 610, such as a packaged medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0134] In the computer 500, programs can be installed in the recording unit 580 via the input / output interface 550 by inserting the removable recording medium 610 into the drive 600. Alternatively, programs can be received by the communication unit 590 via a wired or wireless transmission medium and installed in the recording unit 580. Furthermore, programs can be pre-installed in the ROM 520 or the recording unit 580.

[0135] The program executed by the computer 500 may be a program that is processed chronologically in the order described herein, or it may be a program that is processed in parallel or at necessary times, such as when a call is made.

[0136] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

[0137] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0138] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0139] Furthermore, each step described in the above-mentioned process flow (for example, a flowchart) can be executed by a single device, or it can be divided and executed by multiple devices.

[0140] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0141] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0142] <3. Addendum> Furthermore, this technology can also be configured as follows. (1) An image input unit that receives surgical field images of the patient's eye, The eye tracking unit tracks the eyeballs in the aforementioned surgical field image, A display image generation unit sets multiple regions with different display characteristics relative to the surgical field image and generates a display image in which the boundaries of the multiple regions indicate at least one of a specific position, specific direction, and specific size relative to the eye. Equipped with, The aforementioned display image generation unit, Based on the eye tracking results, the display mode of one or all of the plurality of regions is changed, and at least one of the position, direction and size of the boundary is changed. Image processing device. (2) A preoperative planning receiving unit that receives preoperative images and at least one piece of information on the position, direction, and size of indicators based on the preoperative plan for the eye, An information storage unit compares the preoperative image with the surgical field image at the start of surgery to change at least one piece of information regarding the position, direction, and size of the indicator to match the surgical field image at the start of surgery, and stores the surgical field image at the start of surgery and at least one piece of information regarding the changed position, direction, and size of the indicator. Furthermore, The image processing apparatus described in (1) above. (3) The aforementioned eyeball tracking unit is By comparing the surgical field image at the start of the procedure with the surgical field image in real time, the eyeball is tracked in the surgical field image in real time, and relational information is output showing the relationship between at least one piece of information on the position, direction, and size of the eyeball in the surgical field image in real time and at least one piece of information on the position, direction, and size of the modified index. The aforementioned display image generation unit, The boundary is positioned based on at least one piece of information regarding the position, direction, and size of the modified index; the position, direction, and size of the boundary in the real-time surgical field image are modified based on the relational information to eliminate at least one change in the position, direction, and size of the boundary relative to the eyeball in the surgical field image at the start of the surgery; and the display image is generated. The image processing apparatus described in (2) above. (4) The aforementioned display image generation unit, Based on the eyeball tracking results, the plurality of regions are modified by coordinate transformation so that at least one of the position, direction, and size of the boundary is changed, and the display image is generated. An image processing device as described in any one of the above (1) to (3). (5) The aforementioned display image generation unit, Based on the results of tracking the eyeball, the boundary line indicating the boundary is changed, and at least one of the position, direction, and size of the boundary is changed. An image processing device as described in any one of the above (1) to (3). (6) The aforementioned display image generation unit, The process of adjusting the same type of parameter is performed for two or more of the aforementioned regions. An image processing device as described in any one of the above (1) to (5). (7) The aforementioned display image generation unit, The process involves adjusting different types of parameters for two or more of the aforementioned regions. An image processing device as described in any one of the above (1) to (6). (8) The aforementioned display image generation unit, The intensity of processing for any or all of the aforementioned regions is reduced as it moves away from the boundary. An image processing device as described in any one of the above (1) to (7). (9) The aforementioned display image generation unit, The aforementioned display image is generated as either a display image for the left eye of stereoscopic viewing or a display image for the right eye of stereoscopic viewing, or the aforementioned display image is generated as a display image for the left eye of stereoscopic viewing and a display image for the right eye of stereoscopic viewing, respectively. An image processing device as described in any one of the above (1) to (8). (10) The aforementioned eyeball tracking unit is Track the eyeball in one or both of the surgical field images for stereoscopic vision of the left eye and the right eye, The aforementioned display image generation unit, Based on the tracking results of one or both of the eyeballs for stereoscopic viewing (left eye and right eye), the position, direction, and size of the boundary are changed to generate the display image for the stereoscopic left eye and the display image for the stereoscopic right eye. The image processing apparatus described in (9) above. (11) The aforementioned display image generation unit, In the stereoscopic display image for the left eye and the stereoscopic display image for the right eye, the boundary is positioned at the same location. The image processing apparatus described in (9) or (10) above. (12) The aforementioned display image generation unit, To generate the display images for the left eye and the right eye of stereoscopic viewing, in which at least one of the boundary position, direction, and size is different, the display images are generated so as to indicate at least one of the specified position, specified direction, and specified size. The image processing apparatus described in (9) or (10) above. (13) The aforementioned display image generation unit, Based on the depth information of the desired position of the three-dimensional image, which includes the display image for the left eye and the display image for the right eye of stereoscopic viewing, the display image for the left eye and the display image for the right eye of stereoscopic viewing are shifted to generate the three-dimensional image. An image processing device as described in any one of the above (9) to (12). (14) The aforementioned display image generation unit, If the eyeball moves out of the tracking range of the eyeball, the last displayed image before the eyeball moved out is maintained. An image processing device as described in any one of the above (1) through (13). (15) The aforementioned display image generation unit, Depending on the time period, the display mode or size of any or all of the above-mentioned areas is changed. An image processing device as described in any one of the above (1) through (14). (16) The aforementioned display image generation unit, The boundary generates the display image showing a plurality of specific positions. An image processing device as described in any one of the above (1) to (15). (17) The aforementioned display image generation unit, In response to the change in the size of the eyeball, the size of any or all of the multiple regions is changed. An image processing device as described in any one of the above (1) to (16). (18) The aforementioned display image generation unit, The size of one or all of the above regions is changed according to the time period. An image processing device as described in any one of the above (1) through (17). (19) The display mode of each of the plurality of regions differs depending on the difference in brightness of each of the plurality of regions. An image processing device as described in any one of the above (1) through (18). (20) The aforementioned specific position is the toric axis position of the intraocular lens. The aforementioned display image generation unit, The brightness of each of the plurality of regions is changed so that the boundary indicates the toric axis arrangement position. The image processing apparatus described in (19) above. (twenty one) The aforementioned specific location is one of the following: the toric axis position of the intraocular lens, the incision position for forceps insertion, the incision position for anterior capsulotomy, the axial position, the center of the limbus, the center of the pupil, the center of the preoperative pupil, the axial position, and the center of the anterior capsulotomy margin. An image processing device as described in any one of the above (1) to (20). (twenty two) The image processing device We received surgical field images of the patient's eye. Tracking the eyeballs in the aforementioned surgical field image, A plurality of regions with different display characteristics are set for the surgical field image, and a display image is generated in which the boundaries of the plurality of regions indicate at least one of a specific position, specific direction, and specific size relative to the eye. This includes, The aforementioned image processing device is Based on the eye tracking results, the display mode of one or all of the plurality of regions is changed, and at least one of the position, direction and size of the boundary is changed. Image processing methods. (twenty three) A surgical microscope that obtains images of the surgical field relative to the patient's eye, An image processing device that generates a display image, A display device that displays the aforementioned image, Equipped with, The aforementioned image processing device is An image input unit that receives the aforementioned surgical field image, The eye tracking unit tracks the eyeballs in the aforementioned surgical field image, A display image generation unit sets multiple regions with different display characteristics relative to the surgical field image and generates a display image in which the boundaries of the multiple regions indicate at least one of a specific position, specific direction, and specific size relative to the eye. Equipped with, The aforementioned display image generation unit, Based on the eye tracking results, the display mode of one or all of the plurality of regions is changed, and at least one of the position, direction and size of the boundary is changed. Surgical microscope system. (twenty four) An image processing method using an image processing device described in any one of (1) to (21) above. (twenty five) A surgical microscope system comprising an image processing device as described in any one of (1) to (21) above. [Explanation of symbols]

[0143] 1. Surgical microscope system 10 Surgical microscope 11 Objective lens 12 Eyepieces 13 Image Processing Device 13A Control Unit 13a Preoperative Planning Receipt Department 13b Image Input Section 13c Registration Section 13d Information Storage Unit 13e Eye tracking unit (eye tracking unit) 13f Display image generation section 14 monitors 20 patient beds 51 Light source 52 Observation Optical System 52a Half mirror 53 Front Image Capture Unit 54 Fault Imaging Unit 55 Presentation section 56 Interface section 57 speakers 500 Computers 510 CPU 520 ROM 530 RAM 540 bus 550 Input / Output Interfaces 560 Input section 570 Output section 580 Records Section 590 Communications Department 600 Drive 610 Removable recording media

Claims

1. An image input unit that receives surgical field images of the patient's eye, The eye tracking unit tracks the eyeballs in the aforementioned surgical field image, A display image generation unit sets multiple regions with different display characteristics relative to the surgical field image and generates a display image in which the boundaries of the multiple regions indicate at least one of a specific position, specific direction, and specific size relative to the eye. Equipped with, The aforementioned display image generation unit, Based on the eye tracking results, the display mode of one or all of the plurality of regions is changed, and at least one of the position, direction and size of the boundary is changed. A preoperative plan receiving unit that receives a preoperative image based on a preoperative plan for the eye and at least one piece of information regarding the position, direction, and size of an indicator related to the procedure based on the preoperative plan, An information storage unit compares the preoperative image with the surgical field image at the start of surgery to change at least one piece of information regarding the position, direction, and size of the indicator to match the surgical field image at the start of surgery, and stores the surgical field image at the start of surgery and at least one piece of information regarding the changed position, direction, and size of the indicator. Furthermore, The aforementioned eyeball tracking unit is By comparing the surgical field image at the start of the procedure with the surgical field image in real time, the eyeballs in the surgical field image in real time are tracked. The aforementioned display image generation unit, The boundary is positioned based on at least one piece of information regarding the position, direction, and size of the modified index, and based on the eyeball tracking results, the position, direction, and size of the boundary in the real-time surgical field image is modified so that at least one of the relationship between the position, direction, and size of the boundary and the eyeball in the surgical field image at the start of the surgery remains unchanged. Image processing device.

2. The aforementioned display image generation unit, Based on the eyeball tracking results, the plurality of regions are modified by coordinate transformation so that at least one of the position, direction, and size of the boundary is changed, and the display image is generated. The image processing apparatus according to claim 1.

3. The aforementioned display image generation unit, Based on the results of tracking the eyeball, the boundary line indicating the boundary is changed, and at least one of the position, direction, and size of the boundary is changed. The image processing apparatus according to claim 1.

4. The aforementioned display image generation unit, The process of adjusting the same type of parameter is performed for two or more of the aforementioned regions. The image processing apparatus according to claim 1.

5. The aforementioned display image generation unit, The process involves adjusting different types of parameters for two or more of the aforementioned regions. The image processing apparatus according to claim 1.

6. The aforementioned display image generation unit, The intensity of processing for any or all of the aforementioned regions is reduced as it moves away from the boundary. The image processing apparatus according to claim 1.

7. The aforementioned display image generation unit, The aforementioned display image is generated as either a display image for the left eye of stereoscopic viewing or a display image for the right eye of stereoscopic viewing, or the aforementioned display image is generated as a display image for the left eye of stereoscopic viewing and a display image for the right eye of stereoscopic viewing, respectively. The image processing apparatus according to claim 1.

8. The aforementioned eyeball tracking unit is Track the eyeball in one or both of the surgical field images for stereoscopic vision of the left eye and the right eye, The aforementioned display image generation unit, Based on the tracking results of the eyeball in one or both of the field images for stereoscopic left eye and stereoscopic right eye, the position, direction, and size of the boundary are changed to generate the display image for stereoscopic left eye and the display image for stereoscopic right eye. The image processing apparatus according to claim 7.

9. The aforementioned display image generation unit, In the stereoscopic display image for the left eye and the stereoscopic display image for the right eye, the boundary is positioned at the same location. The image processing apparatus according to claim 7.

10. The aforementioned display image generation unit, To generate the display images for the left eye and the right eye of stereoscopic viewing, in which at least one of the boundary position, direction, and size is different, the display images are generated so as to indicate at least one of the specified position, specified direction, and specified size. The image processing apparatus according to claim 7.

11. The aforementioned display image generation unit, Based on the depth information of the desired position of the three-dimensional image, which includes the display image for the left eye and the display image for the right eye of stereoscopic viewing, the display image for the left eye and the display image for the right eye of stereoscopic viewing are shifted to generate the three-dimensional image. The image processing apparatus according to claim 7.

12. The aforementioned display image generation unit, If the eyeball moves out of the tracking range of the eyeball, the last displayed image before the eyeball moved out is maintained. The image processing apparatus according to claim 1.

13. The aforementioned display image generation unit, Depending on the time period, the display mode of one or all of the above-mentioned areas is changed. The image processing apparatus according to claim 1.

14. The aforementioned display image generation unit, The boundary generates the display image showing a plurality of specific positions. The image processing apparatus according to claim 1.

15. The aforementioned display image generation unit, In response to the change in the size of the eyeball, the size of any or all of the multiple regions is changed. The image processing apparatus according to claim 1.

16. The aforementioned display image generation unit, The size of one or all of the above regions is changed according to the time period. The image processing apparatus according to claim 1.

17. The display mode of each of the plurality of regions differs depending on the difference in brightness of each of the plurality of regions. The image processing apparatus according to claim 1.

18. The aforementioned specific position is the toric axis position of the intraocular lens. The aforementioned display image generation unit, The brightness of each of the plurality of regions is changed so that the boundary indicates the toric axis arrangement position. The image processing apparatus according to claim 17.

19. The aforementioned specific location is one of the following: the toric axis position of the intraocular lens, the incision position for forceps insertion, the incision position for anterior capsulotomy, the axial position, the center of the limbus, the center of the pupil, the center of the preoperative pupil, the axial position, and the center of the anterior capsulotomy margin. The image processing apparatus according to claim 1.

20. The image processing device We received surgical field images of the patient's eye. Tracking the eyeballs in the aforementioned surgical field image, A plurality of regions with different display characteristics are set for the surgical field image, and a display image is generated in which the boundaries of the plurality of regions indicate at least one of a specific position, specific direction, and specific size relative to the eye. This includes, The aforementioned image processing device is Based on the eye tracking results, the display mode of one or all of the plurality of regions is changed, and at least one of the position, direction and size of the boundary is changed. The aforementioned image processing device The system receives a preoperative image of the eye based on the preoperative plan, and at least one piece of information regarding the position, direction, and size of the indicators related to the procedure based on the preoperative plan. By comparing the preoperative image with the surgical field image at the start of the surgery, at least one piece of information regarding the position, direction, and size of the indicator is changed to match the surgical field image at the start of the surgery, and the surgical field image at the start of the surgery and at least one piece of information regarding the changed position, direction, and size of the indicator are stored. This further includes, The aforementioned image processing device is By comparing the surgical field image at the start of the procedure with the surgical field image in real time, the eyeballs in the surgical field image in real time are tracked. The boundary is positioned based on at least one piece of information regarding the position, direction, and size of the modified index, and based on the eyeball tracking results, the position, direction, and size of the boundary in the real-time surgical field image is modified so that at least one of the relationship between the position, direction, and size of the boundary and the eyeball in the surgical field image at the start of the surgery remains unchanged. Image processing methods.

21. A surgical microscope that obtains images of the surgical field relative to the patient's eye, An image processing device that generates a display image, A display device that displays the aforementioned image, Equipped with, The aforementioned image processing device is An image input unit that receives the aforementioned surgical field image, The eye tracking unit tracks the eyeballs in the aforementioned surgical field image, A display image generation unit sets multiple regions with different display characteristics relative to the surgical field image and generates a display image in which the boundaries of the multiple regions indicate at least one of a specific position, specific direction, and specific size relative to the eye. Equipped with, The aforementioned display image generation unit, Based on the eye tracking results, the display mode of one or all of the plurality of regions is changed, and at least one of the position, direction and size of the boundary is changed. The aforementioned image processing device is A preoperative plan receiving unit that receives a preoperative image based on a preoperative plan for the eye and at least one piece of information regarding the position, direction, and size of an indicator related to the procedure based on the preoperative plan, An information storage unit compares the preoperative image with the surgical field image at the start of surgery to change at least one piece of information regarding the position, direction, and size of the indicator to match the surgical field image at the start of surgery, and stores the surgical field image at the start of surgery and at least one piece of information regarding the changed position, direction, and size of the indicator. Furthermore, The aforementioned eyeball tracking unit is By comparing the surgical field image at the start of the procedure with the surgical field image in real time, the eyeballs in the surgical field image in real time are tracked. The aforementioned display image generation unit, The boundary is positioned based on at least one piece of information regarding the position, direction, and size of the modified index, and based on the eyeball tracking results, the position, direction, and size of the boundary in the real-time surgical field image is modified so that at least one of the relationship between the position, direction, and size of the boundary and the eyeball in the surgical field image at the start of the surgery remains unchanged. Surgical microscope system.

Images