Image processing apparatus, image processing method, and surgical microscope system

Abstract

An image processing apparatus (13) according to an embodiment of the present disclosure includes an image input unit (13b) configured to receive a surgical field image of a patient's eye, an eye tracking unit (13e) configured to track an eye in the surgical field image, and a display image generator (13f) configured to set a plurality of regions with different display modes for the surgical field image and generate a display image in which a boundary between the plurality of regions indicates at least one of a specific position, a specific direction, or a specific size with respect to the eye. Based on a result of tracking the eye, the display image generator (13f) changes the display mode of any or all of the plurality of regions and changes at least one of a position, a direction, or a size of the boundary.

Description

Technical Field

[0001] This disclosure relates to image processing apparatus, image processing methods, and surgical microscope systems. Background Technology

[0002] As a method of refractive correction in ophthalmology, the insertion of an artificial lens, called an intraocular lens (IOL), into the eye to eliminate refractive errors of the lens and improve visual functions such as visual acuity is widely practiced. As an intraocular lens, the one inserted into the lens capsule is most widely used as a replacement for the lens removed through cataract surgery. Besides the lens capsule, there are various other types of intraocular lenses, such as those fixed (indwelling) in the ciliary sulcus (phakic IOLs).

[0003] When performing ophthalmic surgeries such as cataract surgery, in order to improve postoperative visual function, the surgeon performs the surgery so that, according to the preoperative plan, the incision location and shape, as well as the posture of the implant to be inserted, such as an intraocular lens, relative to the eye, are appropriate. At this time, it is desirable to present information about the appropriate incision location, incision shape, implant posture, etc., in a surgical format that is beneficial to the surgeon. Therefore, Patent Document 1 proposes a technique for changing the position of markers (patterns) indicating the preoperative plan based on the results of eye tracking.

[0004] Reference List

[0005] Patent documents

[0006] Patent Document 1: JP 2016-112358 A Summary of the Invention

[0007] Technical issues

[0008] However, because the markers based on the preoperative plan are overlaid on the surgical field image, occlusion occurs, making a portion of the surgical field image invisible due to the markers. Therefore, it is difficult for the surgeon to see the surgical field image, thus hindering the high-precision execution of the surgery according to the preoperative plan.

[0009] Therefore, this disclosure presents an image processing apparatus, an image processing method, and a surgical microscope system capable of performing surgery according to a preoperative plan with high precision.

[0010] Solution to the problem

[0011] An image processing apparatus according to an embodiment of the present disclosure includes: an image input unit for receiving a surgical field image of a patient's eye; an eye tracking unit for tracking an eyeball in the surgical field image; and a display image generation unit for setting multiple regions with different display modes in the surgical field image, and generating a display image representing the boundaries of the multiple regions for at least one of a specific position, a specific orientation, and a specific size of the eye, wherein the display image generation unit changes the display mode of any one or all of the multiple regions based on the eye tracking result, and changes at least one of the position, orientation, and size of the boundaries.

[0012] An image processing method according to an embodiment of the present disclosure, executed by an image processing device, includes: receiving a surgical field image of a patient's eye; tracking an eyeball in the surgical field image; and setting multiple regions with different display modes for the surgical field image, generating a display image of the boundaries of the multiple regions representing at least one of a specific position, a specific orientation, and a specific size of the eye, wherein the image processing device changes the display mode of any one or all of the multiple regions based on the eyeball tracking result, and changes at least one of the position, orientation, and size of the boundaries.

[0013] A surgical microscope system according to an embodiment of the present disclosure includes: a surgical microscope for acquiring a surgical field image of a patient's eye; an image processing device for generating a display image; and a display device for displaying the display image, wherein the image processing device includes: an image input unit for receiving the surgical field image; an eye-tracking unit for tracking an eyeball in the surgical field image; and a display image generation unit for setting multiple regions with different display modes in the surgical field image, and generating a display image in which the boundaries of the multiple regions represent at least one of a specific position, a specific orientation, and a specific size of the eye, wherein the display image generation unit changes the display mode of any one or all of the multiple regions based on the eye-tracking result, and changes at least one of the position, orientation, and size of the boundaries. Attached Figure Description

[0014] Figure 1 This is a diagram illustrating an example of a schematic configuration of a surgical microscope system according to an embodiment of the present disclosure.

[0015] Figure 2 This is a diagram illustrating an example of a schematic configuration of a surgical microscope according to an embodiment of the present disclosure.

[0016] Figure 3 This is a diagram illustrating an example of a schematic configuration of an image processing apparatus according to an embodiment of the present disclosure.

[0017] Figure 4 This is an illustration of Example 1, showing a display image according to an embodiment of the present disclosure.

[0018] Figure 5 It is a first illustration used to explain the generation of display images according to embodiments of the present disclosure.

[0019] Figure 6 This is a second illustration used to explain the generation of display images according to embodiments of the present disclosure.

[0020] Figure 7 This is an illustration of Example 2, showing a display image according to an embodiment of the present disclosure.

[0021] Figure 8 This is an illustration of example 3 of a display image according to an embodiment of the present disclosure.

[0022] Figure 9 This is an illustration of example 4 of a display image according to an embodiment of the present disclosure.

[0023] Figure 10 This is an illustration of example 5, showing a display image according to an embodiment of the present disclosure.

[0024] Figure 11 This is an illustration of example 6, showing a display image according to an embodiment of the present disclosure.

[0025] Figure 12 This is an illustration of example 7, showing a display image according to an embodiment of the present disclosure.

[0026] Figure 13 This is an illustration of example 8 of a display image according to an embodiment of the present disclosure.

[0027] Figure 14 This is an illustration of example 9 of a display image according to an embodiment of the present disclosure.

[0028] Figure 15 This is an illustration of example 10 of a display image according to an embodiment of the present disclosure.

[0029] Figure 16 This is an illustration of example 11 of a display image according to an embodiment of the present disclosure.

[0030] Figure 17 This is a diagram illustrating the changes in the boundary presentation based on the tracking status according to embodiments of this disclosure.

[0031] Figure 18 This is a first diagram used to illustrate how the boundary presentation changes over time according to embodiments of the present disclosure.

[0032] Figure 19 This is a second diagram illustrating how the boundary presentation changes over time according to embodiments of the present disclosure.

[0033] Figure 20 This is a third diagram illustrating how the boundary presentation changes over time according to embodiments of the present disclosure.

[0034] Figure 21 This is a diagram illustrating the changes in the boundary presentation based on the tracking results according to embodiments of this disclosure.

[0035] Figure 22 This is an illustration of example 12 of a display image according to an embodiment of the present disclosure.

[0036] Figure 23 This is an illustration of example 13 of a display image according to an embodiment of the present disclosure.

[0037] Figure 24 This is an illustration of example 14 of a display image according to an embodiment of the present disclosure.

[0038] Figure 25 This is a fourth diagram illustrating how the boundary presentation changes over time according to embodiments of the present disclosure.

[0039] Figure 26 This is an illustration of example 15 of a display image according to an embodiment of the present disclosure.

[0040] Figure 27 This is a first illustration of example 16 of a display image according to an embodiment of the present disclosure.

[0041] Figure 28 This is a second illustration of example 16 of a display image according to an embodiment of the present disclosure.

[0042] Figure 29 This is an illustration of example 17 of a display image according to an embodiment of the present disclosure.

[0043] Figure 30 This is a diagram illustrating an example of a schematic configuration of a computer according to an embodiment of the present disclosure. Detailed Implementation

[0044] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the apparatus, method, system, etc., according to the present disclosure are not limited to these embodiments. Furthermore, in the following embodiments, substantially the same parts are labeled with the same reference numerals, and repeated descriptions are omitted.

[0045] One or more embodiments (including examples and variations) described below can be implemented independently. On the other hand, at least some of the various embodiments described below can be implemented by appropriately combining them with at least some of the other embodiments. These various embodiments may include novel features that differ from each other. Therefore, the various embodiments can help solve different purposes or problems and can exhibit different effects.

[0046] This disclosure will be described in the order shown below.

[0047] 1. Implementation Method

[0048] 1-1. Example of a schematic configuration of a surgical microscope system

[0049] 1-2. Schematic examples of surgical microscope configurations

[0050] 1-3. Examples of illustrative configurations of image processing apparatus and image processing devices

[0051] 1-4. Functions and Effects

[0052] 2. Examples of illustrative computer configurations

[0053] 3. Supplementary Explanation

[0054] <1. Implementation Method>

[0055] <1-1. Example of a schematic configuration of a surgical microscope system>

[0056] Reference Figure 1 An example of a schematic configuration of a surgical microscope system 1 according to an embodiment is described. Figure 1 This is a diagram illustrating an example of a schematic configuration of a surgical microscope system 1 according to an embodiment.

[0057] like Figure 1 As shown, the surgical microscope system 1 includes a surgical microscope 10 and a surgical bed 20. The surgical microscope system 1 is a system for eye surgery. The patient lies on the surgical bed 20 to undergo eye surgery. Furthermore, the surgeon, acting as a physician, performs the surgery while observing the patient's eye using the surgical microscope 10.

[0058] The surgical microscope 10 includes an objective lens 11, an eyepiece 12, an image processing device 13, and a monitor 14.

[0059] Objective lens 11 and eyepiece 12 are lenses used to magnify and observe the eyes of the patient to be operated on.

[0060] The image processing device 13 performs predetermined image processing on the image captured by the objective lens 11 to output various images and various information.

[0061] The monitor 14 displays images captured by the objective lens 11, various images generated by the image processing device 13, and various information. The monitor 14 can be set up separately from the surgical microscope 10.

[0062] In the surgical microscope system 1, for example, the surgeon performs surgery while viewing the patient's eyes through the eyepiece 12 and observing them through the objective lens 11. Furthermore, the surgeon performs surgery while reviewing the surgical field image, various images (e.g., images before image processing, images after image processing, etc.), and various information displayed on the monitor 14. Alternatively, surgery can be performed using only the images from the monitor 14.

[0063] <1-2. Examples of illustrative configurations of surgical microscopes>

[0064] Reference Figure 2 An example of an illustrative configuration of the surgical microscope 10 according to this embodiment is described. Figure 2 This is a diagram illustrating an example of a schematic configuration of the surgical microscope 10 according to this embodiment.

[0065] like Figure 2 As shown, in addition to the objective lens 11, eyepiece 12, image processing device 13, and monitor 14, the surgical microscope 10 also includes a light source 51, an observation optical system 52, a front image capture unit 53, a tomographic image capture unit 54, a presentation unit 55, an interface unit 56, and a speaker 57. Furthermore, the monitor 14 and the presentation unit 55 correspond to display devices.

[0066] The light source 51 emits illumination light to illuminate the patient's eyes under the control of the control unit 13A included in the image processing device 13.

[0067] The observation optical system 52 includes optical elements such as an objective lens 11, a semi-reflective mirror 52a, and a lens (not shown). The observation optical system 52 guides the light reflected from the patient's eye (observation light) to the eyepiece 12 and the front image capturing unit 53.

[0068] Specifically, the light reflected from the patient's eye, as observation light, enters the semi-reflective mirror 52a via the objective lens 11, a lens (not shown), etc. Approximately half of the observation light incident on the semi-reflective mirror 52a passes through it as is and is incident on the eyepiece 12 via the transmissive presentation unit 55. On the other hand, the remaining half of the observation light incident on the semi-reflective mirror 52a is reflected by the semi-reflective mirror 52a and incident on the front image capture unit 53.

[0069] For example, the front image capture unit 53 includes a camera or the like. The front image capture unit 53 receives and photoelectrically converts the observation light incident from the observation optical system 52, thereby capturing an image of the patient's eye viewed from the front (i.e., an image obtained by capturing the patient's eye approximately from the direction of the eye axis) (i.e., the front image). Under the control of the image processing device 13, the front image capture unit 53 captures (captures) the front image and provides the acquired front image to the image processing device 13.

[0070] The tomographic image acquisition unit 54 includes, for example, optical coherence tomography (OCT), a Schemimpflug camera, etc. Under the control of the image processing device 13, the tomographic image acquisition unit 54 captures (images of cross-sections of the patient's eye) and provides the acquired tomographic images to the image processing device 13. Here, the tomographic image is an image of a cross-section in a direction approximately parallel to the axial direction of the patient's eye.

[0071] It should be noted that, for example, the tomographic image acquisition unit 54 uses infrared light to acquire tomographic images through the principle of interference, but at this time, the optical path of the infrared light and part of the optical path of the observation light in the observation optical system 52 can be a common optical path.

[0072] The eyepiece 12 converges the observation light incident from the observation optics system 52 through the presentation unit 55 to form an optical image of the patient's eye. As a result, the optical image of the patient's eye is observed by the surgeon who is viewing through the eyepiece 12.

[0073] The presentation unit 55, which includes a transmissive or non-transmissive display device, is disposed between the eyepiece 12 and the observation optical system 52. The presentation unit 55 transmits observation light incident from the observation optical system 52 and directs the observation light onto the eyepiece 12. It also presents (displays) various images (e.g., frontal images, tomographic images, etc.) and various information provided by the image processing device 13 as needed. For example, various images and information can be superimposed and presented on the optical image of the patient's eye, or various images and information can be presented in the peripheral portion of the optical image without interfering with the optical image. Furthermore, the transmissive and non-transmissive display devices are configured to be switchable and can be switched as needed. For example, there are transmissive and non-transmissive modes, and these modes can be changed by the surgeon or others, switching between the transmissive and non-transmissive display devices.

[0074] The image processing device 13 includes a control unit 13A that controls the operation of the entire surgical microscope 10. For example, the control unit 13A changes the illumination conditions of the light source 51 or changes the zoom magnification of the observation optical system 52. Furthermore, the control unit 13A controls the image acquisition of the front image acquisition unit 53 and the tomographic image acquisition unit 54 based on the operating information of the surgeon or others provided from the interface unit 56.

[0075] The interface unit 56 includes, for example, a communication unit. It receives commands from operation units such as a touch panel, foot switch, controller, and remote control superimposed on the monitor 14, or communicates with external devices. The interface unit 56 provides information corresponding to the surgeon's operations to the image processing device 13. Furthermore, the interface unit 56 outputs device control information provided by the image processing device 13 for controlling external devices to the external devices.

[0076] The monitor 14, in response to the control unit 13A of the image processing device 13, displays various images and information, such as frontal images, on the display screen.

[0077] In response to the control unit 13A of the image processing device 13, for example, if a dangerous situation is detected during surgery, the speaker 57 outputs sounds such as a buzzer, a melody, or a message (voice) to notify the surgeon of the dangerous situation. Note that the surgical microscope 10 may include a rotating light or indicator light for notifying the surgeon of dangerous situations.

[0078] In the surgical microscope system 1 with the above configuration, by using a display screen that presents one or both of a specific position and a specific size (relative to a specific position and a specific size of the eye) based on the preoperative plan due to the different display modes of the boundaries of multiple regions, occlusion does not occur. The surgeon can easily see the surgical field image and also grasp the aforementioned specific position or specific size, enabling high-precision execution of ophthalmic surgery according to the preoperative plan. The difference in display mode refers to the difference in parameters related to display conditions, such as brightness difference, saturation difference, color temperature difference, color difference, contrast difference, sharpness difference, etc.

[0079] <1-3. Illustrative configuration of image processing device and examples of image processing>

[0080] Reference Figure 3 An illustrative configuration of the image processing apparatus 13 according to an embodiment and an example of image processing are described. Figure 3 This is a diagram illustrating an example of a schematic configuration (configuration and processing flow) of the image processing apparatus 13 according to an embodiment.

[0081] like Figure 3As shown, the image processing device 13 includes a preoperative planning receiving unit 13a, an image input unit 13b, a registration unit 13c, an information storage unit 13d, an eye tracking unit 13e, and a display image generation unit 13f.

[0082] The preoperative planning receiving unit 13a receives preoperative planning information for the patient's eye (e.g., preoperative images of the preoperative plan, posture information, etc.). The posture information of the preoperative plan includes information (size information, position information, orientation information, etc.) regarding indicators (surgery-related indicators) based on locations such as the limbus in the preoperative images, the position of the indicators, and the direction of the indicators around the eye axis (the position of the rotation direction around the eye axis). For example, the size, position, and direction of the indicators around the eye axis include the location, shape, and size of the incision, as well as the position and orientation of the implant to be inserted, such as an intraocular lens. It should be noted that the direction around the eye axis is defined by the angle of the rotation direction around the eye axis relative to a reference line orthogonal to the eye axis. However, the position of the indicator's coordinate system and the position of its rotation direction around the eye axis correspond to the indicator's position information (position information for a specific location).

[0083] Image input unit 13b receives the image from front image capture unit 53 (see reference). Figure 2 The system receives surgical field images (front view) and provides the received surgical field images (e.g., surgical field images at the start of surgery, real-time surgical field images during surgery, etc.) to the registration unit 13c, eye-tracking unit 13e, display image generation unit 13f, etc.

[0084] Registration unit 13c compares the preoperative image of the preoperative plan with the surgical field image at the start of the surgery to obtain the correspondence between the preoperative image of the preoperative plan and the surgical field image at the start of the surgery, such as the transformation parameters from the preoperative image to the surgical field image at the start of the surgery (e.g., coordinate transformation parameters). Then, registration unit 13c provides the obtained relationship information about the transformation parameters, together with the surgical field image at the start of the surgery, to information storage unit 13d.

[0085] The information storage unit 13d, based on the relationship information provided from the registration unit 13c and the surgical field image at the start of the surgery, transforms (changes) the posture information of the preoperative plan according to the surgical field image at the start of the surgery, and stores the surgical field image at the start of the surgery and the posture information of the preoperative plan after transformation according to the surgical field image at the start of the surgery.

[0086] The eye-tracking unit 13e tracks the eye in the real-time surgical field image by comparing the surgical field image at the start of surgery with the real-time surgical field image. Furthermore, the eye-tracking unit 13e provides the display image generation unit 13f with relationship information (e.g., transformation parameters) representing the relationship between the pose information of the eye in the real-time surgical field image and the pose information of the preoperative plan stored in the information storage unit 13d, as the tracking result. Similar to the pose information of the preoperative plan, the pose information of the eye includes information (size information, position information, orientation information, etc.) related to the size of the eye, the position of the eye, and the direction of the eye around the eye axis (position in the rotational direction around the eye axis). However, the position in the eye's coordinate system and the position in the rotational direction around the eye axis both correspond to the position information of the eye.

[0087] The display image generation unit 13f processes the real-time surgical field image, sets multiple regions with different display modes for the real-time surgical field image, and generates a display image in which the boundaries of each region represent a specific position or size relative to the patient's eye. At this time, the display image generation unit 13f processes the real-time surgical field image (i.e., each region) according to the transformed pose information of the pre-operative plan, so that the boundaries of each region represent a specific position or size. Furthermore, the display image generation unit 13f processes the real-time surgical field image based on the eye tracking results of the real-time surgical field image, changes the pose (position, orientation, size, etc.) of the boundaries of each region, and generates a display image. At this time, the display image generation unit 13f processes the position and size of the boundaries in the real-time surgical field image based on the relationship information provided by the eye tracking unit 13e, so that the relationship between the position and size of the boundaries remains unchanged relative to the eye in the surgical field image at the start of the surgery, and generates a display image.

[0088] (Example 1 showing an image)

[0089] Example 1 of a display image according to an implementation method will be described. Figure 4 This is an illustration of Example 1, showing a display image according to an embodiment.

[0090] like Figure 4As shown, in the displayed image, a boundary K1 is presented between two regions (left region and right region) with different display modes. Boundary K1 represents a specific location based on preoperative planning or other plans, i.e., a specific location related to the surgery. The boundary K1 is transformed (changed) by shifting the aforementioned amount of movement in the direction of eye movement, thereby eliminating the change in the pose of boundary K1 relative to the eyeball based on the direction and amount of eye movement. The displayed image with this boundary K1 is displayed on the display screen by either the monitor 14 or the presentation unit 55. It should be noted that when the presentation unit 55 displays a displayed image with boundary K1, the transmissive display device is switched to a non-transmissive display device, and the non-transmissive display device is used.

[0091] exist Figure 4 In the example, boundary K1 is a linear boundary that passes through the central location of the intraocular lens B1, such as a toric IOL (toric intraocular lens), intended for placement to perform astigmatism correction. Boundary K1 represents the boundary line used for aligning the intraocular lens B1 (the target location for mounting the intraocular lens B1). Two markings B1a on the intraocular lens B1 (e.g., three points arranged on a straight line) are aligned with boundary K1. For example, in the case of an astigmatic intraocular lens B1, the toric axis of the intraocular lens B1 must be aligned with the patient's astigmatic axis, and sufficient astigmatism correction cannot be obtained if there is a deviation in the direction around the eye axis (position in the direction of rotation around the eye axis). Therefore, the two markings B1a indicating the toric axis are imprinted on the endpoints of the astigmatic intraocular lens, and the direction of the astigmatic intraocular lens around the eye axis is determined. During the procedure, the marker B1a of the astigmatic intraocular lens is aligned with the boundary K1 in the real-time surgical field image, and the astigmatic intraocular lens is placed in the eye.

[0092] In many ophthalmic surgical guidance systems, preoperative images and images at the start of surgery (surgical field images at the start of surgery) are registered. Then, markers based on the preoperative plan are mapped and displayed on the real-time images by comparing (tracking) the images at the start of surgery and the real-time surgical field images. However, in this method, because the markers based on the preoperative plan are superimposed on the surgical field images, occlusion (masking) occurs, causing a portion of the surgical field image to be invisible due to the markers.

[0093] Therefore, it is difficult for the surgeon to see the surgical field image, making it difficult to perform tasks such as axial alignment of the astigmatic intraocular lens (IOL), IOL centering, incision, and anterior capsulotomy. Consequently, it is difficult to perform the surgery accurately according to the preoperative plan. Therefore, by presenting the specific location or size of the surgery through the boundaries of each region (e.g., boundary K1) with different display modes, rather than through markings, occlusion can be prevented, and the surgery can be performed accurately according to the preoperative plan.

[0094] (Detailed processing procedure)

[0095] The specific processing steps are as follows: registration, tracking, image generation, and image rendering are executed sequentially. Figure 5 and Figure 6 This is a diagram used to illustrate the generation (transformation processing) of a display image according to an embodiment.

[0096] During registration, preoperative planning information, including preoperative images (e.g., images of the eyeball, etc.), and images at the start of surgery (surgical field images at the start of surgery), are received. The orientation and position of the eyeball at the time of preoperative planning are matched with a reference for the orientation and position of the eyeball during surgery. As an alignment method, alignment can be performed automatically, for example, by image processing using the preoperative and start-of-surgery images, or alignment can be performed by the user manually adjusting the position and orientation (e.g., rotation angle, etc.). Upon successful registration, the surgical planning information based on the coordinates of the image at the start of surgery is maintained.

[0097] In the tracking process, the movement of the eyeball is tracked from the start of surgery, and transformation parameters (coordinate transformation parameters) are obtained from the image at the start of surgery to the coordinates at the time of tracking. As a tracking method, for example, feature points can be extracted from both the preoperative image and the image at the start of surgery, and the transformation parameters can be obtained from the correspondence; alternatively, feature points can be extracted from an image, and the movement of these feature points can be searched to obtain the transformation parameters; or the image can be input into a machine learning program, and the transformation parameters can be obtained based on the learned data.

[0098] In image generation, based on the transformation parameters extracted during tracking, such as Figure 5 As shown, the control point coordinates P1 of the region (e.g., modulation region, etc.) indicating a specific location boundary K1 is generated from the surgical plan information obtained through registration, or as... Figure 6 As shown, the control point coordinates P1 of the boundary line indicating boundary K1 are transformed. Subsequently, when the transformation target is the control point coordinates P1 of a region, image processing is performed on the region; and when the transformation target is the control point coordinates P1 of the boundary line, image processing is performed on the region based on the boundary line to produce a display image. This display image includes multiple regions with different display modes.

[0099] exist Figure 5 In this example, the control point coordinates P1 representing the area are four points, but P1 can be three or more points. The area enclosed by these control point coordinates P1 is the processing object. Furthermore, in Figure 6 In the example, the control point coordinates P1 representing the boundary line are two points, but there can be more than two control point coordinates P1. When the boundary has a complex shape, multiple control point coordinates can be added, i.e., multiple control points. When determining the boundary line, one of the left and right regions (or one of the upper and lower regions) of the boundary line is processed. Note that the control point coordinates P1 representing the region or boundary can be connected by straight lines, smoothly connected by spline interpolation, or connected by specific shapes such as semicircles passing through two points. Furthermore, the number of regions to be processed is not limited to one; there can be multiple regions.

[0100] In the image display, the display image generated through image generation processing is displayed as the surgical field video. The display image has boundary K1 (see...). Figure 4 It is displayed on the display screen by one or both of the monitor 14 and the presentation unit 55.

[0101] Here, coordinate transformations, such as affine transformations and homography transformations, can be used. Furthermore, image processing is achieved by adjusting parameters such as luminance (brightness), contrast (shadow), saturation, color temperature, sharpness, and grayscale, and by changing a specific color to another specific color (i.e., by changing the pixel values ​​of the image). Specifically, for example, processing based on calculation formulas (e.g., nonlinear calculations, such as gain adjustment, offset processing, and / or gamma processing), processing via lookup tables (e.g., changing from a specific color to a specific color, converting from a specific luminance value to a specific luminance value to change the contrast), and processing via spatial filters can be used individually or in combination. At this time, the display image generation unit 13f can automatically select and execute processing to make the boundaries more distinct relative to the original surgical field image (original image). Note that, as an example of a transformation from a specific luminance value to a specific luminance value, there is an S-shaped curve change in contrast.

[0102] In image processing, for example, regarding luminance, specific channels are modified. Furthermore, regarding contrast, for example, a gain corresponding to the value of a specific channel is added. Regarding saturation, for example, a uniform gain is added to a specific channel. Regarding color temperature, a different uniform gain is added to each channel. Regarding grayscale, for example, specific channel values ​​are changed. Regarding color variations, for example, a transformation is performed based on pixel values.

[0103] Note that images, for example, have color information in the form of channels. RGB images have three channels: Red, Green, and Blue. Additionally, HSL images include three channels: Hue, Saturation, and Lightness / Luminance (or Intensity). Furthermore, CMYK images include four channels: Cyan, Magenta, Yellow, and Black.

[0104] Furthermore, in image generation, the information mode to be presented and the processing method can be changed based on instructions from users such as surgeons. Examples of information modes to be presented include various information modes (various display modes) corresponding to wounds, anterior capsular incisions, axial alignment of astigmatic intraocular lenses (IOLs used for astigmatism correction), and centering of the IOL. For example, users can select the information mode or processing method to be presented by operating the operation unit.

[0105] Furthermore, since tracking consumes processing time, generating the surgical field image after generating the tracking information increases the delay between the image and reality. Therefore, the delay in image generation can be suppressed by closing the eyes when the tracking information is delayed and using the most recently calculated past tracking information in image generation.

[0106] (Examples 2 to 6 of the images are shown)

[0107] Reference Figures 7 to 11 Examples 2 to 6 describe display images according to the implementation method. Figures 7 to 11 These are illustrations of examples 2 to 6 showing display images according to embodiments. In examples 2 to 6, variations of the display images will be described.

[0108] like Figure 7 As shown, two boundaries, K2 and K3, are presented to indicate the center position in the displayed image. Figure 7 In the example, the intersection of boundaries K2 and K3 indicates, for example, the center location for IOL installation (e.g., axial length position, etc.). Additionally, in Figure 7 In the example, the right region of boundary K2 in the surgical field image is processed, and the lower region of boundary K3 is processed.

[0109] like Figure 8 As shown, the boundary K4, indicating the location of the cut, is presented in the displayed image. Figure 8 In the example, boundary K4 is the two sides of the triangle, and the vertices of the triangle indicate the cut location (e.g., the cut start location, etc.). Furthermore, in Figure 8 In the example, the lower region (triangular region) of the boundary K4 in the surgical field image is processed.

[0110] like Figure 9As shown, the displayed image presents two boundaries, K5 and K6, indicating the location of the cut. Figure 9 In the example, the intersection of boundaries K5 and K6 indicates the cut location (e.g., the cut start point, etc.). Additionally, in Figure 9 In the example, the right region of boundary K5 in the surgical field image is processed, and the lower region of boundary K6 is processed.

[0111] like Figure 10 As shown, the boundary K7, indicating the size and location of the cut, is presented in the displayed image. Figure 10 In the example, boundary K7 indicates, for example, the size and location of the incision (e.g., a continuous incision location, etc.) of a CCC (anterior capsulotomy). Boundary K7 is used, for example, as a boundary with a semi-circular shape (i.e., a semi-circular boundary (a semi-circle used to form the target circle of the anterior capsulotomy)). Additionally, in Figure 10 In the example, the region to the right of boundary K7 in the surgical field image is processed. Note that in addition to the axial length as the center of the annular shape such as the target circle mentioned above, the limbus center, pupil center, preoperative pupil center, visual axis, and anterior capsular incision edge center can also be used.

[0112] like Figure 11 As shown, the image displays a boundary K8 indicating a specific region (i.e., the size and location of that region). Figure 11 In this example, the boundary K8 is a hexagon (or quadrilateral) and indicates the size and location of a specific region. For instance, in the case of a lost eye during tracking, the boundary K8 indicates to the surgeon that the eye has reached the center of the surgical field image. Additionally, in... Figure 11 In the example, the outer region of boundary K8 in the surgical field image is processed.

[0113] (Examples 7 to 9 of the images are shown)

[0114] Reference Figures 12 to 14 Examples 7 to 9 describe display images according to the embodiments. Figures 12 to 14 These are illustrations of examples 7 to 9 showing the displayed images according to the embodiments. In examples 8 and 9, additional points will be described to reduce the difference between the image before and after processing while maintaining the sharpness of the boundary K1.

[0115] like Figure 12As shown, the region to which image processing is applied can be a region on one side of boundary K1. When the region to which image processing is applied is a region on one side of boundary K1, the amount of change in the region on the side to which image processing is applied is larger, while the region on the side on which image processing is not applied is preserved. That is, in the case of the region on the processed side, since modulation is performed at the level where boundary K1 is visible, the advantage is that although the difference between the processed and original images increases in the processed region, there are also regions that are not processed.

[0116] like Figure 13 As shown, the region to which image processing can be applied can be the regions on both sides of boundary K1. Figure 13 In the example, one side of the region (the upper region of boundary K1) is 10% brighter than the original image (the surgical field image before processing), while the other side (the lower region of boundary K1) is 10% darker. By applying different processing to the two sides in this way, the amount of variation from the original image can be reduced while maintaining the clarity of the boundary gap. That is, even though there is no unprocessed region when processing the regions on both sides, the same boundary gap can be achieved by processing the regions on both sides. Therefore, it has the advantage of keeping the difference between the original image and the processed surgical field image within a small range.

[0117] like Figure 14 As shown, information can be transmitted when there is a difference (difference in display mode) between the regions on either side of boundary K1. Therefore, the processing intensity (e.g., modulation intensity) for a region can decrease with increasing distance from boundary K1. Figure 14 In the example, in the region to the left of boundary K1, the processing intensity for that region decreases with increasing distance from boundary K1. For instance, if the processing of the region is for increasing brightness, when the intensity of the processing for increasing the brightness of the region decreases with increasing distance from boundary K1, the brightness of the region also decreases with increasing distance from boundary K1. As described above, by weakening the processing intensity for that region with increasing distance from boundary K1, the difference between the original image and the processed surgical field image can be reduced in the region far from boundary K1. That is, the sharpness of the gap at boundary K1 can be maintained, and the region far from boundary K1 can be made closer to the original image.

[0118] (Examples 10 and 11 of the images are shown)

[0119] Reference Figure 15 and Figure 16 Examples 10 and 11 of the display images according to the implementation method are described. Figure 15 and Figure 16These are illustrations of examples 10 and 11 showing display images according to embodiments. In examples 10 and 11, additional points in the presentation of 3D images (three-dimensional surgical field images) will be described. Typically, 3D images are frequently used in ophthalmic surgery. In this case, there are stereoscopic left-eye images and stereoscopic right-eye images, allowing depth perception to be presented as a parallax difference. Therefore, additional points for the boundary presentation of the stereoscopic left-eye images and stereoscopic right-eye images will be described. It should be noted that, hereinafter, "for stereoscopic left eye" is simply referred to as "for left eye," and "for stereoscopic right eye" is simply referred to as "for right eye."

[0120] like Figure 15 As shown, regarding the boundary rendering of 3D images, there are cases where the boundary is rendered in both the left and right eye images, i.e., boundary K2 is rendered in the right eye image and boundary K3 is rendered in the left eye image, and there are also cases where boundaries K2 and K3 are rendered in only one of the left and right eye images (see...). Figure 7 Note that even though boundaries K2 and K3 are only presented in the image used for one eye, there is an advantage that 3D perception is essentially unaffected and the surgeon can visually identify boundaries K2 and K3 because the image changes from the original image are small. Therefore, boundaries K2 and K3 can be presented only in the image used for one eye, or different boundaries (e.g., boundaries K2 and K3) can be presented in the image used for the left eye and the image used for the right eye.

[0121] exist Figure 15 In the example, different boundaries K2 and K3 are presented in the left-eye and right-eye images (boundary K2 is presented in the right-eye image, and boundary K3 is presented in the left-eye image), and the intersection of boundaries K2 and K3 is presented by fusing boundaries K2 and K3 in the brain. In this way, for example, to present the centering position and the wound position, these two boundaries K2 and K3 can be indicated by the intersection position, etc. Furthermore, information can be added that can be presented through combinations of various boundaries (e.g., boundaries K2 and K3).

[0122] Note that in cases where different boundaries K2 and K3 are presented in the left-eye and right-eye images, different boundaries K2 and K3 can be presented in the left-eye and right-eye images based on information tracked in one-eye images, and different boundaries K2 and K3 can be presented in the left-eye and right-eye images based on information tracked in both eye images, which are left-eye and right-eye images.

[0123] Furthermore, in addition to presenting different boundaries K2 and K3 in both left-eye and right-eye images, the same boundary (e.g., boundary K1) can be presented in both left-eye and right-eye images. When the same boundary is presented in both left-eye and right-eye images, depth perception occurs relative to the boundary, and therefore, the location of the boundary can be controlled. For example, if the ophthalmic surgery is a CCC (anterior capsule incision), the boundary is located at the location of the anterior capsule.

[0124] like Figure 16 As shown, for boundary K1, the process of creating depth gaps in the 3D image can be performed. Figure 16 In this example, as part of image processing, pixels in the left-eye image are shifted to the right, and pixels in the right-eye image are shifted to the left. After processing (after parallax modulation), the fixed position in depth changes due to the altered parallax. The surgeon perceives the boundary K1 as bulging forward. Therefore, the surgeon can easily determine the position of the boundary K1, enabling precise ophthalmic surgery.

[0125] Note that when the intensity of image processing (e.g., modulation, etc.) is low, unlike issues such as marker overlap, no flickering or parallax occurs, even when processing is applied to the image of only one eye or different processing is applied to the individual images of both eyes. Therefore, there is no conflict between the surgical field and depth. Furthermore, when the same processing is applied to the individual images of both eyes, parallax can be generated to locate the desired specific depth position, and it can also be guided to locate the position the user wants to treat.

[0126] (Changes in the appearance of the boundary)

[0127] Additional points regarding changes in boundary presentation based on tracking status or time according to this implementation will be described. First, reference will be made to... Figure 17 Describe the changes that occur based on the boundaries of the tracking state. Next, refer to... Figures 18 to 20 Describe the changes that occur according to time boundaries. (Refer to...) Figure 21 Describe the changes in the boundary (changes in eye size) based on the tracking results. Figure 17 This is a diagram used to illustrate the changes in the boundary presentation based on the tracking state according to the implementation method. Figures 18 to 20 These are the first to third illustrations used to explain how the boundary presentation changes over time according to the implementation method. Figure 21 This is a diagram used to illustrate the changes in the boundary presentation based on the tracking results according to the implementation method.

[0128] In the principle of eye tracking, tracking accuracy deteriorates when the amount of eyeballs appearing in the surgical field decreases. Furthermore, to maintain tracking accuracy, a detection limit can be set. That is, there may be situations where the confidence level of tracking accuracy is low, or where the tracking detection limit is approaching. In such cases, such as... Figure 17 As shown, the processing intensity of a predetermined area can be reduced (e.g., modulation) to make the image of the predetermined area closer to the original image, or the processing intensity of the predetermined area can be increased to make the image of the predetermined area stand out as a warning (warning message). Of course, messages can also be overlaid with these messages and presented at a fixed position on the display screen. In this way, by presenting the above tracking status to users such as surgeons, users can assess the reliability of the tracking.

[0129] Furthermore, in principle, the presentation of information based on the boundaries K1 and K7 of each region should not cause occlusion of the surgical field. However, some surgeons may experience reduced visibility at the boundaries K1 and K7. Therefore, as... Figure 18 and Figure 19 As shown, the positions of boundaries K1 and K7, which present the same information (at a specific location), can be changed.

[0130] exist Figure 18 In the example, the boundary K7 rotates 360 degrees around the axial length of the eye. For instance, in a procedure such as anterior capsulotomy, the boundary K7 rotates at a predetermined speed (e.g., a speed higher than the speed at which the surgeon moves the distal portion of the surgical instrument) from the start of the procedure. This rotation is repeated during the procedure. Thus, the boundary K7 forms the target circle for the incision in the anterior capsule. In this way, visibility can be improved by changing the position of the boundary K7, which presents the same information.

[0131] Here, the predetermined speed is set in advance, for example, a value faster than the average speed at which the surgeon moves the tip of the surgical instrument. However, the rotation speed of the boundary K7 may not be a predetermined speed, and for example, the boundary K7 can be rotated according to the movement of the surgical instrument or the endpoint of the anterior bursal incision edge by detecting the distal end of the surgical instrument moved by the surgeon. Furthermore, the rotation angle of the boundary K7 may be, for example, another angle, such as 180 degrees.

[0132] exist Figure 19 In the example, boundary K1 is translated at a predetermined speed. Boundary K1 moves a predetermined distance to the left from the reference position, returns to the reference position from the left predetermined position, further moves a predetermined distance to the right from the reference position, and returns to the reference position from the right predetermined position. This movement is repeated during the operation (periodic movement). As a result, visibility can be improved by changing the position of boundary K1, which presents the same information.

[0133] In addition, such as Figure 20 As shown, visibility can be improved by periodically reducing the intensity of processing in a predetermined area (e.g., modulation). Figure 20 In the example, the processing intensity gradually decreases from the initial state, then gradually increases, and then returns to the initial state. As described above, visibility can be improved by periodically reducing the processing intensity (the degree to which the image changes from the original image) in a predetermined region.

[0134] In this way, by periodically reducing the processing intensity (the degree to which the image changes from the original image) of a predetermined area, the visibility of the surgical field can be further improved. Furthermore, by periodically shifting the presentation position of the boundary K1, such as the toric axis, it is possible to avoid the boundary K1 clashing with the marker B1a of the astigmatic intraocular lens (see...). Figure 4 The overlap makes the B1a marker difficult to see. In addition, since the tracking accuracy may decrease as the location of the cornea to be tracked approaches the endpoint, the visibility of the surgical field can be enhanced, and the user can be made aware of the high tracking accuracy of bringing the cornea to the center by weakening the intensity of the treatment of the predetermined area.

[0135] It should be noted that, as described above, the positions of boundaries K1 and K7 can be changed at predetermined intervals or switched according to user instructions. For example, the user can switch the positions of boundaries K1 and K7 by operating a control unit such as a touch panel, foot switch, or controller.

[0136] Furthermore, image processing can be performed on the predetermined area as described above, or operation units such as surgical tools operated by the surgeon (e.g., the tip of a surgical tool) can be detected, and image processing (e.g., brightness modulation processing, color modulation processing, etc.) can be performed on areas that do not include operation units.

[0137] In addition, such as Figure 21 As shown, the position and size of boundary K7 can change according to the size of the eye. Since a portion of boundary K7 has a semi-circular shape and indicates the size of the eye, the position of boundary K7 changes according to the size of the eye. For example, when the size of the eye in the surgical field image decreases, the size of the semi-circle of boundary K7 also decreases, and the position of boundary K7 is closer to the center. Conversely, when the size of the eye in the surgical field image increases, the size of the semi-circle of boundary K7 also increases, and the position of boundary K7 is further away from the center. Note that examples of size include the diameter of the CCC, the incision width, and the center.

[0138] (Examples 12 to 14 of the images are shown)

[0139] Reference Figures 22 to 24 Examples 12 to 14 describe display images according to the embodiments. Figures 22 to 24These are illustrations of examples 12 to 14 showing display images according to embodiments.

[0140] like Figure 22 As shown, boundary K7 appears in two specific locations in the displayed image. Figure 22 In the example, each of the two specific locations indicates wound information (e.g., the location where the wound was created). For example, a portion of boundary K7 forms a triangle, and the vicinity of the vertices is the location where the wound was created. Figure 22 In the example, the right region of boundary K7 in the surgical field image is processed.

[0141] like Figure 23 As shown, the boundary K11 in the image represents the width and location of the main wound. Figure 23 In the example, boundary K11 indicates the incision width as the width of the limbus of the triangle formed from the center of the cornea. The location of the main wound, i.e., the location of the incision, is the virtual perpendicular bisector of the triangle. Figure 23 In the example, the left region of boundary K11 in the surgical field image is processed.

[0142] like Figure 24 As shown, in the displayed image, boundary K11 represents the width of the main incision, and boundary K12 represents the position of the main incision. Figure 24 In the example, boundary K11 indicates the incision width, which is the width along the limbus of the triangle formed by the corneal centers. Boundary K12 indicates the location of the main incision, i.e., the virtual perpendicular bisector of the triangle. Figure 24 In the example, the left region of boundary K11 in the surgical field image is processed, and the lower region of boundary K12 is processed. Note that a left-eye image including boundary K11 and a right-eye image including boundary K12 can be presented, and boundaries K11 and K12 can be fused in the brain to achieve a 3D image (see...). Figure 16 ).

[0143] (Changes in the appearance of the boundary)

[0144] Reference Figure 25 Describe the changes that occur based on time boundaries according to the implementation method. Figure 25 This is a fourth diagram used to illustrate how the boundary presentation changes over time according to the implementation method.

[0145] like Figure 25As shown, the width and position of the boundary K11 in the displayed image can be periodically changed. For example, when the width of the boundary K11 (the size of the area within the boundary K11) becomes maximum, the desired incision width is set, and the width of the boundary K11 is narrowed to indicate the incision location. That is, if the width of the incision is indicated only by the width within the limbus, the incision location may be difficult to understand. Therefore, the width of the boundary K11 can be periodically changed to narrow the boundary K11 and indicate the incision location.

[0146] <Luminosity Zone>

[0147] Here, examples 15 to 17 of displayed images will be described in the case where the boundaries M3 to M5 are represented by multiple luminance regions with different luminance.

[0148] (Example 15 showing an image)

[0149] Figure 26 This is an illustration of example 15 of a display image according to an embodiment. For example... Figure 26 As shown, two luminance regions with different luminance are defined, and the boundary M3 of these luminance regions is presented. Boundary M3 serves as a linear boundary, i.e., a line boundary (the target line for mounting the intraocular lens B1). Figure 26 In the example, the right luminance region among the two luminance regions ( Figure 26 The luminance of the shaded area (in the center) is set to be lower than that of the left luminance area. The toric axis is aligned with boundary M3, and the astigmatic intraocular lens is installed. Note that the number of luminance areas is not limited to two, and can be two or more.

[0150] (Example 16 showing an image)

[0151] Figure 27 and Figure 28 This is an illustration of example 16 of a display image according to an embodiment. For example... Figure 27 and Figure 28 As shown, two luminance regions with different luminance are set, and the boundary M4 between the luminance regions is presented. Boundary M4 is a boundary with a semicircular shape, that is, a semicircular boundary (a semicircle used to form the target circle of the anterior bursa incision). Figure 27 and Figure 28 In the example, the boundary M4 of the luminance region is rotated 90 degrees around the eye axis.

[0152] For example, in a procedure such as anterior capsulotomy, from the start of the surgery, the boundary M4 of the luminosity region rotates 360 degrees around the axial length of the eye at a predetermined speed (e.g., the speed at which the surgeon moves the distal portion of the surgical instrument). Thus, the boundary M4 forms a target circle for the anterior capsulotomy. This predetermined speed is a pre-set speed, such as a general value like the average speed at which the surgeon moves the distal end of the surgical instrument. However, the rotation speed of the boundary M4 may not be a predetermined speed; for example, the distal end of the surgical instrument to be moved or the endpoint of the anterior capsulotomy edge can be detected, and the boundary M4 can be rotated according to the movement of the surgical instrument or the movement of the endpoint of the anterior capsulotomy edge. The distal end of the surgical instrument and the endpoint of the anterior capsulotomy edge can be detected using the treatment start part 13g described later. Furthermore, the rotation angle of the boundary M4 can be, for example, another angle, such as 180 degrees.

[0153] (Example 17 showing an image)

[0154] Figure 29 This is an illustration of example 17 of a display image according to an embodiment. For example... Figure 29 As shown, except in Figure 27 and Figure 28 Beyond the boundary M4 shown, multiple (in) further appear. Figure 29 In the example, there are two boundaries M5. Similar to example 10 of the displayed image, boundaries M4 and M5 are formed by the boundaries of two luminance regions with different luminance. Each boundary M5 is a boundary that indicates the location of the cut.

[0155] As explained above, boundaries K1 to K12 and M3 to M5 are not overlapping marks on the surgical field image, but rather boundaries that represent visually recognizable poses (position, orientation, size, etc.). Since boundaries K1 to K12 and M3 to M5 do not hide the surgical field image at the marked location like overlapping marks, they improve the visibility of the surgical field compared to using overlapping marks.

[0156] Additionally, as described above, the surgical field image can be fixed, and the poses of boundaries K1 to K12 and M3 to M5 can be changed so that boundaries K1 to K12 and M3 to M5 are in an appropriate pose (position, orientation, etc.) relative to the fixed surgical field image (e.g., eliminating eyeball displacement in the surgical field image with boundaries K1 to K12 and M3 to M5 relative to the fixed pose). Changing the poses of boundaries K1 to K12 and M3 to M5 changes the extent (e.g., size, shape, etc.) of each region.

[0157] For example, when the poses of boundaries K1 to K12 and M3 to M5 are changed, the display image generation unit 13f generates a display image while simultaneously changing the poses of boundaries K1 to K12 and M3 to M5 according to the displacement of the eyeball based on the pose information of the eyeball. For example, the display image generation unit 13f moves boundaries K1 to K12 and M3 to M5 by the aforementioned amount of movement in the direction of eyeball movement relative to the real-time surgical field image, and changes the poses of boundaries K1 to K12 and M3 to M5 (e.g., the range of each region). That is, by fixing the surgical field image and changing the poses of boundaries K1 to K12 and M3 to M5, the positional relationship between the eyeball and boundaries K1 to K12 and M3 to M5 remains unchanged.

[0158] Furthermore, tracking processing may fail, and there may be moments when the pose (position, orientation, etc.) of the eye in the real-time image cannot be estimated. In such cases, the display image generation unit 13f can continue displaying by maintaining the pose of the image at the last time the pose could be estimated (the last displayed image). In addition to maintaining the pose of the surgical field image at the last time the pose could be estimated, the pose of the surgical field image at the time the pose could be estimated can also be maintained by constant speed, constant angular velocity, constant acceleration, or constant angular acceleration motion. Furthermore, when tracking fails, the display mode (e.g., brightness, color, etc.) of a predetermined area can be changed to identify the failure.

[0159] Although various display images are used as described above, these display images can be selected by the surgeon, staff, etc. The selection of the display image is achieved through input operations of the operating unit by the surgeon, staff, etc. For example, the surgeon, staff, etc., operates the operating unit to select a display mode for displaying the desired display image. In response to this selection, the display image generation unit 13f generates a display image based on the selected display mode. Similarly, the size, position, etc., of various images can be changed by the surgeon, staff, etc. The display image generation unit 13f generates a display image by changing the size, position, etc., of the image according to the input operations of the surgeon, staff, etc., on the operating unit.

[0160] <1-4. Functions and Effects>

[0161] As described above, in this embodiment, the image input unit 13b receives a surgical field image for the patient's eye, the eye-tracking unit 13e tracks the eyeball in the surgical field image, and the display image generation unit 13f sets multiple regions with different display modes for the surgical field image, generating a display image in which the boundaries of each region (e.g., boundaries K1 to K12 and M3 to M5) represent at least one of a specific position, a specific orientation, and a specific size relative to the eye. Furthermore, based on the eye-tracking results, the display modes of any one or all of the regions are changed, and at least one of the position, orientation, and size of the boundaries are changed. This allows occlusion to be prevented by presenting a specific position relative to the eye through the boundaries of each region with different display modes, rather than through markings. Therefore, the surgeon can easily see the surgical field image while simultaneously grasping the specific position or size, enabling accurate surgery and thus high-precision execution of the surgery according to the preoperative plan. Note that displacement includes any change relative to an object such as the eyeball, such as translation, rotation, magnification / reduction, deformation, or combinations thereof.

[0162] Furthermore, since the display image generation unit 13f changes the display mode of any one or all of the multiple regions based on eye tracking results and changes at least one of the position, orientation, and size of the boundaries, for example, processing speed can be improved when changing the display mode of any one of the multiple regions compared to changing all the display modes in the multiple regions. Moreover, compared to changing the display mode of any one of the multiple regions, when the display modes of all the multiple regions are changed, the surgeon can easily visually identify the boundaries of each region, and therefore, surgery can be performed with high precision according to the preoperative plan.

[0163] Additionally, the preoperative planning receiving unit 13a receives information on at least one of the position, orientation, and size (e.g., position, orientation, and size of coordinates) of a preoperative image and an indicator based on a preoperative plan for the patient's eye. The information storage unit 13d compares the preoperative image with the surgical field image at the start of surgery, changes at least one of the position, orientation, and size of the indicator according to the surgical field image at the start of surgery, and stores at least one of the changed position, orientation, and size information of the surgical field image and the indicator at the start of surgery. Therefore, at least one of the changed position, orientation, and size information of the surgical field image and the indicator at the start of surgery can be used for post-processing.

[0164] The eye-tracking unit 13e tracks the eye in the real-time surgical field image by comparing the surgical field image at the start of surgery with the real-time surgical field image, and outputs relational information representing the relationship between at least one piece of information (e.g., position, orientation, size, etc. of coordinates) of the eye's position, orientation, and size in the real-time surgical field image and at least one piece of information of the changed position, orientation, and size of the indicator. The display image generation unit 13f arranges boundaries based on at least one piece of information of the changed position, orientation, and size of the indicator, and changes at least one of the position, orientation, and size of the boundaries in the real-time surgical field image based on the relational information to eliminate the change in at least one of the position, orientation, and size of the boundaries relative to the eye in the surgical field image at the start of surgery, and generates a display image. Therefore, by changing at least one of the position, orientation, and size of the eye relative to the surgical field image at the start of surgery, the relationship between the eye and at least one of the position, orientation, and size of the boundaries K1 to K7 remains unchanged. Therefore, since the surgeon can have detailed control over at least one of the specific position, orientation, and size, surgery can be performed with higher precision according to the preoperative plan.

[0165] Furthermore, the display image generation unit 13f transforms multiple regions through coordinate transformation, causing at least one of the position, orientation, and size of the boundary to change based on eye tracking results, and generates a display image. As a result, it is possible to reliably change at least one of the position, orientation, and size of the boundary based on eye tracking results, and to generate a display image.

[0166] Furthermore, the display image generation unit 13f changes the boundary line indicating the boundary based on the eye tracking results, and changes at least one of the boundary's position, orientation, and size. As a result, processing speed can be improved compared to the case of changing the region. For example, the boundary line can be two control points (transformation points), but the region requires three or more control points.

[0167] Furthermore, the image generation unit 13f performs processing on two or more regions within a plurality of regions, adjusting parameters of the same type (e.g., brightness, color, etc.). This simple processing easily makes the boundaries of each region prominent. Therefore, because the surgeon can easily visually identify the boundaries of each region, surgery can be performed with high precision according to the preoperative plan.

[0168] Furthermore, the image generation unit 13f performs processing on two or more regions across multiple regions, adjusting different types of parameters (e.g., brightness, color, etc.). This simple processing easily makes the boundaries of each region distinct. Therefore, because the surgeon can easily visually identify the boundaries of each region, surgery can be performed with high precision according to the preoperative plan.

[0169] Furthermore, the image generation unit 13f reduces the intensity of processing on any one or all of the multiple regions as the region moves away from the boundary. Therefore, regions far from the boundary can be made close to the original image while maintaining boundary sharpness, and thus, surgery can be performed with high precision according to the preoperative plan.

[0170] Furthermore, the display image generation unit 13f generates a display image including boundaries as either a display image for the left eye (stereoscopic left eye) or a display image for the right eye (stereoscopic right eye), or it generates display images including boundaries separately as both a display image for the left eye and a display image for the right eye. Therefore, compared to the case where the display image including boundaries is generated as both a left-eye and a right-eye display image, processing speed can be improved when the display image including boundaries is generated as either a left-eye or a right-eye display image. Furthermore, when the display image including boundaries is generated as both a left-eye and a right-eye display image, the positional accuracy of the boundaries can be improved compared to the case where the display image including boundaries is generated as either a left-eye or a right-eye display image.

[0171] Furthermore, the eye-tracking unit 13e tracks the eye in surgical field images for one or both of the left and right eyes, and the image generation unit 13f generates images for the left eye and the right eye by changing at least one of the position, orientation, and size of the boundary based on the tracking results of the eye in one or both of the left and right eyes. As a result, when using only one of the surgical field images for the left and right eyes, the processing speed is improved compared to using both surgical field images. When using both surgical field images, the positional accuracy of the boundary is improved compared to using only one of the surgical field images.

[0172] Furthermore, the display image generation unit 13f sets boundaries at the same positions in the display images for the left eye and the display images for the right eye. As a result, the surgeon can easily grasp the position of the boundaries, thus enabling high-precision surgery according to the preoperative plan.

[0173] Furthermore, the display image generation unit 13f generates at least one different display image in terms of position, orientation, and size of the boundary between the left-eye image and the right-eye display image, to indicate at least one of a specific position, orientation, and size. Therefore, at least one of a specific position, orientation, and size can be indicated at multiple boundaries, and thus the surgery can be performed with high precision according to the preoperative plan.

[0174] Furthermore, the display image generation unit 13f displaces the display images for the left and right eyes based on depth information of the desired positioning of the three-dimensional images, including the display images for the left eye and the display images for the right eye, and generates a three-dimensional image. As a result, the surgeon can more easily grasp the position of the boundaries, thus enabling high-precision surgery according to the preoperative plan.

[0175] Furthermore, when the eye deviates from the eye tracking unit 13e's tracking of the eye, the display image generation unit 13f maintains the last displayed image before the eye deviates. Therefore, surgical interruptions due to the disappearance of the displayed image can be avoided, and thus, surgery can be performed with high precision according to the pre-operative plan.

[0176] Furthermore, the image generation unit 13f changes the display mode of any one or all of the multiple regions according to the time period. As a result, the displayed image can be made close to the original image, or the boundaries of the displayed image can be made periodically more distinct. Therefore, since the surgeon can reliably visually identify the original image and boundaries, surgery can be performed with high precision according to the preoperative plan.

[0177] Furthermore, the display image generation unit 13f generates a display image in which boundaries indicate multiple specific locations. Therefore, more specific locations can be presented, and thus, surgery can be performed with high precision according to the preoperative plan.

[0178] Furthermore, the image generation unit 13f displays the size of any one or all of the multiple regions according to changes in the size of the eyeball. Therefore, the size of one or more regions can be changed according to the size of the eyeball, and the position and size of the boundaries can be altered, enabling high-precision execution of surgery according to the preoperative plan.

[0179] Furthermore, the image generation unit 13f changes the size of any one or all of the multiple regions according to a time period. Therefore, the size of one or more regions can be changed periodically, and the position and size of the boundaries can be altered, thus enabling high-precision execution of surgery according to the pre-operative plan. Additionally, for example, by reducing the area, the displayed image can be made closer to the original image, and the surgeon can reliably visually identify the original image, allowing for high-precision execution of surgery according to the pre-operative plan.

[0180] Furthermore, the display mode for each of the multiple regions differs based on the brightness variations within each region. This makes it easy to clearly define the boundaries of each region. Therefore, because the surgeon can easily visually identify the boundaries of each region, surgery can be performed with high precision according to the pre-operative plan.

[0181] Furthermore, the specific location is the toric axis arrangement position of the intraocular lens, and the image generation unit 13f displays the change in brightness of each of multiple regions, such that the boundary indicates the toric axis arrangement position. This makes it easy to make the toric axis arrangement position prominent. Therefore, since the surgeon can easily visually identify the toric axis arrangement position, the surgery can be performed with high precision according to the preoperative plan.

[0182] Furthermore, this specific location can be any one of the following: the toricineal axis arrangement of the intraocular lens, the incision location for forceps insertion, the incision location of the anterior capsular incision, the axial length, the center of the limbus, the center of the pupil, the preoperative pupil center, the visual axis location, and the center of the anterior capsular incision edge. This allows any one of these locations to become clearly visible at the boundary of each region. Therefore, the surgeon can easily visually confirm these specific locations, enabling highly precise execution of the surgery according to the preoperative plan.

[0183] Furthermore, the display image generation unit 13f changes the display mode of any one or all of the multiple regions according to the eye tracking state of the eye tracking unit 13e. As a result, depending on the eye tracking state, the displayed image can be made close to the original image or the boundaries of the displayed image can be made clear, allowing the surgeon to grasp the eye tracking state (e.g., the reliability of tracking).

[0184] <2. Examples of illustrative computer configurations>

[0185] The above series of processes can be performed by hardware or software. In the case where the processes are performed by software, the program constituting the software is installed in the computer. Examples of computers include computers integrated into dedicated hardware, and general-purpose personal computers capable of performing various functions by installing various programs.

[0186] Figure 30 This diagram illustrates an example of a schematic configuration of a computer 500 that performs the above series of processes via a program.

[0187] like Figure 30 As shown, the computer 500 includes a central processing unit (CPU) 510, a read-only memory (ROM) 520, and a random access memory (RAM) 530.

[0188] CPU 510, ROM 520, and RAM 530 are interconnected via bus 540. Input / output interface 550 is further connected to bus 540. Input unit 560, output unit 570, recording unit 580, communication unit 590, and driver 600 are connected to input / output interface 550.

[0189] Input unit 560 includes a keyboard, mouse, microphone, imaging element, etc. Output unit 570 includes a display, speaker, etc. Recording unit 580 includes a hard disk, non-volatile memory, etc. Communication unit 590 includes a network interface, etc. Driver 600 drives removable recording medium 610 such as a hard disk, optical disk, magneto-optical disk, or semiconductor memory.

[0190] In the computer 500 configured as described above, for example, the CPU 510 loads the program recorded in the recording unit 580 into the RAM 530 via the input / output interface 550 and the bus 540 and executes the program, thereby performing the series of processes described above.

[0191] For example, a program executed by computer 500 (i.e., CPU 510) can be provided by recording it on a removable recording medium 610, such as a packaging medium. Alternatively, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0192] In computer 500, by attaching removable recording medium 610 to drive 600, a program can be installed in recording unit 580 via input / output interface 550. Alternatively, the program can be received by communication unit 590 and installed in recording unit 580 via wired or wireless transmission medium. Furthermore, the program can be pre-installed in ROM 520 or recording unit 580.

[0193] It should be noted that the program executed by the computer 500 may be a program in which processing is performed sequentially in the order described in this specification, or it may be a program in which processing is performed in parallel or at necessary timed intervals, such as when a call is made.

[0194] Furthermore, in this specification, "system" means a group of multiple components (devices, modules (components), etc.), and it is irrelevant whether all components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, as well as a device in which multiple modules are housed in one housing, are all systems.

[0195] Furthermore, the implementation of this technology is not limited to the above-described implementation, and various modifications can be made without departing from the spirit of this technology.

[0196] For example, this technology can have a cloud computing configuration, in which a function is shared and processed collaboratively by multiple devices via a network.

[0197] Furthermore, each step described in the above processing flow (e.g., flowchart) can be performed by a single device or can be shared and performed by multiple devices.

[0198] Furthermore, in cases where a step includes multiple processes, the multiple processes included in that step may be executed by a single device or may be shared and executed by multiple devices.

[0199] Furthermore, the effects described in this specification are merely illustrative and not limiting, and effects other than those described in this specification may be provided.

[0200] <3. Supplementary Explanation>

[0201] It should be noted that this technology may also have the following configurations. (1)

[0203] An image processing apparatus, comprising:

[0204] An image input unit receives surgical field images of the patient's eyes;

[0205] An eye-tracking unit that tracks the eye in the surgical field images described above; and

[0206] The display image generation unit sets multiple regions with different display modes for the surgical field image, and generates a display image whose boundaries represent at least one of the specific position, specific direction, and specific size of the eye.

[0207] Among them, the display image generation unit

[0208] Based on the eye tracking results, the display mode of any one or all of the plurality of regions is changed, and at least one of the position, orientation, and size of the boundary is changed. (2)

[0210] The image processing apparatus according to (1) further includes:

[0211] The preoperative planning receiving unit receives information on at least one of the following: the position, orientation, and size of preoperative images of the eye, and indicators based on the preoperative planning; and

[0212] An information storage unit compares the preoperative image with the surgical field image at the start of surgery, changes at least one piece of information regarding the position, orientation, and size of the indicator based on the surgical field image at the start of surgery, and stores the surgical field image at the start of surgery and at least one piece of information regarding the position, orientation, and size of the indicator after the change. (3)

[0214] According to the image processing apparatus described in (2), wherein

[0215] The eye-tracking unit

[0216] The eyeball is tracked in the real-time surgical field image by comparing the surgical field image at the start of surgery with the real-time surgical field image, and relational information is output. This relational information represents the relationship between at least one piece of information regarding the position, orientation, and size of the eyeball in the real-time surgical field image and at least one piece of information regarding the position, orientation, and size of changes in the indicators.

[0217] Display Image Generation Unit

[0218] The boundary is arranged based on at least one piece of information regarding the position, orientation, and size of the altered indicator; the position, orientation, and size of the boundary in the real-time surgical field image are changed based on the relational information to eliminate any changes in the position, orientation, and size of the boundary in the surgical field image relative to at least one piece of the eyeball at the start of surgery; and the display image is generated. (4)

[0220] The image processing apparatus according to any one of (1) to (3), wherein

[0221] Display Image Generation Unit

[0222] The coordinate transformation alters the multiple regions, causing at least one of the boundary's position, orientation, and size to change based on eye-tracking results, thereby generating a displayed image. (5)

[0224] The image processing apparatus according to any one of (1) to (3), wherein

[0225] Display Image Generation Unit

[0226] Based on eye tracking results, the boundary line indicating the boundary is changed, and at least one of the boundary's position, orientation, and size is changed. (6)

[0228] The image processing apparatus according to any one of (1) to (5), wherein

[0229] Display Image Generation Unit

[0230] Perform the process of adjusting parameters of the same type in two or more of the multiple regions. (7)

[0232] The image processing apparatus according to any one of (1) to (6), wherein

[0233] Display Image Generation Unit

[0234] Perform the process of adjusting different types of parameters in two or more of the multiple regions. (8)

[0236] The image processing apparatus according to any one of (1) to (7), wherein

[0237] Display Image Generation Unit

[0238] As the distance from the boundary increases, the processing intensity for any or all of the plurality of regions is reduced. (9)

[0240] The image processing apparatus according to any one of (1) to (8), wherein

[0241] Display Image Generation Unit

[0242] The displayed image can be generated as either a stereoscopic left-eye image or a stereoscopic right-eye image, or the displayed image can be generated as either a stereoscopic left-eye image or a stereoscopic right-eye image. (10)

[0244] According to the image processing apparatus described in (9), wherein

[0245] The eye-tracking unit

[0246] In the surgical field image, the eyeball is tracked for one or both of the stereoscopic left and stereoscopic right eyes, and

[0247] Display Image Generation Unit

[0248] Based on the tracking results of one or both of the stereoscopic left and stereoscopic right eyes, the position, orientation, and size of the boundary are changed to generate a display image for the stereoscopic left eye and a display image for the stereoscopic right eye. (11)

[0250] According to the image processing apparatus described in (9) or (10), wherein

[0251] Display Image Generation Unit

[0252] In the stereoscopic left-eye image and the stereoscopic right-eye image, the boundary is set in the same position. (12)

[0254] According to the image processing apparatus described in (9) or (10), wherein

[0255] Display Image Generation Unit

[0256] The display image is generated in which at least one of the position, orientation, and size of the boundary differs between the display image for the stereoscopic left eye and the display image for the stereoscopic right eye, so as to indicate at least one of the specific position, the specific orientation, and the specific size. (13)

[0258] The image processing apparatus according to any one of (9) to (12), wherein

[0259] Display Image Generation Unit

[0260] Based on depth information of the desired positioning of the three-dimensional images, including the display images for the stereoscopic left eye and the display images for the stereoscopic right eye, the display images for the stereoscopic left eye and the display images for the stereoscopic right eye are moved, and a three-dimensional image is generated. (14)

[0262] The image processing apparatus according to any one of (1) to (13), wherein

[0263] When the eyeball is no longer tracked by the eye-tracking unit,

[0264] Display Image Generation Unit

[0265] The last displayed image is retained until the eyeball deviates. (15)

[0267] The image processing apparatus according to any one of (1) to (14), wherein

[0268] Display Image Generation Unit

[0269] The display mode of any one or all of the multiple regions can be changed according to the time period. (16)

[0271] The image processing apparatus according to any one of (1) to (15), wherein

[0272] Display Image Generation Unit

[0273] Generate a display image in which the boundaries indicate multiple specific locations. (17)

[0275] The image processing apparatus according to any one of (1) to (16), wherein

[0276] Display Image Generation Unit

[0277] The size of any one or all of the plurality of regions can be changed according to the change in the size of the eyeball. (18)

[0279] The image processing apparatus according to any one of (1) to (17), wherein

[0280] Display Image Generation Unit

[0281] The size of any or all of the multiple regions can be changed according to the time period. (19)

[0283] The image processing apparatus according to any one of (1) to (18), wherein

[0284] The display mode of each of the plurality of regions differs according to the difference in brightness of each of the plurality of regions. (20)

[0286] According to the image processing apparatus described in (19), wherein

[0287] The specific location refers to the position of the toric axis arrangement of the intraocular lens, and

[0288] Display Image Generation Unit

[0289] The luminance of each of the plurality of regions is changed such that the boundary indicates the position of the complex surface axis arrangement. (twenty one)

[0291] The image processing apparatus according to any one of (1) to (20), wherein

[0292] The specific location is any one of the following: the location of the toric surface axis of the intraocular lens, the incision location for forceps insertion, the incision location of the anterior capsule incision, the axial length, the center of the limbus, the center of the pupil, the preoperative pupil center, the visual axis location, or the center of the edge of the anterior capsule incision. (twenty two)

[0294] An image processing method performed by an image processing device.

[0295] The method includes:

[0296] Receive surgical field images of the patient's eyes;

[0297] Tracking the eyeball in the surgical field image; and

[0298] Multiple regions with different display modes are defined for the surgical field image, and a display image is generated with the boundaries of the multiple regions representing at least one of a specific position, a specific direction, and a specific size relative to the eye.

[0299] Among them, image processing device

[0300] Based on the eye tracking results, the display mode of any one or all of the plurality of regions is changed, and at least one of the position, orientation, and size of the boundary is changed. (twenty three)

[0302] A surgical microscope system, comprising:

[0303] Surgical microscope, which obtains images of the surgical field of the patient's eye;

[0304] Image processing apparatus, generating and displaying images; and

[0305] The display device displays the image.

[0306] The image processing device includes:

[0307] An image input unit receives the aforementioned surgical field image;

[0308] An eye-tracking unit that tracks the eye in the surgical field images described above; and

[0309] The display image generation unit sets multiple regions with different display modes for the surgical field image, and generates a display image representing the boundaries of the multiple regions for at least one of the specific position, specific orientation, and specific size of the eye; and

[0310] Display Image Generation Unit

[0311] Based on the eye tracking results, the display mode of any one or all of the plurality of regions is changed, and at least one of the position, orientation, and size of the boundary is changed. (twenty four)

[0313] An image processing method using an image processing apparatus according to any one of (1) to (21). (25)

[0315] A surgical microscope system comprising an image processing device according to any one of (1) to (21).

[0316] Reference number list

[0317] 1. Surgical Microscope System

[0318] 10. Surgical microscope

[0319] 11 Objective lens

[0320] 12 eyepieces

[0321] 13 Image processing device

[0322] 13A Control Unit

[0323] 13a Preoperative planning receiving unit

[0324] 13b Image Input Unit

[0325] 13c Registration Unit

[0326] 13D Information Storage Unit

[0327] 13e Eye-tracking unit

[0328] 13f Display Image Generation Unit

[0329] 14. Monitors

[0330] 20 beds

[0331] 51 Light Source

[0332] 52. Observation of optical systems

[0333] 52a semi-reflective mirror

[0334] 53 Forward Image Capture Unit

[0335] 54 Tomographic Image Capture Units

[0336] 55 presentation units

[0337] 56 Interface Units

[0338] 57 speakers

[0339] 500 computers

[0340] 510 CPU

[0341] 520ROM

[0342] 530 RAM

[0343] 540 bus

[0344] 550 Input / Output Interface

[0345] 560 Input Unit

[0346] 570 Output Unit

[0347] 580 recording units

[0348] 590 Communication Unit

[0349] 600 drive

[0350] 610 Removable recording media.

Claims

1. An image processing apparatus, comprising: The image input unit receives surgical field images of the patient's eyes; An eye-tracking unit tracks the eye in the surgical field image; as well as The display image generation unit defines multiple regions with different display modes for the surgical field image, and generates a display image whose boundaries represent at least one of the following: a specific position, a specific direction, and a specific size of the eye. Among them, the display image generation unit Based on the eye tracking results, the display mode of any one or all of the plurality of regions is changed, and at least one of the position, orientation and size of the boundary is changed; The image processing apparatus further includes: The preoperative planning receiving unit receives information on at least one of the location, orientation, and size of preoperative images and indicators based on the preoperative plan for the eye; and The information storage unit compares the preoperative image with the surgical field image at the start of the surgery, changes at least one of the position, orientation, and size of the indicator based on the surgical field image at the start of the surgery, and stores at least one of the changed position, orientation, and size of the indicator in the surgical field image at the start of the surgery.

2. The image processing apparatus according to claim 1, wherein The eye-tracking unit The eyeball is tracked in the real-time surgical field image by comparing the surgical field image at the start of surgery with the real-time surgical field image, and relational information is output. This relational information represents the relationship between at least one of the eyeball's position, orientation, and size in the real-time surgical field image and at least one of the altered position, orientation, and size of the indicator. The display image generation unit The boundary is arranged based on at least one of the altered position, orientation, and size of the indicator, and at least one of the position, orientation, and size of the boundary in the real-time surgical field image is changed based on the relational information to eliminate changes in the position, orientation, and size of the boundary relative to the eyeball in the surgical field image at the start of surgery, and the display image is generated.

3. The image processing apparatus according to claim 1, wherein The display image generation unit The coordinate transformation alters the multiple regions, causing at least one of the position, orientation, and size of the boundary to change based on the eye tracking results, and generates the display image.

4. The image processing apparatus according to claim 1, wherein The display image generation unit Based on the eye tracking results, the boundary line indicating the boundary is changed, and at least one of the position, orientation, and size of the boundary is changed.

5. The image processing apparatus according to claim 1, wherein The display image generation unit For two or more of the multiple regions, perform the process of adjusting the same type of parameters.

6. The image processing apparatus according to claim 1, wherein The display image generation unit For two or more of the multiple regions, perform processing to adjust different types of parameters.

7. The image processing apparatus according to claim 1, wherein The display image generation unit As the distance from the boundary increases, the processing intensity for any one or all of the multiple regions decreases.

8. The image processing apparatus according to claim 1, wherein The display image generation unit The display image is generated as either a display image for the left stereoscopic eye or a display image for the right stereoscopic eye, or the display image is generated as a display image for the left stereoscopic eye and a display image for the right stereoscopic eye, respectively.

9. The image processing apparatus according to claim 8, wherein The eye-tracking unit Tracking the eyeball in surgical field images for either or both of the stereoscopic left eye and stereoscopic right eye, and The display image generation unit Based on the eye tracking results for one or both of the stereoscopic left eye and stereoscopic right eye, the position, orientation, and size of the boundary are changed to generate a display image for the stereoscopic left eye and a display image for the stereoscopic right eye.

10. The image processing apparatus according to claim 8, wherein The display image generation unit In the display image for the stereoscopic left eye and the display image for the stereoscopic right eye, the boundary is set in the same position.

11. The image processing apparatus according to claim 8, wherein The display image generation unit The display image is generated such that at least one of the position, orientation, and size of the boundary between the display image for the stereoscopic left eye and the display image for the stereoscopic right eye is different, in a manner that indicates at least one of the specific position, the specific orientation, and the specific size.

12. The image processing apparatus according to claim 8, wherein The display image generation unit The three-dimensional image is generated by displacing the display images for the stereoscopic left eye and the display images for the stereoscopic right eye based on the depth information of the desired positioning of the three-dimensional images, including the display images for the stereoscopic left eye and the display images for the stereoscopic right eye.

13. The image processing apparatus according to claim 1, wherein When the eyeball deviates from the eye-tracking unit's tracking of the eyeball, The display image generation unit The last displayed image was maintained before the eyeball deviated.

14. The image processing apparatus according to claim 1, wherein The display image generation unit The display mode of any one or all of the multiple regions can be changed according to the time period.

15. The image processing apparatus according to claim 1, wherein The display image generation unit Generate display images indicating multiple specific locations from the boundaries.

16. The image processing apparatus according to claim 1, wherein The display image generation unit The size of any one or all of the plurality of regions can be changed according to the change in the size of the eyeball.

17. The image processing apparatus according to claim 1, wherein The display image generation unit The size of any one or all of the multiple regions can be changed according to the time period.

18. The image processing apparatus according to claim 1, wherein The display mode of each of the multiple regions differs based on the difference in brightness of each of the multiple regions.

19. The image processing apparatus according to claim 18, wherein The specific location refers to the position of the toric axis arrangement of the intraocular lens, and The display image generation unit The luminance of each of the plurality of regions is changed such that the boundary indicates the position of the complex surface axis arrangement.

20. The image processing apparatus according to claim 1, wherein The specific location is any one of the following: the location of the tortuous axis of the intraocular lens, the incision location for forceps insertion, the incision location of the anterior capsule incision, the axial length, the center of the limbus, the center of the pupil, the preoperative pupil center, the visual axis location, or the center of the edge of the anterior capsule incision.

21. An image processing method executed by an image processing device, The method includes: Receive surgical field images of the patient's eyes; Track the eyeball in the surgical field image; as well as Multiple regions with different display modes are defined for the surgical field image, and the boundaries of the multiple regions are generated to represent display images for at least one of the following: a specific location, a specific orientation, and a specific size of the eye. Among them, the image processing device Based on the eye-tracking results, the display mode of any one or all of the plurality of regions is changed, and at least one of the position, orientation, and size of the boundary is changed. Receive information on at least one of the location, orientation, and size of preoperative images and indicators based on the preoperative plan for the eye; as well as By comparing the preoperative image with the surgical field image at the start of the surgery, at least one of the position, orientation, and size of the indicator is changed according to the surgical field image at the start of the surgery, and at least one of the changed position, orientation, and size of the indicator is stored in the surgical field image at the start of the surgery.

22. A surgical microscope system, comprising: Surgical microscopes are used to obtain surgical field images of the patient's eyes; Image processing device, which generates and displays images; as well as The display device displays the image. The image processing device includes: The image input unit receives the surgical field image; An eye-tracking unit tracks the eye in the surgical field image; and The display image generation unit defines multiple regions with different display modes for the surgical field image, and generates a display image whose boundaries represent at least one of the following: a specific position, a specific direction, and a specific size of the eye. Among them, the display image generation unit Based on the eye-tracking results, the display mode of any one or all of the plurality of regions is changed, and at least one of the position, orientation, and size of the boundary is changed. The image processing apparatus further includes: The preoperative planning receiving unit receives information on at least one of the location, orientation, and size of preoperative images and indicators based on the preoperative plan for the eye; and The information storage unit compares the preoperative image with the surgical field image at the start of the surgery, changes at least one of the position, orientation, and size of the indicator based on the surgical field image at the start of the surgery, and stores at least one of the changed position, orientation, and size of the indicator in the surgical field image at the start of the surgery.

Images