Surgical microscope system, image processing method, program, and image processing device

Abstract

A surgical microscope system pertaining to an embodiment of the present invention is provided with a microscope optical system, an imaging device, and an image processing device. The imaging device captures an image of a visual range including an eye as an operation subject, through the microscope optical system. The image processing device magnifies a first region in an image including the eye to be operated on which is based on the image captured by the imaging device, and reduces the size of a second region other than the first region in the image so that the second region can be displayed in a region other than a magnification region which is the first region magnified in the image.

Description

[Technical field]

[0001] The present technology relates to a surgical microscope system, an image processing method, a program, and an image processing device that can be applied to ophthalmic surgery or the like. [Background technology]

[0002] In the ophthalmic microscope described in Patent Document 1, observation images of the left and right eyes of a patient are acquired by user operation. A captured image is generated by performing processing such as changing the display magnification and correcting the image on the captured observation image. The user can switch between the observation image and the captured image at a desired timing. This makes it possible to clearly and selectively present the observation image and the captured image with a simple configuration (see, for example, paragraphs

[0059] and

[0077] of the specification of Patent Document 1, Figures 1 and 6). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-12431 A Summary of the Invention [Problem to be solved by the invention]

[0004] Thus, there is a demand for technology that allows for a sufficient understanding of an image of an object when observing using an ophthalmic microscope.

[0005] In view of the above circumstances, an object of the present technology is to provide a surgical microscope system, an image processing method, a program, and an image processing device that are capable of fully grasping an image including the eye of a surgical target. [Means for solving the problem]

[0006] In order to achieve the above object, a surgical microscope system according to an embodiment of the present technology includes a microscope optical system, an imaging device, and an image processing device. The imaging device captures an image of a visual field including the eye that is the treatment target through the microscope optical system. The image processing device enlarges a first area in an image including the eye of the treatment subject based on an image captured by the imaging device, and reduces a second area other than the first area in the image so that it can be displayed in an area other than the enlarged area, which is the enlarged first area in the image.

[0007] In this surgical microscope system, a field of view including the eye to be treated is captured through a microscope optical system. A first area in the captured image is enlarged, and a second area other than the first area is reduced so that it can be displayed in an area other than the enlarged area, which is the enlarged first area. This makes it possible to fully grasp the image including the eye to be treated.

[0008] The image may be at least a part of a captured image of the eye to be treated.

[0009] The image processing device may set, for the second area, a peripheral area adjacent to a periphery of the first area and an area other than the peripheral area, and reduce the peripheral area.

[0010] The image processing device may acquire treatment information including at least one of information regarding the eye to be treated or information regarding a treatment to be performed on the eye, and determine the first area based on the treatment information.

[0011] The image processing device may acquire the treatment information based on the image.

[0012] The image processing device may acquire the treatment information from an external source.

[0013] The treatment information may include at least one of the contents of the treatment, an instrument used in the treatment, an illuminated area, an eye part, a lesion, or a contour of the image.

[0014] The image processing device may be configured to be able to accept an instruction from a user, and may be configured to determine the first region based on the instruction from the user.

[0015] The image processing device may set a predetermined position within the first area as a reference position, and enlarge the first area using the reference position as a reference.

[0016] The image processing device may enlarge the first area such that a distance between the reference position and each position in the first area is increased.

[0017] The image processing device may reduce the second area using the reference position as a reference.

[0018] The image processing device may reduce the second area so that a distance between the reference position and each of the outermost positions of the image is maintained.

[0019] The image processing device may set, for the second area, a peripheral area adjacent to the periphery of the first area and an area other than the peripheral area, and reduce the second area so that the distance between the reference position and each position in the area other than the peripheral area is maintained.

[0020] The image processing device may acquire treatment information including at least one of information regarding the eye to be treated or information regarding a treatment to be performed on the eye, and determine the reference position based on the treatment information.

[0021] The image processing device may be configured to be able to accept an instruction from a user, and may be configured to determine the reference position based on the instruction from the user.

[0022] The image processing device may be configured to be capable of generating an auxiliary image representing an enlarged state of the first region and a reduced state of the second region.

[0023] The auxiliary image may be an image in which the enlargement state and the reduction state are expressed using at least one of a grid pattern, brightness, saturation, hue, and a boundary line.

[0024] An image processing method according to one embodiment of the present technology is an image processing method executed by a computer system, comprising: acquiring an image including an eye to be treated; a first region in the image is enlarged; and a second region other than the first region in the image is reduced so as to be displayable in a region other than the enlarged region, which is the enlarged first region in the image.

[0025] A program according to one embodiment of the present technology causes a computer system to execute the following steps. Acquiring an image including the eye to be treated. A step of enlarging a first region in the image and reducing a second region in the image other than the first region so that the second region can be displayed in a region in the image other than the enlarged region, which is the enlarged first region.

[0026] An image processing device according to an embodiment of the present technology includes an image acquisition unit and an image processing unit. The image acquisition unit acquires an image including the eye of a treatment target. The image processing unit enlarges a first region in the image and reduces a second region other than the first region in the image so that it can be displayed in a region other than the enlarged region, which is the enlarged first region in the image. [Brief description of the drawings]

[0027] [Figure 1] 1 is a block diagram showing an example of the configuration of a surgical microscope system according to a first embodiment of the present technology. [Diagram 2] FIG. 1 is a block diagram showing an example of the configuration of an image processing device. [Diagram 3] 13 is a flowchart showing an example of generating a scaled image. [Figure 4] FIG. 11 is a schematic diagram for explaining generation of a scaled image. [Diagram 5]10A and 10B are diagrams for explaining an example of a method for enlarging an enlargement target area and a method for reducing a reduction target area. [Figure 6] 13 is a graph for explaining an example of setting a magnification for each position. [Figure 7] 13 is a flowchart showing a specific example of a process for generating a scaled image. [Figure 8] 10 is a schematic diagram showing a change in magnification of a target image of an eye that is a treatment target. FIG. [Figure 9] FIG. 13 is a schematic diagram showing an example of detection of a reference position during laser coagulation. [Figure 10] FIG. 13 is a schematic diagram showing an example of detection of a reference position during vitreous resection. [Figure 11] 11 is a schematic diagram showing an example of detection of a reference position when peeling off the internal limiting membrane. FIG. [Figure 12] FIG. 13 is a schematic diagram showing an example of detection of a reference position when draining subretinal fluid. [Figure 13] 11 is a schematic diagram showing an example of detection of a reference position when a proliferative membrane is ablated; FIG. [Figure 14] 13 is a flowchart showing processing of a presentation image regarding magnification at each position of a reduced or enlarged image according to a second embodiment of the present technology. [Figure 15] 11A to 11C are schematic diagrams illustrating an example of an auxiliary image that represents the magnification at each position of the enlarged or reduced image. [Figure 16] 11A to 11C are schematic diagrams illustrating an example of an auxiliary image that represents the magnification at each position of the enlarged or reduced image. [Figure 17] 11A to 11C are schematic diagrams illustrating an example of an auxiliary image that represents the magnification at each position of the enlarged or reduced image. [Figure 18] 11A to 11C are schematic diagrams illustrating an example of an auxiliary image that represents the magnification at each position of the enlarged or reduced image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0029] <First embodiment> [Surgical microscope system] 1 is a block diagram showing a configuration example of a surgical microscope system 100 according to a first embodiment of the present technology. As shown in the figure, the surgical microscope system 100 includes a head lens 101, a microscope 102, an image capturing device 103, an image processing device 104, a display device 105, a microscope control input unit 106, a microscope control unit 107, a user input unit 108, and an instrument 109.

[0030] The front lens 101 can be disposed at a position between the microscope 102 and the patient's eye. The front lens 101 is selectively used from a plurality of lenses having different viewing angles, such as 90° and 120°, suitable for various treatments.

[0031] The microscope 102 magnifies the light emitted from the front lens 101 and makes it incident on the imaging device 103. The microscope 102 includes a microscope optical system for constituting an optical microscope. The specific configuration of the microscope 102 (microscope optical system) is not limited and may be designed arbitrarily.

[0032] The imaging device 103 is mounted on the microscope 102, and captures an image of the eye to be treated via the front lens 101 and the microscope 102. That is, the imaging device 103 can capture an image of the visual field range of the microscope optical system.

[0033] In this embodiment, the imaging device 103 generates a captured image of the eye to be treated. The specific configuration of the imaging device 103 is not limited, and for example, a digital camera equipped with an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor is used. Also, for example, an infrared camera equipped with an infrared illuminator such as an infrared LED may be used. The imaging device 103 outputs the captured image to the image processing device 104.

[0034] The image processing device 104 is called a CCU (Camera Control Unit) and is capable of performing various image processing on the captured image output from the imaging device 103. In this embodiment, the image processing device 104 generates an enlarged or reduced image, which will be described in detail later. In addition, the specific content of the image processing that can be performed by the image processing device 104 is not limited and may be set arbitrarily. For example, when the image of the eye to be treated is inverted by the front lens 101, information on the inversion, such as the presence or absence of inversion and the range of the inversion area, may be generated and output by the image processing device 104.

[0035] The display device 105 can display various images based on the images (image information) output from the image processing device 104. For example, the above-mentioned enlarged or reduced images are displayed by the display device 105. In addition, various GUIs (Graphical User Interfaces) for performing treatment, information on the eye to be treated, etc. may be displayed.

[0036] The display device 105 is, for example, a general display or a head-mounted display. The display device 105 may be a plurality of displays, for example, a display for a surgeon and a display for an assistant may be separately prepared.

[0037] The microscope control input unit 106 accepts operation inputs to the microscope 102 by the user. For example, the user can use the microscope control input unit 106 to perform operation inputs such as moving the field of view of the microscope 102 in any direction. The microscope control input unit 106 is, for example, a foot switch.

[0038] The microscope control unit 107 controls the microscope 102 based on an input signal supplied from the microscope control input unit 106. For example, the microscope control unit 107 can adjust the lens barrel position of the microscope 102 in response to the input signal, and move the field of view of the microscope 102. The microscope control unit 107 can also control the magnification, field of view range, etc. of the microscope 102.

[0039] The user input unit 108 receives instructions input by a user. For example, the user input unit 108 is configured with an input device such as a mouse or a keyboard. In this embodiment, for example, instructions related to the generation and display of the enlarged or reduced image described above are input. In addition, information related to the content of the treatment performed by the user, the technique of the treatment, etc. may be input. In addition, the information that can be input by the user is not limited and may be set arbitrarily.

[0040] The instrument 109 is a device such as a surgical tool used in a treatment. The instrument 109 includes, for example, a laser probe and a vitreous cutter. In this embodiment, the image processing device 104 and the instrument 109 are connected, and various information related to the instrument 109 is input to the image processing device 104. For example, information such as the type of the instrument 109, the operation of the instrument, and the use state of the instrument is input. This information is included in the treatment information.

[0041] 1 illustrates the connection between the instrument 109 and the image processing device 104, but the instrument 109 may be connected to the microscope control unit 107. Various information output from the instrument 109 may be used to control the microscope 102.

[0042] [Functional configuration of image processing device] 2 is a block diagram showing an example of the configuration of the image processing device 104. The image processing device 104 has hardware necessary for configuring a computer, such as a GPU, a CPU, a ROM, a RAM, and a HDD. The CPU loads a program according to the present technology, which is pre-recorded in the ROM or the like, into the RAM and executes the program, thereby executing the image processing method according to the present technology.

[0043] For example, the image processing device 104 can be realized by any computer such as a PC (Personal Computer). Of course, hardware such as a GPU, an FPGA, or an ASIC may also be used. The image processing device 104 may also be realized by any other computer system.

[0044] 2, in this embodiment, the CPU executes a predetermined program to realize the image processing device 104 as functional blocks, including an image information acquisition unit 110, an image recognition unit 111, a presentation image generation unit 112, a control unit 113, and an interface unit 114. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize each block.

[0045] The program is installed in the image processing device 104 via, for example, various recording media. Alternatively, the program may be installed via the Internet or the like.

[0046] The type of recording medium on which the program is recorded is not limited, and any computer-readable recording medium may be used. For example, any recording medium that non-temporarily records data may be used.

[0047] The image information acquisition unit 110 generates a target image, which is an image including the eye to be treated, based on the captured image from the imaging device 103. The target image can also be said to be an image that is a target for generating a reduced or enlarged image. For example, the captured image may be used as the target image as is. Alternatively, a part of the captured image may be used as the target image. Also, for example, the range (contour) of the front lens of the captured image may be generated as the target image. In other words, the image information acquisition unit 110 may acquire image information obtained by performing preprocessing on the captured image captured by the imaging device 103 as the target image.

[0048] In the present disclosure, an image includes a still image and a moving image. Of course, a plurality of frame images included in a moving image are also included in an image. Furthermore, acquisition of image information includes acquisition of an image signal including image information. Furthermore, the data format of the image information is not limited and may be set arbitrarily.

[0049] The image recognition unit 111, under the control of the control unit 113, executes image recognition processing on the target image acquired by the image information acquisition unit 110. In this embodiment, the image recognition processing by the image recognition unit 111 detects treatment information.

[0050] The treatment information is information including at least one of information on the eye to be treated or information on the treatment to be performed on the eye. For example, by image recognition, the contents of the treatment to be performed, the technique to be performed, the instruments (surgical tools) used in the treatment, the positions of each part of the surgical tools, the part of the eye, the illuminated range (for example, the center of gravity of the illumination range of the aiming beam), the affected area, the range of the front lens 101, the contour of the target image, etc. are detected as the treatment information. Of course, this is not limited to this, and at least one of these pieces of information or other pieces of information may be detected.

[0051] The specific method of detecting treatment information by image recognition is not limited, and any technology may be used. For example, any image recognition method such as edge detection or pattern matching may be used, and the algorithm is not particularly limited. In addition, any machine learning algorithm using, for example, DNN (Deep Neural Network) may be used. For example, by using AI (artificial intelligence) that performs deep learning, it is possible to improve the detection accuracy of treatment information.

[0052] It is also possible to generate treatment information based on not only the target image but also the captured image output from the imaging device 103.

[0053] The presentation image generating unit 112 generates an image (hereinafter, referred to as a presentation image) to be displayed by the display device 105 under the control of the control unit 113. In this embodiment, a scaled image is generated based on the target image output from the image information acquiring unit 110, and is output as the presentation image.

[0054] The interface unit 114 accepts various instructions input by the user input unit 108. The interface unit 114 also accepts various information related to the appliance 109 output from the appliance 109. The interface unit 114 accepts user instructions and information related to the appliance 109 via, for example, a Universal Serial Bus (USB) terminal or a High-Definition Multimedia Interface (HDMI) (registered trademark) terminal.

[0055] Alternatively, it is also possible to receive user instructions and information about the instrument 109 by using wireless LAN communication such as WiFi or short-range wireless communication such as Bluetooth (registered trademark). That is, the communicative connection between the image processing device 104 and the user input unit 108 and the communicative connection between the image processing device 104 and the instrument 109 shown in Fig. 1 may be realized by wire or wirelessly.

[0056] The control unit 113 controls each unit. For example, the control unit 113 controls the image recognition unit 111, the presentation image generation unit 112, and the interface unit 114 based on an input instruction, and controls each unit so as to generate a scaled image from the input target image.

[0057] The control unit 113 can also execute judgment of the procedure to be performed in the treatment based on the treatment information output from the image recognition unit 111 and the interface unit 114. As described above, the recognition result by the image recognition unit 111 and information on the instrument 109 are included in the treatment information. The judgment result judged by the control unit 113 based on the treatment information and the calculation result of a predetermined calculation are also included in the treatment information.

[0058] In this embodiment, an image acquiring unit that acquires a target image including the eye that is the treatment target is realized by the image information acquiring unit 110 and the control unit 113. In addition, the user input unit 108, the control unit 113, and the interface unit 114 can also be said to be a receiving unit configured to be able to accept instructions from the user.

[0059] Furthermore, the presentation image generating unit 112 and the control unit 113 realize an image processing unit that enlarges a first region in a target image and reduces a second region other than the first region in the target image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the target image. The operation of this image processing unit is related to the generation of the enlarged or reduced image, and will be described in detail below.

[0060] Fig. 3 is a flow chart showing an example of the generation of the enlarged or reduced image. Fig. 4 is a schematic diagram for explaining the generation of the enlarged or reduced image. The image shown at the top of Fig. 4 is an example of the target image 10, and the image shown at the bottom is an example of the enlarged or reduced image. Note that in the target image 10 and the enlarged or reduced image 20 shown in Fig. 4, the contents of the image of the eye or the like to be treated are omitted.

[0061] First, a target image 10 is input (step 101). As shown in Fig. 4, an enlargement target region 11 is determined for the input target image 10 (step 102). The enlargement target region 11 is typically a region that the practitioner wishes to focus on and is an important region in the treatment. Of course, it is not limited to such a region.

[0062] The method of determining the region to be enlarged is not limited, and any method may be adopted. For example, the region to be enlarged may be determined based on an instruction input by a user. Alternatively, the region to be enlarged may be automatically determined based on treatment information. For example, the region to be enlarged may be determined based on the content of the treatment, the position of the part of the instrument, etc.

[0063] The area in the target image 10 other than the area to be enlarged 11 becomes the area to be reduced 12. That is, step 102 can be said to be a step of determining the area to be enlarged 11 and the area to be reduced 12. In this embodiment, the area to be enlarged corresponds to the first area, and the area to be reduced corresponds to the second area.

[0064] The enlargement target area 11 is enlarged, and the reduction target area 12 is reduced to generate the enlarged / reduced image 20 (step 103). Specifically, as shown in Fig. 4, the enlargement target area 11 in the target image 10 is enlarged. Then, the reduction target area 12 in the target image 10 is reduced so that it can be displayed in an area 22 other than the enlargement area 21, which is the enlarged enlargement target area in the target image 10 (i.e., the enlargement target area after enlargement).

[0065] In this manner, the enlarged target area 11 is enlarged and the reduced target area 12 other than the enlarged target area 11 is reduced to generate the enlarged / reduced image 20. This makes it possible to simultaneously enlarge the enlarged target area 11 and confirm the entire image (surgical field) without losing any of the contents (information) of the entire original image. As a result, it becomes possible to fully grasp the target image 10 including the eye to be treated.

[0066] In this disclosure, enlarging an image means displaying (the contents of) the image in an area larger than the original display area, and reducing an image means displaying (the contents of) the image in an area smaller than the original display area.

[0067] The method of enlarging an image to display it in an area larger than the original display area and the method of reducing an image to display it in an area smaller than the original display area are not limited. For example, the entire area to be enlarged 11 may be enlarged at a predetermined magnification. Alternatively, different magnifications may be assigned to multiple areas in the area to be enlarged 11, and the enlargement process may be performed according to the assigned magnifications. Alternatively, only a part of the area to be enlarged 11 may be enlarged, so that the area to be enlarged as a whole is displayed in an area larger than the original display area. Even when different enlargement magnifications (including the same magnification) are set, it is possible to sufficiently prevent image breakdown from occurring by performing any image processing, such as a complementation process based on pixel information.

[0068] Similarly, as for the method of reducing the image, for example, the entire reduction target area 12 may be reduced by a predetermined magnification. Alternatively, different magnifications may be assigned to multiple areas within the reduction target area 12, and reduction processing may be performed according to the assigned magnifications. Alternatively, only a portion of the reduction target area 12 may be reduced, so that the reduction target area as a whole is displayed in an area smaller than the original display area. Even when different reduction magnifications (including equal magnification) are set, it is possible to sufficiently prevent image breakdown from occurring by performing any image processing, such as interpolation processing, based on pixel information.

[0069] Fig. 5 is a diagram for explaining an example of a method for enlarging the enlargement target region 31 and a method for reducing the reduction target region 32. For example, as shown in Fig. 5, for the reduction target region 32, a peripheral region 33 adjacent to the periphery of the enlargement target region 31 and a separated region 34 other than the peripheral region 33 are set. Then, by reducing the peripheral region 33, the entire reduction target region is reduced. Therefore, for the separated region 34, the same image as the target image 30 (image at the same magnification) is displayed as is.

[0070] 5, a predetermined position within the region to be enlarged 31 is set as a reference position 35. For example, the reference position 35 is a position that serves as a reference for the enlargement process of the region to be enlarged 31. For example, the region to be enlarged 31 may be enlarged such that the distance between the reference position 35 and each position in the region to be enlarged 31 is enlarged.

[0071] Furthermore, for example, the same reference position 35 may be a position used as a reference for the reduction process of the reduction target region 32. For example, the reduction target region 32 may be reduced so that the distance between the reference position 35 and each position in the separated region 34 is maintained. In this case, the reduction target region 32 is reduced so that each position in the peripheral region 33 is included between the edge of the enlarged region 31 and the boundary between the peripheral region 33 and the separated region 34.

[0072] It should be noted that the outermost portion 39 of the target image 30 is included in the separated region 34. Therefore, the distance between the reference position 35 and each position of the outermost portion 39 of the target image 30 is maintained. For example, suppose that the peripheral region 33 and the separated region 34 are not set and the entire region to be reduced 32 is reduced. Even in this case, the region to be reduced 32 is reduced so that the distance between the reference position 35 and each position of the outermost portion 39 of the target image 30 is maintained. This makes it possible to prevent loss of image at the edges of the target image 30.

[0073] By performing magnification processing on each of the above positions, an enlarged or reduced image 40 is generated, which has a reference position 45 having coordinates common to the reference position 35, an enlarged region 41 obtained by enlarging the region to be enlarged 31, a reduced peripheral region 43 obtained by reducing the peripheral region 33, and a separation region 44 in which the distance between the reference position 45 and each position is maintained. The reduced peripheral region 43 and the separation region 44 can also be called a reduced region 42.

[0074] In this embodiment, the peripheral region 33 corresponds to a peripheral region adjacent to the periphery of the first region, and the separated region corresponds to a region other than the peripheral region.

[0075] Here, the magnification at each position of the original target image 30 is defined as follows. (distance from reference position 45 in enlarged / reduced image 40) / (distance from reference position 35 in original target image 30) For example, the magnification ratio between position 36 of the target image 30 and position 46 of the enlarged / reduced image 40 is R1 / r1. Similarly, the magnification ratio between position 37 of the target image 30 and position 47 of the enlarged / reduced image 40 is R2 / r2. Moreover, the magnification ratio between position 38 of the target image 30 and position 48 of the enlarged / reduced image 40 is R2 / r2.

[0076] For each position in the region to be enlarged 31, the magnification is set to be greater than 1. For each position in the separated region 34, the magnification is set to be 1. For the peripheral region 33, the magnification at each position is appropriately set to transition from the magnification at each position on the edge of the enlarged region 41 (greater than 1) to the magnification at each position on the boundary between the peripheral region 33 and the separated region 34 (equal to 1).

[0077] In the present disclosure, the boundary between two regions is considered to be included in each of the two regions. Therefore, the boundary between the peripheral region 33 and the separated region is considered to be included in both the peripheral region 33 and the separated region .

[0078] Fig. 6 is a graph for explaining an example of setting the magnification at each position. As shown in Fig. 6, by setting the magnification at each position of the enlargement target area 31 and the magnification at each position of the reduction target area 32 (the peripheral area 33 and the separated area 34), it is possible to generate the enlarged or reduced image 40.

[0079] 6 indicates the magnification of the enlargement or reduction performed on the target image 30. The horizontal axis indicates the distance from the reference position 35. In other words, the upper graph in FIG. 6 is a graph showing the change in the magnification of the region to be enlarged 31 or the region to be reduced 32 based on the distance from the reference position 35.

[0080] As shown in the upper graph of Fig. 6, when the distance from the reference position 35 is A, the image is enlarged at a high magnification. Hereinafter, the range from the reference position 35 to the distance A will be referred to as a high magnification portion. The magnification of the high magnification portion is determined arbitrarily. For example, if the user wants to grasp one location in more detail, the image may be enlarged at a higher magnification.

[0081] Furthermore, between distance A and distance B, the process switches from enlargement to reduction at distance D. That is, the area from reference position 35 to distance D becomes area to be enlarged 31. The point at distance D from reference position 35 can also be said to be the edge of area 31 to be enlarged.

[0082] Between distance D and distance B, the area around the enlarged target area 31 is reduced. That is, the area between distance D and distance B becomes the surrounding area 33. Also, between distance B and distance C, the magnification is 1x. Distance C represents the image edges of the target image 30 and the enlarged / reduced image 40. That is, the area between distance B and distance C becomes the separated area 34.

[0083] In this embodiment, the image edges of the target image 30 correspond to the respective outermost positions of the target image 30. That is, the presentation image generating unit 112 reduces the reduction target area 32 so that the distance between the reference position 35 and each outermost position of the target image 30 is maintained.

[0084] 6 is a graph showing the distance of the target image 30 and the enlarged / reduced image 40 from the reference position 35. The horizontal axis shows the distance of the target image 30 from the reference position 35. The vertical axis shows the distance of the enlarged / reduced image 40 from the reference position 45.

[0085] As shown in the graph at the bottom of Fig. 6, dotted line 50 is a straight line with a slope of 1. In other words, dotted line 50 is a position that is equidistant from reference position 35 of target image 30 and from reference position 45 of enlarged / reduced image 40, and indicates a state in which target image 30 and enlarged / reduced image 40 are at the same magnification. Furthermore, when dotted line 50 and solid line overlap, the positional relationship between corresponding positions in target image 30 and enlarged / reduced image 40 is equal.

[0086] The solid line 51 from the reference position 35 to the distance D is a curve with a slope greater than 1. That is, in the enlarged area 31 from the reference position 35 to the distance D, the enlarged or reduced image 40 is enlarged, so the distance from the reference position 35 is longer than the distance from the reference position 35 of the target image 30. Moreover, the slope of the solid line 51 is constant from the reference position 35 to the distance A because the magnification is constant.

[0087] The solid line 52 from the distance D to the distance B is a curve with a slope smaller than 1. That is, in the reduction area 42 from the distance D to the distance B, the target image 30 and the reduced image 40 are reduced in size, so that they approach a position where the distances from the reference position 35 to the target image 30 and the reduced image 40 are equal.

[0088] Moreover, the solid line 53 coincides with the dotted line 50 when it reaches the distance B. That is, the positional relationship between the positions of the separation region 34 of the target image 30 and the enlarged / reduced image 40 becomes equal.

[0089] In this embodiment, the length of each of the regions, distance A, distance B, distance C, and distance D from the reference position, is determined by the control unit 113. For example, the control unit 113 determines the length of each region based on the type of the performed procedure input from the interface unit 114.

[0090] In order to prevent loss within the range of the enlarged or reduced image 40, the relationship between the distance from the reference position 35 between the target image 30 and the enlarged or reduced image 40, as shown in the graph at the bottom of Figure 6, must be continuous and reach a part where they are equal within the image range (from the reference position 35 to distance C) (the point where the dotted line 50 and the solid line 53 coincide).

[0091] Also, if the equal portion within the image range is not reached, a loss will occur. For example, if the solid line exceeds the dotted line 50 by the distance C, the image edge of the target image 30 will be outside the image range of the enlarged / reduced image 40, and a loss will occur.

[0092] As shown in the upper graph of Fig. 6, as a means for determining the magnification for each position on the target image 30, first, the positional relationship from the reference position 35 is determined as shown in the lower graph of Fig. 6. Then, the magnification for each position on the upper graph of Fig. 6 is determined by differentiating the pixel positions of the enlarged / reduced image 40 in the lower graph of Fig. 6 with the pixel positions of the target image 30, which makes design easier.

[0093] The image edge may be set arbitrarily. For example, the image edge may be set to the position of the outline of the head lens 101 so that the enlargement / reduction process is completed within the range of the head lens 101, which is an important surgical field that is highly noticeable to the user.

[0094] There is no limitation on the shape and area of ​​the region to be enlarged 31. The user may determine the region to be enlarged 31 to have any shape and area.

[0095] The magnification is not limited to a specific value. For example, the magnification may be determined according to the direction from the reference position. For example, the magnification may be set to be higher in the direction opposite to the direction in which the shaft of the surgical tool is located. In other words, the magnification may be determined by estimating the degree of attention (difficulty of viewing) from the user based on the shadow of the surgical tool or the like.

[0096] It is possible to set a specific pixel in the target image 30 as the reference position. And each pixel included in each region can be set as the position of that region. It is possible to generate a scaled image based on the distance between the pixel serving as the reference position (hereinafter referred to as the reference pixel) and each pixel in each region.

[0097] It is possible that a pixel does not exist at a position where the distance from the reference pixel is enlarged or reduced in the enlarged or reduced image 40. In this case, it is possible to generate the enlarged or reduced image 40 without loss by performing an interpolation process of pixel information using, for example, pixels around that position.

[0098] Furthermore, regarding the setting of the reference position 35 within the region to be enlarged 31, after the region to be enlarged 31 is set, the reference position may be set at a predetermined position, such as the center of the region to be enlarged 31. Alternatively, after the reference position is set, the region to be enlarged 31 may be set based on the reference position 35. In this case, it is also possible to regard the determination of the region to be enlarged 31 and the setting of the reference position 35 as equivalent processes. Of course, the method of generating the enlarged / reduced image 40 is not limited, and any other method may be adopted.

[0099] There may also be cases where enlarged or reduced images are generated continuously for multiple frame images (multiple target images) that make up a moving image. In this case, the enlargement target area (reference position) may be set individually for each target image. In other words, there may also be cases where the enlargement target area (reference position) for each target image is set to change dynamically.

[0100] Fig. 7 is a flow chart showing a specific example of a process for generating a scaled image 70. Fig. 8 is a schematic diagram showing a change in the magnification of a target image 60 in which an eye to be treated is photographed. In the above explanation, an image obtained through a front lens is processed because a case of photographing the inside of the eye is dealt with, but when photographing the anterior part of the eye as in the case of Fig. 8, it is preferable to remove the front lens from the optical path or to remove it from the system so that it does not enter the photographing range.

[0101] During surgery, the surgeon's focus varies depending on the procedure being performed, and the degree of focus varies for each position in the surgical field. Therefore, in order to improve the convenience of the surgeon, it is desirable to estimate the surgeon's focus and, as a result, determine the magnification of the image at each position based on the procedure being performed. Below, a specific example of processing including such magnification determination will be described.

[0102] The user inputs information about the procedure to be performed on the eye to the interface unit 114. The control unit 113 determines the procedure to be performed from the input information (step 201). For example, when performing a procedure to remove the vitreous body of a patient, the user inputs information to the interface unit 114 that a vitreous body removal will be performed.

[0103] The method of determining the performed procedure is not limited. For example, the performed procedure may be determined by receiving a signal indicating that a specific device for performing the performed procedure is connected to the surgical microscope system 100. Furthermore, the performed procedure may be determined based on information on the operation mode of a specific device, for example, a vitreous surgery device (for example, a laser coagulation mode or a vitrectomy mode). Also, for example, the performed procedures are displayed in a menu on the display device 105, and the user can specify the performed procedure via an input device.

[0104] In this embodiment, the information on the performed procedure corresponds to information on a treatment to be performed on the eye. In this embodiment, the performed procedure corresponds to the content of the treatment.

[0105] The target image 60 acquired by the image information acquisition unit 110 is input to the image recognition unit 111 (step 202).

[0106] 8, a target image 60 including an iris 61, a pupil 62, a limbus 63, an eyelid 64, a blood vessel 65, and a surgical tool 66 is displayed on the display device 105. That is, the target image 30 corresponds to the surgical field of a person who will perform the surgery.

[0107] In this embodiment, the control unit 113 detects the reference position 67 at the position of the tip of the surgical tool 66 based on the performed procedure input from the interface unit 114 (step 203).

[0108] In this embodiment, the iris 61, pupil 62, limbus 63, eyelid 64, and blood vessels 65 of the eye correspond to information about the eye to be treated. Also, in this embodiment, the surgical tool 66 corresponds to an instrument used in the treatment.

[0109] The image recognition unit 111 and the interface unit 114 can also be considered as an information acquisition unit that acquires treatment information including at least one of information about the eye that is the treatment target and information about the treatment to be performed on the eye.

[0110] An example of detection of a reference position based on each performed procedure will be described with reference to FIGS.

[0111] Figure 9 is a schematic diagram showing an example of detection of the reference position during laser coagulation. Laser coagulation refers to a procedure in which the retina is coagulated by irradiating a laser to treat retinal diseases. Laser coagulation is performed to burn new blood vessels or fix the area around a retinal tear to prevent retinal detachment.

[0112] As shown in Fig. 9, a reference position 73 is detected at the position of an aiming beam emitted from a laser probe 72. The aiming beam is emitted to a portion of the retina to be coagulated. That is, the reference position 73 is detected within a range illuminated by the aiming beam, which is the target of the user's attention. In this embodiment, the reference position 73 is detected at the center of gravity of an illuminated range 74.

[0113] The method of detecting the reference position 73 during laser coagulation is not limited. For example, the reference position 73 may be detected at a position at a certain distance 75 from the tip of the laser probe 72 in the tip-facing direction. Also, for example, the reference position may be set at the tip of the laser probe 72.

[0114] The setting of distance 75 used to detect the reference position is not limited. For example, distance 75 used to detect reference position 73 may be a predetermined fixed value in the range of 0 to 5 mm, or may be dynamically changed in the range of 0 to 5 mm during treatment.

[0115] Furthermore, for example, the range of distance 75 may be set based on the thickness of laser probe 72, the optical zoom magnification of the microscope, and the distance on the image. The distance between the tip of a surgical tool such as laser probe 72 and the depth of the retina estimated by stereoscopic vision using stereoscopic images corresponding to both eyes of the microscope may also be set as the basis.

[0116] Here, the distance used to detect the reference position may be a distance in a three-dimensional space, or may be a distance when orthogonally projected in the direction of the image plane.

[0117] Fig. 10 is a schematic diagram showing an example of detection of the reference position during vitrectomy. Vitrectomy refers to a procedure for removing the vitreous body using a vitreous cutter for removing the vitreous body in the eyeball. For example, vitrectomy is performed to remove and aspirate bleeding and cloudiness of the vitreous body to make the vitreous body transparent.

[0118] As shown in Fig. 10, a reference position 77 is detected at the tip of the vitreous cutter 76. The method for detecting the reference position during vitreous resection is not limited. For example, the reference position 77 may be detected at an opening 78 of the vitreous cutter 76 or at a position at a certain distance from the tip of the vitreous cutter 76.

[0119] Fig. 11 is a schematic diagram showing an example of detection of a reference position during internal limiting membrane peeling. Internal limiting membrane peeling refers to a procedure for peeling off the internal limiting membrane. For example, internal limiting membrane peeling is performed in a macular hole where a hole is created in the retina at the fovea of ​​the fundus, or in a premacular membrane where a membrane is stretched in front of the retina at the back of the eyeball to block the macula, etc.

[0120] 11, the image information acquisition unit 110 acquires a target image 80 in which an ILM forceps 81, a macular region 82, a fovea centralis 83, and an optic disc 84 are captured. The control unit 113 detects a reference position 85 at the position of the fovea centralis 83 of the macular region 82 on the retina.

[0121] The method of detecting the reference position 85 when peeling off the internal limiting membrane is not limited. For example, the reference position 85 may be detected at the midpoint of the tip of the ILM forceps 81 or at the internal dividing point between the fovea centralis 83 and the tip of the ILM forceps 81. In addition, the same reference position as when peeling off the internal limiting membrane may be used as the reference position when peeling off the premacular membrane.

[0122] FIG. 12 is a schematic diagram showing an example of detection of a reference position when draining subretinal fluid. Subretinal fluid drainage refers to a procedure for removing fluid (subretinal fluid) that has accumulated under the retina. For example, subretinal fluid drainage is performed when a procedure such as retinal detachment is performed.

[0123] 12, the image information acquisition unit 110 acquires a target image in which the backflush needle 86 and the retinal break 87 are captured. The control unit 113 detects a reference position 88 at the center of gravity of the range of the retinal break 87 in the vicinity of the backflush needle 86.

[0124] The method of detecting the reference position 88 when draining subretinal fluid is not limited. For example, the reference position 88 may be detected at the tip of the backflush needle 86. Also, for example, the reference position 88 may be detected at the internal division point between the center of gravity of the range of the retinal tear 87 and the tip of the backflush needle 86.

[0125] In this embodiment, the retinal tear 87 corresponds to the lesion. In addition, various other eye conditions that are the subject of treatment, such as retinal detachment and neovascularization, correspond to the lesion.

[0126] 13 is a schematic diagram showing an example of detection of a reference position during proliferative membrane resection. Proliferative membrane resection refers to a procedure for removing a proliferative membrane adhered to the retina. For example, proliferative membrane resection is performed when removing a proliferative membrane in proliferative diabetic retinopathy.

[0127] 13, a reference position 91 is detected at a pipe tip portion of a vitreous scissors blade 90. The method of detecting the reference position 91 during proliferative membrane resection is not limited. For example, the reference position 91 may be detected at the tip of the vitreous scissors blade 90, or may be detected at an internal division point between the pipe tip portion of the vitreous scissors blade 90 and the tip of the vitreous scissors blade 90.

[0128] The method of detecting the reference position 35 is not limited. For example, the reference position 35 may be determined based on an instruction input by a user. Also, for example, the tip of the surgical tool may always be set as the reference position 35 on the assumption that the user focuses on the tip of the surgical tool. Of course, the method is not limited to the above-mentioned surgical tools, and a method of detecting the reference position may be set for each of various surgical tools such as an intraocular illumination fiber.

[0129] The control unit 113 determines the magnification at each position of the target image based on the detected reference position (step 204). In this embodiment, the control unit 113 sets a circular enlargement target region 31 centered on the reference position 35. That is, the presentation image generation unit 112 enlarges the enlargement target region 31 with the reference position 35 as a reference.

[0130] The presentation image generation unit 112 generates an enlarged or reduced image 70 for the target image 60 in accordance with the control of the control unit 113. That is, the presentation image generation unit 112 enlarges the enlargement target region 68, and reduces the region other than the enlargement target region 68 so that it can be displayed in the region other than the enlarged region 71, which is the enlarged enlargement target region 68 in the target image 60.

[0131] Furthermore, enlargement and reduction processes are restricted in an area farther from the reference position 67 than the reduction target area. That is, in an area farther from the reference position 67 than the reduction target area, the positional relationship between the target image 60 and the enlarged / reduced image 70 becomes equal. For example, the area outside the reference position 67 and the corneal limbus 63, which includes the eyelids 64, blood vessels 65, etc., is displayed so that the positional relationship between the target image 60 and the enlarged / reduced image 70 becomes equal.

[0132] The presentation image generating unit 112 generates the enlarged or reduced image 70 based on the determined magnification at each position, and displays the generated enlarged or reduced image 70 on the display device 105 (step 205).

[0133] When an end instruction to end the enlargement / reduction is input to the interface unit 114, the control unit 113 ends the generation of the enlarged / reduced image 70 (Yes in step 206). When the user requires a new enlarged / reduced image 70, an instruction to regenerate the enlarged / reduced image 70 is input to the interface unit 114, and the process returns to step 202 (No in step 206).

[0134] The order of the processes for generating the enlarged or reduced image 70 is not limited. For example, the determination of the performed procedure in step 201 may be performed after the input of the target image in step 202. In this case, the method of determining the performed procedure may be based on information obtained from within the target image.

[0135] For example, as shown in FIG. 9, when a laser probe 72 is displayed within the input target image and the laser emitted from the laser probe 72 is irradiating a lesion such as a retinal tear, the performed procedure is determined to be laser coagulation.

[0136] Also, for example, as shown in FIG. 11, since the peripheral area of ​​the target image 80 is dark, it is determined that the procedure is being performed on the vitreous retina, ILM forceps 81 are being used, and the tip of the ILM forceps 81 is close to the macula 82, so the procedure is determined to be internal limiting membrane peeling.

[0137] That is, the procedure to be performed may be determined based on the type of surgical tool, the position or movement of the tip of the surgical tool, or the affected area of ​​the eye in the target image.

[0138] As described above, the image processing device 104 according to this embodiment acquires a target image 10 including an eye to be treated. The enlargement target region 11 in the target image 10 is enlarged, and the reduction target region 12 other than the enlargement target region 11 in the target image 10 is reduced so as to be displayable in a region 22 other than the enlargement region 21, which is the enlarged enlargement target region 11 in the target image 10. This makes it possible to fully grasp the target image 10 including the eye to be treated.

[0139] In ophthalmic surgery, there is a need to enlarge the surgical field to perform delicate procedures, while at the same time there is a contradictory need to grasp the situation of the entire surgical field, including areas other than the surgical site. The latter is particularly evident in cases where surgery is performed while observing a wide area of ​​the retina using a wide-angle observation system.

[0140] For example, when performing ophthalmic surgery using surgical tools such as forceps and a vitreous cutter, the vitreous may become entangled in the surgical tool, causing the retina to be pulled at a site other than the site where the surgery is being performed. As a result, retinal tears or retinal detachment may occur.

[0141] When performing intraocular surgery like this, complications or other eye damage may occur at a location other than the location of the surgical target, so it is desirable to grasp the overall condition inside the eye, including areas other than the location of the surgical target.

[0142] Therefore, this technology estimates the area of ​​interest of the user and displays the surgical field image without any loss by enlarging that area while reducing the surrounding area, thereby achieving both an enlarged view and an understanding of the entire surgical field.

[0143] This allows the user to grasp the overall situation of the target image (surgical field) when performing various procedures in eye surgery, while enlarging the target of interest for detailed observation.

[0144] <Second embodiment> When a scaled image is generated and displayed as in the first embodiment, some users may become confused because they cannot understand which part of the displayed image is scaled or reduced. In order to avoid such confusion and to help understand the displayed surgical field, a surgical microscope system according to a second embodiment of the present technology will be described below. In the following description, the description of the same parts as those in the configuration and operation of the surgical microscope system 100 described in the above embodiment will be omitted or simplified.

[0145] In the first embodiment, a scaled image in which the magnification of a target image is changed is generated by the presentation image generation unit 112. In the second embodiment, the presentation image generation unit 112 generates a scaled image and an auxiliary image that represents the magnification of the scaled image.

[0146] 14 is a flowchart showing the processing of a presentation image related to the magnification at each position of a scaled image according to the second embodiment of the present technology. The processing from step 301 to step 304 is similar to the processing from step 201 to step 204 described in the first embodiment, and therefore the description thereof will be omitted.

[0147] 14, an auxiliary image expressing the magnification of the enlarged or reduced image is generated by the presentation image generation unit 112 (step 305). The auxiliary image generated by the presentation image generation unit 112 is supplied to the display device 105 and displayed on a display or the like (step 306).

[0148] Moreover, the presentation image generating unit 112 can be said to be a generating unit capable of generating an auxiliary image that expresses an enlarged state of the first region and a reduced state of the second region.

[0149] 15 to 18 are schematic diagrams showing examples of auxiliary images expressing the magnification at each position of the enlarged or reduced image.

[0150] 15, it is assumed that a reference position 121 is detected when a captured target image 120 is an image in which lattice lines are drawn. The presentation image generating unit 112 generates an auxiliary image 122 indicating what kind of enlarged or reduced image will be generated when the target image 120 is processed into an enlarged or reduced image at a magnification determined from the reference position 121.

[0151] The display device 105 displays the auxiliary image 122 in a state in which it is overlaid on the enlarged or reduced image. Note that the image in a state in which the auxiliary image 122 is overlaid on the enlarged or reduced image may itself be used as the auxiliary image. Here, overlay includes displaying two images in a superimposed state. Displaying two images in a superimposed state can also be called superimposing.

[0152] As shown in FIG. 16, an auxiliary image 125 is generated in which pixel values ​​of the enlarged or reduced image are processed based on the magnification at each position of the enlarged or reduced image. For example, an enlarged area 126, a reduced area 127, and an area 128 of the same size are processed and displayed so that their luminance, saturation, or hue are different. That is, an enlarged area (area larger than the same size), a reduced area (area smaller than the same size), and an area of ​​the same size are processed and displayed so that their luminance, saturation, or hue are different. As a processing method, for example, a method of adding or multiplying a fixed value to the luminance value of each area is considered. As another processing method, for example, a method of replacing values ​​such as color difference, hue value, and saturation value for each area with a fixed value, or a method of adding or multiplying a value by a fixed value for each area is considered.

[0153] The presentation image generating unit 112 supplies the generated auxiliary image 125 to the display device 105 in place of the enlarged or reduced image. The number of regions of the auxiliary image 125 is not limited. For example, the auxiliary image may be color-coded into four regions: a high magnification portion, an enlarged region 126, a reduced region 127, and a normal magnification region 128.

[0154] The method of displaying the regions differently is not limited. For example, in addition to the color space defined by the YCbCr values ​​as a space that defines the luminance, saturation, and hue, other color space values ​​such as YIQ, HVS, and LCH may be used, and the base color space may be Lab, Lch, XYZ, etc. other than RGB.

[0155] Also, for example, luminance value or color processing may be performed based on the magnification of each pixel of the scaled image. In the processing related to luminance value, for example, the magnification value or its conversion value may be added to or multiplied with the luminance value of each pixel. In the processing related to color, for example, the magnification value or its conversion value at the position of each pixel may be added to or multiplied with the chrominance, hue, and saturation values ​​of each pixel.

[0156] 17, an auxiliary image 130 is generated in which a frame line indicating the magnification in each region of the enlarged or reduced image is displayed. For example, a frame line 131 indicating the edge of the enlarged region and a frame line 132 indicating the edge of the reduced region are displayed in the auxiliary image 130.

[0157] The display device 105 displays the auxiliary image 130 in a state where it is overlaid on the enlarged or reduced image. Note that the number of frame lines is not limited, and a frame line indicating a high magnification portion may be displayed.

[0158] As shown in Fig. 18, at least one of the auxiliary images generated in Fig. 15 to Fig. 17 is displayed in a part of the enlarged / reduced image 135 using PiP (Picture-in-Picture). For example, the auxiliary image 137 is reduced and displayed outside the range of the enlarged area and the reduced area based on the reference position 136. That is, the auxiliary image 137 is displayed outside the area in the enlarged / reduced image 135 that the user is paying attention to.

[0159] When an end instruction to end the presentation of the auxiliary image is input to the interface unit 114, the control unit 113 ends the output of the auxiliary image (Yes in step 307). When the user requires a new auxiliary image, an instruction to regenerate the auxiliary image is input to the interface unit 114, and the process returns to step 202 (No in step 307). That is, the user can arbitrarily switch the display of the auxiliary image.

[0160] There are no limitations on the location and reduction ratio of auxiliary image 137. For example, the size and position of frame 138 in which auxiliary image 137 is displayed and the reduction ratio of auxiliary image 137 may be determined automatically or arbitrarily by the user.

[0161] Also, the enlarged / reduced image or the enlarged / reduced image with the auxiliary image overlaid may be displayed, and the target image may be displayed within frame 138. That is, the region to be enlarged is enlarged, and the image before (within frame 138) and after (outside frame 138 of the presented image) the region to be reduced is reduced can be compared.

[0162] This makes it possible to prevent the user's understanding of the surgical field from being confused by an auxiliary image that indicates which parts of the enlarged or reduced image are enlarged and which parts are reduced.

[0163] <Other embodiments> The present technology is not limited to the above-described embodiment, and various other embodiments can be realized.

[0164] In the above first and second embodiments, a reference position is set, and the region to be enlarged is enlarged based on the reference position. This is not limited to the above, and the region to be enlarged may be arbitrarily determined by the user. In other words, when the region to be enlarged is arbitrarily determined by the user, the detection of the reference position may not be performed.

[0165] In the first embodiment described above, the high magnification part is set arbitrarily. However, it is not limited to this, and may be set based on the detected reference position. For example, in the case of inner limiting membrane peeling in FIG. 11, the range of the high magnification part may be set based on the distance between the first reference position (the same position as the reference position 85) detected on the fovea 83 and the second reference position detected on the tip of the ILM forceps 81. Of course, the range of the high magnification part may be set to a fixed distance.

[0166] In the above first embodiment, the range of the enlargement target area (from the reference position to the edge of the enlargement target area) is determined by the control unit 113, and the reduction ratio and range of the reduction target area (from the edge of the enlargement target area in the radial direction to the edge of the reduction target area in the same radial direction) are determined based on the enlargement ratio and range of the enlargement target area. However, the present invention is not limited to this, and the ratio and range of the enlargement target area may be determined arbitrarily.

[0167] 6, for example, when distance B, which is the edge of the area to be reduced, is set to a fixed value, the length from the reference position to distance D, which is the range of the area to be enlarged, is made longer. In other words, the length from distance D to distance B is made shorter.

[0168] In this case, the reduction area becomes narrower, so the target image displayed in the reduction target area is reduced further. Also, the target image is displayed at the same size at distance B, which is the edge of the reduction target area.

[0169] In other words, based on the range and magnification of the area to be enlarged set by the user, the enlarged / reduced image is generated so that a 1x area (solid line 53) in which the positional relationship between the target image and the enlarged / reduced image is equal is set between distance D and distance C (from the reduced area to the image edge of the enlarged / reduced image).

[0170] The information processing method and program related to the present technology may be executed by linking a computer installed in a communication terminal with another computer capable of communicating via a network, etc., thereby constructing an information processing device related to the present technology.

[0171] That is, the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing are both systems.

[0172] The information processing method and program execution according to the present technology by a computer system includes both cases where image recognition, determination of a procedure to be performed, generation of a scaled image, etc. are executed by a single computer, and cases where each process is executed by a different computer. In addition, the execution of each process by a specific computer includes having another computer execute a part or all of the process and acquiring the result.

[0173] In other words, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by multiple devices via a network.

[0174] The configurations of the image information acquisition unit, image recognition unit, presentation image generation unit, control unit, etc., and the control flow of the communication system, etc., described with reference to the drawings, are merely one embodiment, and can be arbitrarily modified within the scope of the present technology. In other words, any other configuration, algorithm, etc. for implementing the present technology may be adopted.

[0175] The effects described in this disclosure are merely examples and are not limiting, and other effects may be present. The description of multiple effects above does not necessarily mean that these effects are exhibited simultaneously. It means that at least one of the effects described above can be obtained depending on the conditions, etc., and it is of course possible that effects not described in this disclosure will be exhibited.

[0176] It is also possible to combine at least two of the characteristic features of each embodiment described above. In other words, the various characteristic features described in each embodiment may be arbitrarily combined without distinction between the embodiments.

[0177] The present technology can also be configured as follows. (1) a microscope optical system; an imaging device that captures an image of a visual field including the eye to be treated through the microscope optical system; an image processing device that enlarges a first region in an image including the eye of the treatment target based on an image captured by the imaging device, and reduces a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image; A surgical microscope system comprising: (2) The surgical microscope system according to (1), The image is at least a part of a photographed image of the eye to be treated. Surgical microscope system. (3) A surgical microscope system according to (1) or (2), The image processing device sets, for the second area, a peripheral area adjacent to the periphery of the first area and an area other than the peripheral area, and reduces the peripheral area. Surgical microscope system. (4) A surgical microscope system according to any one of (1) to (3), The image processing device acquires treatment information including at least one of information on the eye to be treated or information on a treatment to be performed on the eye, and determines the first region based on the treatment information. Surgical microscope system. (5) The surgical microscope system according to (4), The image processing device acquires the treatment information based on the image. Surgical microscope system. (6) A surgical microscope system according to (4) or (5), The image processing device acquires the treatment information from an external source. Surgical microscope system. (7) A surgical microscope system according to any one of (4) to (6), The treatment information includes at least one of the contents of the treatment, an instrument used in the treatment, an illuminated area, an eye part, a lesion, or a contour of the image. Surgical microscope system. (8) A surgical microscope system according to any one of (1) to (7), the image processing device is configured to be able to accept an instruction from a user; and configured to determine the first region based on an instruction from the user. Surgical microscope system. (9) A surgical microscope system according to any one of (1) to (8), The image processing device sets a predetermined position in the first area as a reference position, and enlarges the first area with respect to the reference position. Surgical microscope system. (10) The surgical microscope system according to (9), The image processing device expands the first area so that a distance between the reference position and each position in the first area is expanded. Surgical microscope system. (11) A surgical microscope system according to (9) or (10), The image processing device reduces the second area based on the reference position. Surgical microscope system. (12) A surgical microscope system according to any one of (9) to (11), The image processing device reduces the second region so that a distance between the reference position and each of the outermost positions of the image is maintained. Surgical microscope system. (13) A surgical microscope system according to any one of (9) to (12), The image processing device sets, for the second area, a peripheral area adjacent to a periphery of the first area and an area other than the peripheral area, and reduces the second area so that a distance between the reference position and each position of the area other than the peripheral area is maintained. Surgical microscope system. (14) A surgical microscope system according to any one of (9) to (13), The image processing device acquires treatment information including at least one of information about an eye to be treated or information about a treatment to be performed on the eye, and determines the reference position based on the treatment information. Surgical microscope system. (15) A surgical microscope system according to any one of (9) to (14), the image processing device is configured to be able to accept an instruction from a user; and determining the reference position based on an instruction from the user. Surgical microscope system. (16) A surgical microscope system according to any one of (1) to (15), The image processing device is configured to be capable of generating an auxiliary image that represents an enlargement state of the first region and a reduction state of the second region. Surgical microscope system. (17) The surgical microscope system according to (16), The auxiliary image is an image in which the enlargement state and the reduction state are expressed using at least one of a grid pattern, brightness, saturation, hue, and a boundary line. Surgical microscope system. (18) Acquire an image including the eye to be treated; Enlarging a first region in the image and reducing a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image. The image processing method according to the present invention is carried out by a computer system. (19) acquiring an image including the eye to be treated; enlarging a first region in the image and reducing a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image; A program that causes a computer system to execute the above. (20) An image acquisition unit for acquiring an image including an eye to be treated; an image processing unit that enlarges a first region in the image and reduces a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image; An image processing device comprising: [Explanation of symbols]

[0178] 10…Target image 11…Expanded area 12…Reduced area 20...Enlarged / reduced image 21…Expanded area 22…reduced area 35...Reference position 100...Surgical microscope system 102...Microscope 104...Image processing device 108...User input section 109...Equipment 110...Image information acquisition unit 111…Image recognition section 112...presentation image generating unit 113...Control unit 114...Interface section 137...Auxiliary image

Claims

1. A microscope optical system; an imaging device that captures an image of a visual field including the eye to be treated through the microscope optical system; an image processing device that enlarges a first region in an image including the eye of the treatment target based on an image captured by the imaging device, and reduces a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image; Equipped with The image processing device has a recognition unit that recognizes an instrument used in a treatment performed on the eye from the captured image, and a setting unit that sets a predetermined position in the first area as a reference position, and sets the magnification in a direction opposite to a direction in which the instrument exists so as to be high when a tip of the instrument and the reference position approximately coincide with each other. Surgical microscope system.

2. 2. The surgical microscope system of claim 1, The image is at least a part of a photographed image of the eye to be treated. Surgical microscope system.

3. 2. The surgical microscope system of claim 1, The image processing device sets a peripheral area adjacent to a periphery of the first area and an area other than the peripheral area for the second area, and reduces the peripheral area. Surgical microscope system.

4. 2. The surgical microscope system of claim 1, The image processing device acquires treatment information including at least one of information on the eye to be treated and information on the treatment, and determines the first region based on the treatment information. Surgical microscope system.

5. 5. A surgical microscope system according to claim 4, The image processing device acquires the treatment information based on the image. Surgical microscope system.

6. 5. A surgical microscope system according to claim 4, The image processing device acquires the treatment information from an external source. Surgical microscope system.

7. 5. A surgical microscope system according to claim 4, The treatment information includes at least one of the contents of the treatment, the instrument, the illuminated area, the eye part, the affected area, or the contour of the image. Surgical microscope system.

8. 2. The surgical microscope system of claim 1, the image processing device is configured to be able to accept an instruction from a user; and determining the first region based on an instruction from the user. Surgical microscope system.

9. 2. The surgical microscope system of claim 1, The image processing device expands the first area so that a distance between the reference position and each position in the first area is expanded. Surgical microscope system.

10. 2. The surgical microscope system of claim 1, The image processing device reduces the second area based on the reference position. Surgical microscope system.

11. 2. The surgical microscope system of claim 1, The image processing device reduces the second area so that a distance between the reference position and each of the outermost positions of the image is maintained. Surgical microscope system.

12. 2. The surgical microscope system of claim 1, The image processing device sets, for the second area, a peripheral area adjacent to a periphery of the first area and an area other than the peripheral area, and reduces the second area so that a distance between the reference position and each position of the area other than the peripheral area is maintained. Surgical microscope system.

13. 2. The surgical microscope system of claim 1, The image processing device acquires treatment information including at least one of information about an eye to be treated or information about a treatment to be performed on the eye, and determines the reference position based on the treatment information. Surgical microscope system.

14. 2. The surgical microscope system of claim 1, the image processing device is configured to be able to accept an instruction from a user; and determining the reference position based on an instruction from the user. Surgical microscope system.

15. 2. The surgical microscope system of claim 1, The image processing device is configured to be capable of generating an auxiliary image that represents an enlargement state for the first region and a reduction state for the second region. Surgical microscope system.

16. 16. A surgical microscope system according to claim 15, The auxiliary image is an image in which the enlargement state and the reduction state are expressed using at least one of a grid pattern, brightness, saturation, hue, and a boundary line. Surgical microscope system.

17. Obtaining an image including the eye to be treated; Recognizing from the image an instrument to be used in a procedure to be performed on the eye; enlarging a first region in the image and reducing a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image; A predetermined position within the first region is set as a reference position; When the tip of the instrument and the reference position are substantially aligned, the magnification is set to be high in the direction opposite to the direction in which the instrument is present. The image processing method according to the present invention is carried out by a computer system.

18. acquiring an image including the eye to be treated; enlarging a first region in the image and reducing a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image; Recognizing from the image an instrument used in a procedure performed on the eye; setting a predetermined position within the first region as a reference position; A step of setting the magnification in a direction opposite to the direction in which the instrument is present so as to be high when the tip of the instrument and the reference position are substantially aligned; A program that causes a computer system to execute the above.

19. An image acquisition unit that acquires an image including the eye to be treated; an image processing unit that enlarges a first region in the image and reduces a second region other than the first region in the image so that the second region can be displayed in a region other than the enlarged region, which is the enlarged first region in the image; a recognition unit that recognizes an instrument used in a treatment performed on the eye from the image, and a setting unit that sets a predetermined position within the first area as a reference position, The image processing unit sets the magnification in a direction opposite to the direction in which the instrument is present so as to be high when the tip of the instrument and the reference position are substantially aligned. Image processing device.

Images