Medical Observation System, Medical Observation Device, and Medical Observation Method

Abstract

To provide a medical observation system, a medical observation device and a medical observation method which allow a user to stably observe an affected portion at a distant position by magnifying the affected portion.SOLUTION: A medical observation system comprises: an imaging device which obtains a surgical field image by imaging a surgical field; a three-dimensional information generation unit which generates three-dimensional information regarding the surgical field, based on the surgical field image captured by the imaging device; a setting unit which sets at least one region-of-interest on the basis of the at least one surgical field image captured by the imaging device at predetermined timing; an estimation unit which estimates an existence position of the region-of-interest from within the surgical field image captured at timing different from the predetermined timing on the basis of the three-dimensional information and the position of the region-of-interest set by the setting unit; a magnified image generation unit which generates a magnified surgical field image in which the estimated region-of-interest is magnified by a predetermined magnification; and a display control unit which outputs at least the magnified surgical field image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a medical observation system, a medical observation device, and a medical observation method.

Background Art

[0002] In recent years, there has been an increasing number of cases where a surgical field image is displayed on a large-screen display device by a medical observation device such as a surgical endoscope or a surgical microscope, and a surgeon performs surgery while monitoring the surgical field image. The endoscope is used while being inserted into the body. Therefore, the lens of the endoscope is soiled or fogged due to bleeding from the body or the scattering of smoke and oil accompanying the use of an energy device that cuts and peels tissues or seals blood vessels by high-frequency current or ultrasonic vibration. Therefore, it was necessary to frequently remove the endoscope and clean the lens. Therefore, in order to suppress dirt and fogging of the lens, for example, as shown in Patent Document 1, a technique of observing a surgical field (operative field) by enlarging it from a distant position has been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique disclosed in Patent Document 1 tracks an electronic zoom to that location using feature points. Therefore, for example, in a surgical operation, when an endoscope inserted into the body cavity is frequently moved in various directions during the operation for observation, the image captured by the endoscope involves large movements, so the tracking performance for feature points is not sufficient. In addition, when a treatment is applied to the target tissue, the appearance of that part changes, so there is a problem that it becomes difficult to track the feature points. As a result, it has not been possible to stably observe the portion to be enlarged.

[0005] Therefore, the present disclosure proposes a medical observation system, a medical observation device, and a medical observation method that can stably observe an affected area by magnifying it from a remote position.

Means for Solving the Problems

[0006] In order to solve the above problems, a medical observation system according to one aspect of the present disclosure includes an imaging device that images an operative field to obtain an operative field image, a three-dimensional information generation unit that generates three-dimensional information of the operative field from the operative field image imaged by the imaging device, a setting unit that sets at least one region of interest in at least one of the operative field images imaged by the imaging device at a predetermined timing, an estimation unit that estimates the position where the region of interest exists from among the operative field images imaged at a timing different from the predetermined timing based on the three-dimensional information and the position of the region of interest set by the setting unit, an enlarged image generation unit that generates an enlarged operative field image obtained by magnifying the estimated region of interest at a predetermined magnification, and a display control unit that outputs at least the enlarged operative field image.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0009] (First Embodiment) [Configuration of a Medical Observation System According to the First Embodiment] FIG. 1 is a diagram showing an example of a schematic configuration of an endoscope surgery system 5000 to which a medical observation system according to the present disclosure can be applied. In FIG. 1, a state where a surgeon (doctor) 5061 is performing a surgery on a patient 5071 on a patient bed 5069 using the endoscope surgery system 5000 is illustrated. A scopist 5062 is holding an endoscope 5001 and inserting it into the body cavity of the patient 5071. An assistant 5063 is holding a surgical instrument 5017 and inserting it into the body cavity of the patient 5071.

[0010] In endoscopic surgery, instead of making an incision in the abdominal wall to open the abdomen, a plurality of cylindrical trocars called trocars 5025a to 5025d are punctured into the abdominal wall. Then, from the trocars 5025a to 5025d, the lens barrel 5003 of the endoscope 5001 and other surgical instruments 5017 are inserted into the body cavity of the patient 5071. In the example of FIG. 1, as the other surgical instruments 5017, a pneumoperitoneum tube 5019, an energy treatment instrument 5021, and forceps 5023 are inserted into the body cavity of the patient 5071. The pneumoperitoneum tube 5019 sends gas into the body cavity to inflate the body cavity of the patient 5071 for the purpose of securing the visual field by the endoscope 5001 and the working space for the surgeon 5061. The energy treatment instrument 5021 is a treatment instrument that performs tissue incision and dissection or blood vessel sealing by high-frequency current or ultrasonic vibration. Although not shown in FIG. 1, the pneumoperitoneum tube 5019 and the energy treatment instrument 5021 are connected to a control device (not shown), and the surgical instrument 5017 that receives instructions from the surgeon 5061 or the like performs a predetermined operation. The surgical instrument 5017 shown in the figure is merely an example, and as the surgical instrument 5017, various surgical instruments generally used in endoscopic surgery, such as forceps and retractors, may be used.

[0011] An image of the surgical field in the body cavity of the patient 5071 captured by the endoscope 5001 (hereinafter referred to as the surgical field image) is displayed on the display device 50. While viewing the surgical field image displayed on the display device 50 in real time, the surgeon 5061 performs a treatment such as excising the affected part using the energy treatment instrument 5021 or the forceps 5023. Also, while viewing the surgical field image displayed on the display device 50 in real time, the scopist 5062 adjusts the position of the endoscope 5001 so that the affected part is reflected in the surgical field image. During the surgery, the pneumoperitoneum tube 5019, the energy treatment instrument 5021, and the forceps 5023 are held by the surgeon 5061 or the assistant 5063 or the like.

[0012] [Schematic Configuration of Endoscope] The endoscope 5001 is composed of a lens barrel 5003 (also referred to as a scope) with a region of a predetermined length inserted into the body cavity of the patient 5071 from its tip, and a camera head 5005 connected to the proximal end of the lens barrel 5003. In the example of FIG. 1, an endoscope 5001 configured as a so-called rigid endoscope having a rigid lens barrel 5003 is illustrated, but the endoscope 5001 may be configured as a so-called flexible endoscope having a flexible lens barrel 5003.

[0013] An opening in which an objective lens is fitted is provided at the tip of the lens barrel 5003. A light source device (not shown) is connected to the endoscope 5001, and the light generated by the light source device is guided to the tip of the lens barrel 5003 by a light guide extending inside the lens barrel 5003 and irradiated toward the observation target in the body cavity of the patient 5071 through the objective lens. Note that the endoscope 5001 may be a direct-view endoscope, a forward-oblique-view endoscope, or a side-view endoscope.

[0014] An optical system and an imaging element are provided inside the camera head 5005, and the reflected light (observation light) from the observation target is condensed onto the imaging element by the optical system. The observation light is photoelectrically converted by the imaging element, and an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image, is generated. The image signal is transmitted as RAW data to a camera control unit (CCU) 12a. Note that the camera head 5005 is equipped with a function of adjusting the magnification and focal length by appropriately driving its optical system.

[0015] In addition, for example, in order to support stereoscopic vision (3D display) or the like, a plurality of imaging elements may be provided in the camera head 5005. In this case, a plurality of relay optical systems are provided inside the lens barrel 5003 to guide the observation light to each of the plurality of imaging elements.

[0016] The endoscopic surgery system 5000 includes an input device that receives various information inputs and instruction inputs from the user, i.e., the surgeon 5061, the scopist 5062, or the assistant 5063. For example, the user inputs various information related to the surgery, such as the patient's physical information and information about the surgical procedure, via the input device. Also, for example, the user inputs an instruction to change the imaging conditions (such as the type of irradiation light, magnification, and focal length) by the endoscope 5001 and an instruction to drive the surgical instrument 5017 such as the energy treatment instrument 5021 via the input device.

[0017] The type of the input device is not limited, and the input device may be various known input devices. As the input device, for example, a mouse, a keyboard, a touch panel, a switch, and / or a lever, etc. may be applied. FIG. 1 shows an example in which the scopist 5062 performs information input using a foot switch 5057 which is an example of the input device. For example, the scopist 5062 sets a region of interest in the surgical field image via the foot switch 5057. Details will be described later. When a touch panel is used as the input device, the touch panel may be provided on the display surface of the display device 50.

[0018] [Explanation of the Configuration of the Medical Observation System According to the First Embodiment] FIG. 2 is a functional block diagram showing the functional configuration of the medical observation system 10a applied to endoscopic surgery. The medical observation system 10a is, for example, a system that is applied to the above-described endoscopic surgery system 5000 and monitors the surgical field image by the endoscope 5001 inserted into the patient's body cavity during the surgery. In particular, the medical observation system 10a is a system that always displays an enlarged surgical field image with the set region of interest enlarged regardless of the position and orientation of the endoscope 5001 based on the three-dimensional position of the surgical field.

[0019] The medical observation system 10a includes an imaging device 42a and a camera control unit 12a. The imaging device 42a is mounted on the camera head 5005 of the endoscope 5001 described above, and images the surgical field in the patient's 5071 body cavity to obtain a surgical field image. The camera control unit 12a generates a surgical field image and generates three-dimensional information of the surgical field when the imaging device 42a performs imaging.

[0020] The imaging device 42a includes an image sensor 44a. The image sensor 44a is composed of an image sensor (photoelectric conversion element) such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and converts light from the surgical field into an electrical signal.

[0021] The camera control unit 12a includes a three-dimensional information generation unit 14, a development processing unit 18, a target area setting unit 20, a target area estimation unit 22, a three-dimensional map data storage unit 24, a zoom processing unit 26, and a display control unit 40. The camera control unit 12a always generates an enlarged surgical field image with the target area enlarged and displays it on the display device 50 regardless of the position and orientation of the endoscope. Note that the camera control unit 12a is an example of the medical observation device in the present disclosure.

[0022] The three-dimensional information generation unit 14 calculates the three-dimensional position of, for example, the surgical field image in the body cavity imaged by the image sensor 44a. The three-dimensional information generation unit 14 includes a map generation unit 15 and a self-position estimation unit 16. The map generation unit 15 generates a three-dimensional map (hereinafter simply referred to as a map) indicating the three-dimensional position of the surgical field and the three-dimensional position of the target area described later. The method of generating the map will be described later. The self-position estimation unit 16 estimates the self-position and orientation of the endoscope 5001 at the predetermined timing based on the generated map and the surgical field image captured at the predetermined timing.

[0023] The developing processing unit 18 performs developing processing to convert imaging data into a visible image. The developing processing unit 18 performs various image processes for displaying an image, such as developing processing (demosaicing processing), on the RAW data output by the imaging element 44a. More specifically, the developing processing unit 18 makes the RAW data into visible image data by applying a preset digital gain and gamma curve to the RAW data. Note that it is desirable to adjust in advance the digital gain and gamma curve to be set so that image data that is easy for the operator 5061 or scopist 5062 to view is generated.

[0024] The attention area setting unit 20 designates an area of interest, such as a tumor to be removed by surgery, from among the operative field images captured by the imaging element 44a and converted by the developing processing unit 18 to be visible. More specifically, the operator of the medical observation system 10a sets at least one area of interest from among the operative field images while monitoring the operative field images on a display device 50 such as a liquid crystal monitor. A specific method for setting the area of interest will be described later. Note that the attention area setting unit 20 is an example of the setting unit in the present disclosure.

[0025] The attention area estimation unit 22 estimates the position where the area of interest exists in the operative field image at an arbitrary timing. Note that the attention area estimation unit 22 is an example of the estimation unit in the present disclosure.

[0026] The three-dimensional map data storage unit 24 stores the three-dimensional map of the operative field generated by the map generation unit 15 described above. Note that the three-dimensional map stored in the three-dimensional map data storage unit 24 is updated as time passes.

[0027] The zoom processing unit 26 generates an enlarged operative field image in which the operative field estimated at that timing is enlarged based on the position where the region of interest estimated by the region of interest estimation unit 22 exists. Note that the zoom processing unit 26 is an example of the enlarged image generation unit in the present disclosure. The zoom processing unit 26 performs electronic zoom processing on the operative field image, for example, by interpolating pixel values between pixels. The interpolation of pixel values may be performed using a known method such as the two-nearest neighbor method, the bilinear method, the bicubic method, or the Lanczos method. Further, the zoom processing unit 26 may perform electronic zoom by adding super-resolution processing.

[0028] Note that the zoom magnification may be a predetermined magnification set in advance, or may be automatically determined by the zoom processing unit 26 from the size of the region of interest. Further, a user such as the scopist 5062, who is an operator, may specify the magnification.

[0029] The display control unit 40 performs display control to output the operative field image generated by the development processing unit 18 and the enlarged operative field image generated by the zoom processing unit 26 to the display device 50. As the display device 50, various known display devices such as a liquid crystal display device or an EL (Electro Luminescence) display device can be applied. Note that the display device 50 includes at least a first display area 52a for displaying the enlarged operative field image. Further, as shown in FIG. 2, the display device 50 may include a second display area 52b for displaying the operative field image together with the first display area 52a. In that case, the display device 50 may include the first display area 52a and the second display area 52b in one monitor, or the display device 50 may be composed of two different monitors, and each monitor may include the first display area 52a and the second display area 52b, respectively.

[0030] [Explanation of method for generating three-dimensional map] Next, a method for the map generation unit 15 to generate a three-dimensional map of the operative field will be described. FIG. 3 is a diagram for explaining a method for the map generation unit 15 to generate a three-dimensional map of the operative field.

[0031] FIG. 3 shows a state in which the imaging device 42a observes the stationary object 100 in the three-dimensional space XYZ with the point in space as the reference position O. The imaging device 42a captures the surgical field image K(x, y, t) at a predetermined timing, for example, at time t, and captures the surgical field image K(x, y, t+Δt) at a timing different from the predetermined timing, for example, at time t+Δt. The time interval Δt is set to, for example, 33 msec. The reference position O may be set arbitrarily, but it is preferably set at a position that does not move with time. In the surgical field image K(x, y, t), x represents the horizontal coordinate of the image, and y represents the vertical coordinate of the image.

[0032] First, the map generation unit 15 detects feature points, which are characteristic pixels, from the surgical field image K(x, y, t) and the surgical field image K(x, y, t+Δt). The feature point is, for example, a pixel whose pixel value differs by a predetermined value or more from adjacent pixels. The feature point is preferably a point that stably exists even as time passes. For example, pixels that form an edge in the image are often used. Here, for the sake of simplicity of the following explanation, it is assumed that the feature points A1, B1, C1, D1, E1, F1, H1, which are the vertices of the object 100, are detected from the surgical field image K(x, y, t).

[0033] Next, the map generation unit 15 searches for points corresponding to the feature points A1, B1, C1, D1, E1, F1, H1 from the surgical field image K(x, y, t+Δt). Specifically, based on the pixel value of the feature point A1, the pixel values in the vicinity of the feature point A1, etc., a point having the same characteristics is searched for from the surgical field image K(x, y, t+Δt). By this search process, it is assumed that the feature points A2, B2, C2, D2, E2, F2, H2 corresponding to the feature points A1, B1, C1, D1, E1, F1, H1 are respectively detected from the surgical field image K(x, y, t+Δt).

[0034] Subsequently, based on the principle of three-dimensional measurement, the map generation unit 15 calculates, for example, the three-dimensional coordinates (X A , Y A , Z A ) of the point A in space from the two-dimensional coordinates of the feature point A1 on the in-vivo image K(x, y, t+Δt) and the two-dimensional coordinates of the feature point A2 on the in-vivo image K(x, y, t+Δt). As a set of the thus calculated three-dimensional coordinates (X A , Y A , Z A ), a three-dimensional map D(X, Y, Z) of the space where the object 100 is placed is generated. The generated three-dimensional map D(X, Y, Z) is stored in the three-dimensional map data storage unit 24. Note that the three-dimensional map D(X, Y, Z) is an example of the three-dimensional information in the present disclosure.

[0035] During the time interval Δt, since the position and orientation of the imaging device 42a change, the map generation unit 15 also estimates the position and orientation of the imaging device 42a simultaneously. Mathematically, taking the three-dimensional coordinates of each feature point constituting the object 100, the position and orientation of the imaging device 42a as unknowns, a set of simultaneous equations is established based on the two-dimensional coordinates of the feature points observed in the in-vivo image K(x, y, t) and the in-vivo image K(x, y, t+Δt). The map generation unit 15 estimates the three-dimensional coordinates of each feature point constituting the object 100 and the position and orientation of the imaging device 42a by solving this set of simultaneous equations.

[0036] In this way, by detecting a plurality of feature points from the surgical field image K(x, y, t) captured by the imaging device 42a and detecting points corresponding to those feature points from the surgical field image K(x, y, t+Δt), a three-dimensional map D(X, Y, Z) of the environment observed by the imaging device 42a can be generated. Further, the position and orientation of the imaging device 42a, that is, its own position can be estimated. Also, by repeatedly executing the above-described processing, for example, feature points that were not initially visible become visible, and thus the three-dimensional map D(X, Y, Z) can be expanded. Also, by repeating the processing, the three-dimensional positions of the same feature points can be repeatedly calculated, so that, for example, by performing an averaging process, the calculation error can be reduced. In this way, the three-dimensional map D(X, Y, Z) stored in the three-dimensional map data storage unit 24 is updated at any time. Note that the technique of creating a three-dimensional map of the environment and specifying the own position of the imaging device 42a is generally called SLAM (Simultaneous Localization and Mapping) technology.

[0037] The basic principle of the SLAM technology using a monocular camera is described, for example, in “Andrew J. Davison, ‘Real-Time Simultaneous Localization and Mapping with a Single Camera’, Proceedings of the 9th IEEE International Conference on Computer Vision Volume 2, 2003, pp.1403-1410”. Also, the SLAM technology that estimates the three-dimensional position of a subject using the camera image of the subject is particularly also called Visual SLAM.

[0038] [Explanation of the method for setting the region of interest] The region of interest is set by the action of the region of interest setting unit 20. Specifically, the region of interest setting unit 20 performs this by superimposing and displaying a region of interest frame indicating the region of interest on the surgical field image and specifying the size, shape, and position of the region of interest frame.

[0039] FIG. 4 is a diagram showing an example of a method for setting a target frame. FIG. 4A is a diagram showing an example of an operative field image K(x, y) observed with the endoscope 5001. In the following description, information on the timing (e.g., time) at which the operative field image is captured is omitted, and the operative field image is simply described as K(x, y). FIG. 4B is a diagram showing an example of a state in which the orientation of the endoscope 5001 is adjusted so that the affected part to be set as the target area is reflected in the center of the operative field image K(x, y), and the target area setting unit 20 sets a target frame 110 indicating the target area. FIG. 4C is a diagram showing an example of an enlarged operative field image L(x, y) obtained by enlarging and displaying the area including the target frame 110 at a predetermined magnification.

[0040] The scopist 5062 moves the endoscope 5001 so that a specific position to be enlarged, such as an affected part, is reflected in the center (an example of a predetermined position) of the operative field image K(x, y), for example, while looking at the operative field image K(x, y) shown in FIG. 4A.

[0041] As shown in FIG. 4B, when the specific position to be focused on is reflected in the center (an example of a predetermined position) of the operative field image K(x, y), the scopist 5062 steps on the foot switch 5057 (FIG. 1) to instruct the target area setting unit 20 to set the target area. At this time, a setting signal for instructing the setting of the target area is generated using the depression of the foot switch 5057 as a trigger. Then, the target area setting unit 20 sets the target area by displaying a target frame 110 of a predetermined size in the center of the operative field image K(x, y) as shown in FIG. 4B on the condition that the setting signal is input. Note that the size and shape of the target frame 110 may be set arbitrarily, which will be described in detail later.

[0042] Note that the method by which the attention area setting unit 20 sets the attention area is not limited to the method described above. For example, a touch panel may be installed by laminating it on the screen of the display device 50, and by detecting the operation of the touch panel, the attention area may be set at the position where the touch panel is pressed. Also, the position and shape of the attention area may be set using a mouse. Furthermore, the attention area setting unit 20 may set the position and shape of the attention area based on operations such as gestures.

[0043] FIG. 5 is a diagram showing another example of a method for setting an attention frame. FIG. 5A is a diagram showing an example of an operative field image K(x, y) observed with the endoscope 5001. While looking at the operative field image K(x, y) displayed on the display device 50, the scopist 5062 designates the position of the area to be focused on using an input device such as a touch panel or a mouse. The attention area setting unit 20 causes the attention area instruction information 105 indicating the designated area to be displayed superimposed on the operative field image K(x, y).

[0044] Subsequently, the attention area setting unit 20 sets an attention frame 110 at the position of the input attention area instruction information 105. The attention area setting unit 20 causes the set attention frame 110 to be displayed superimposed on the operative field image K(x, y) as shown in FIG. 5B. Note that the attention frame 110 may be a frame with a preset size and shape, or may be a closed area imitating the attention area instruction information 105.

[0045] Thereafter, regardless of the position and orientation of the endoscope 5001, the zoom processing unit 26 generates and displays an enlarged operative field image L(x, y) obtained by enlarging the set attention frame 110 at a predetermined magnification as shown in FIG. 5C.

[0046] Note that the attention area setting unit 20 may set an attention area by using the above-described three-dimensional map D(X, Y, Z) and taking into account conditions such as the distance in the three-dimensional space and the distance from the imaging system being within a certain range. Further, the display form of the attention frame 110 is not limited to those shown in FIGS. 4 and 5. Variations in the display form of the attention frame 110 will be described later (see FIG. 18). Furthermore, the attention area setting unit 20 may set the position and shape of the attention area based on operations such as gestures.

[0047] Subsequently, as shown in FIG. 4C, the zoom processing unit 26 generates an enlarged surgical field image L(x, y) by enlarging the area including the attention frame 110 of the surgical field image K(x, y) at a predetermined magnification. At this time, as shown in FIG. 4C, the attention frame 110 is also enlarged and displayed at a predetermined magnification. Then, the display control unit 40 outputs the generated enlarged surgical field image L(x, y) to the display device 50 for display. The surgeon 5061 performs the surgery while observing the enlarged surgical field image L(x, y) displayed on the display device 50.

[0048] Although not shown in FIG. 4, after generating the enlarged surgical field image L(x, y), the medical observation system 10a repeats the imaging and display of the surgical field image K(x, y) at a predetermined time interval Δt. And each time the surgical field image K(x, y) is imaged, the generation and display of a new enlarged surgical field image L(x, y) are repeated.

[0049] [Explanation of the method for estimating the existence position of the attention area] Thereafter, when the observation time of the surgical field image K(x, y) elapses, the position and orientation of the endoscope 5001 may change with time. Then, the attention area estimation unit 22 estimates the existence position of the attention area from the surgical field image K(x, y). The zoom processing unit 26 generates an enlarged surgical field image L(x, y) by enlarging the estimated attention area at a predetermined magnification. The display control unit 40 outputs the enlarged surgical field image L(x, y) to the display device 50 for display as shown in FIG. 4C. By continuing such processing, the medical observation system 10a continues to display the enlarged surgical field image L(x, y) on the display device 50.

[0050] Here, a method for the attention area estimation unit 22 to estimate the existence position of the attention area from the intraoperative field image K(x, y) when the position and orientation of the endoscope 5001 change will be described.

[0051] Based on the position and orientation of the endoscope 5001 at a predetermined timing, for example, time t, the position and orientation of the endoscope 5001 at a timing different from the predetermined timing, for example, time t + Δt, and the three-dimensional map D(X, Y, Z), the attention area estimation unit 22 estimates at which position in the intraoperative field image K(x, y, t + Δt) the attention frame 110 at time t will be observed at time t + Δt.

[0052] Specifically, the attention area estimation unit 22 identifies how a plurality of feature points in the vicinity of the set attention frame 110 move from time t to time t + Δt based on the position and orientation of the endoscope 5001. Then, the attention area estimation unit 22 estimates the position of the attention area based on the movement state of the identified feature points.

[0053] Note that the area set as the attention area is generally often the affected part that is the target of the surgery. The affected part is likely to be resected, bleed, or greatly deformed by the surgery. Therefore, even if a feature point is set inside the attention area, there is a possibility that the feature point will disappear over time. Therefore, it is desirable to extract the feature points from the area excluding the periphery of the attention area from the intraoperative field image K(x, y) after setting the attention area.

[0054] FIG. 6 is an image showing an example of setting an area for extracting feature points. As shown in FIG. 6, the above-described map generation unit 15 sets a mask 120 around the screen while avoiding the central part of the screen where the attention frame 110 is set. Then, the map generation unit 15 extracts feature points only inside the set mask 120. Since the area of the set mask 120 is away from the attention frame 110 indicating the position of the attention area, it is assumed that there is little deformation during the operation. Therefore, feature points can be stably detected inside the mask 120 regardless of the passage of time. And since feature points can be stably extracted, the stability of the estimation accuracy of the three-dimensional map D(X, Y, Z) and the position and orientation of the endoscope 5001 is improved.

[0055] Note that in the surgical field image K(x, y), there may be cases where surgical instruments such as forceps 5023 or objects unrelated to the surgical field such as the fingers of the surgeon are reflected inside the mask 120. The feature points constituting these objects are likely to move irregularly over time. That is, since there is no guarantee that they stably exist in the surgical field image K(x, y), it is desirable to extract feature points after removing these objects. Therefore, the map generation unit 15 may be provided with a function of removing objects such as surgical instruments and fingers registered in advance from the surgical field image K(x, y). This removal function is, for example, a function of recognizing a pre-registered object by image recognition and not including the area where the recognized object exists in the calculation target.

[0056] [Explanation of the image displayed by the medical observation system according to the first embodiment] FIG. 7 is a diagram showing an example of an image to be displayed by the medical observation system 10a. As shown in FIG. 7, the display control unit 40 outputs the surgical field image K(x, y) monitored by the scopist 5062 to the display device 50a (second display area 52b) for display. Further, the display control unit 40 outputs the magnified surgical field image L(x, y) monitored by the surgeon 5061 to a display device 50b (first display area 52a) different from the display device 50a for display. By adopting such a display form, the surgeon 5061 and the scopist 5062 can arrange the display devices 50a and 50b at positions that are easy to view respectively. Therefore, the surgeon 5061 can easily proceed with the surgery while observing the magnified surgical field image L(x, y). Also, the scopist 5062 can easily adjust the position of the endoscope 5001 while observing the surgical field image K(x, y).

[0057] Note that, as shown in FIG. 7, in the surgical field image K(x, y), the above-mentioned attention frame 110 and the zoom frame 112 indicating the range of the magnified surgical field image L(x, y) may be displayed. The attention frame 110 and the zoom frame 112 move within the surgical field image K(x, y) according to the movement of the endoscope 5001. In this way, by displaying the attention frame 110 and the zoom frame 112 in the surgical field image K(x, y), the scopist 5062 can immediately confirm whether an appropriate range is displayed in the magnified surgical field image L(x, y) by only checking the surgical field image K(x, y). Also, when it is not necessary to display the attention frame 110 and the zoom frame 112, these displays may be independently turned on / off according to the operation instruction of the scopist 5062.

[0058] Note that since the medical observation system 10a generates the three-dimensional map D(X, Y, Z) and estimates the position and orientation of the endoscope 5001, it can calculate the three-dimensional positions of the feature points in the vicinity of the attention area. Therefore, by performing perspective transformation and / or rotation transformation on the captured surgical field image K(x, y), it is also possible to generate and display the magnified surgical field image L(x, y) that always views the attention area from the same direction.

[0059] [Explanation of the Processing Flow Performed by the Medical Observation System According to the First Embodiment] Next, the processing flow performed by the medical observation system 10a according to the first embodiment will be described. FIG. 7 is a flowchart showing an example of the processing flow performed by the medical observation system 10a.

[0060] Hereinafter, the flowchart of FIG. 8 will be described. First, the image sensor 44a captures the surgical field image K(x, y) (step S10).

[0061] The map generation unit 15 extracts feature points from the captured surgical field image K(x, y) (step S11).

[0062] Furthermore, the image sensor 44a captures the surgical field image K(x, y) at a predetermined timing, for example, after Δt seconds (step S12).

[0063] The map generation unit 15 extracts feature points from the captured surgical field image K(x, y) after Δt seconds (step S13).

[0064] The map generation unit 15 generates a three-dimensional map D(X, Y, Z) by calculating the three-dimensional positions of the feature points (step S14).

[0065] The self-position estimation unit 16 estimates the position and orientation of the endoscope 5001 (step S15).

[0066] The region of interest setting unit 20 sets a region of interest in the surgical field image K(x, y) (step S16).

[0067] The zoom processing unit 26 generates an enlarged surgical field image L(x, y). Then, the display control unit 40 causes the generated enlarged surgical field image L(x, y) to be displayed on the display device 50 (step S17).

[0068] The display control unit 40 determines whether there is an instruction to end the process (step S18). If it is determined that there is an end instruction (step S18: Yes), the medical observation system 10a ends the process of FIG. 8. On the other hand, if it is not determined that there is an end instruction (step S18: No), the process proceeds to step S19. Note that the instruction to end the process is performed, for example, by detecting an operation such as turning off the power switch (not shown) of the camera control unit 12a.

[0069] When it is determined as No in step S18, the imaging element 44a captures the surgical field image K(x, y) at a predetermined timing, for example, after Δt seconds (step S19).

[0070] The map generation unit 15 extracts feature points from the captured surgical field image K(x, y) after Δt seconds (step S20).

[0071] The map generation unit 15 updates the three-dimensional map D(X, Y, Z) generated in step S14 by calculating the three-dimensional positions of the feature points (step S21).

[0072] The self-position estimation unit 16 estimates the position and orientation of the endoscope 5001 (step S22).

[0073] The region of interest estimation unit 22 estimates the position of the region of interest in the surgical field image K(x, y) captured in step S19 after Δt seconds (step S23). Then, the process returns to step S17.

[0074] [Explanation of the Effects of the First Embodiment] As described above, according to the medical observation system 10a of the first embodiment, the three-dimensional information generation unit 14 generates a three-dimensional map D(X, Y, Z) (three-dimensional information) of the surgical field from the surgical field image K(x, y) captured by the imaging device 42a. Then, the attention area setting unit 20 (setting unit) sets at least one attention area in the surgical field image K(x, y) captured at a predetermined timing. The attention area estimation unit 22 (estimation unit) estimates the existence position of the attention area from the surgical field image K(x, y) captured at a timing different from the predetermined timing based on the three-dimensional map D(X, Y, Z) and the position of the attention area set by the attention area setting unit 20. Then, the zoom processing unit 26 (magnified image generation unit) generates a magnified surgical field image L(x, y) obtained by magnifying the estimated attention area at a predetermined magnification, and the display control unit 40 outputs at least the magnified surgical field image L(x, y). Therefore, even when the endoscope 5001 equipped with the imaging device 42a changes its position and posture, it is possible to continue observing the affected part by magnifying it from a distant position.

[0075] Also, according to the medical observation system 10a of the first embodiment, the display control unit 40 displays the surgical field image K(x, y) and the magnified surgical field image L(x, y). Therefore, both the magnified surgical field image L(x, y) that the surgeon 5061 wants to see and the surgical field image K(x, y) that the scopist 5062 wants to see can be displayed.

[0076] Also, according to the medical observation system 10a of the first embodiment, the display control unit 40 displays the surgical field image K(x, y) and the magnified surgical field image L(x, y) on two display devices 50a and 50b, respectively. Therefore, the surgeon 5061 and the scopist 5062 can arrange the display devices 50a and 50b at positions where they are easy to view, respectively.

[0077] Also, according to the medical observation system 10a of the first embodiment, the attention area setting unit 20 (setting unit) designates the specific position as the attention area on the condition that a setting signal for instructing the setting of the attention area is generated while matching the specific position of the surgical field image K(x, y) displayed by the display control unit 40 on the display device 50 with a predetermined position of the display device 50. Therefore, the attention area can be easily and surely set by a typical operation.

[0078] Also, according to the medical observation system 10a of the first embodiment, the attention area setting unit 20 (setting unit) sets the attention area at the position indicated by the input device on the surgical field image K(x, y) displayed by the display control unit 40 on the display device 50. Therefore, the attention area can be easily and surely set by an intuitive operation.

[0079] Also, according to the medical observation system 10a of the first embodiment, the imaging device 42a includes one imaging element 44a, and the three-dimensional information generation unit 14 generates a three-dimensional map D(X, Y, Z) (three-dimensional information) of the surgical field based on at least two surgical field images K(x, y) captured by the imaging device 42a at different times. Therefore, it is possible to continuously observe the affected part enlarged from a distant position using the imaging device 42a having a simple configuration of only a monocular camera.

[0080] Also, according to the medical observation system 10a of the first embodiment, the imaging device 42a is mounted on the endoscope 5001. Therefore, when performing surgery or the like using the endoscope 5001, the surgeon 5061 can stably observe the affected part while enlarging it.

[0081] Also, according to the camera control unit 12a (medical observation device) of the first embodiment, the three-dimensional information generation unit 14 generates a three-dimensional map D(X,Y,Z) (three-dimensional information) of the surgical field from the surgical field image K(x,y) that images the surgical field. Then, the attention area setting unit 20 (setting unit) sets at least one attention area in the surgical field image K(x,y) captured at a certain time. The attention area estimation unit 22 (estimation unit) estimates the existence position of the attention area from the surgical field image K(x,y) captured at a time different from the above time, based on the three-dimensional map D(X,Y,Z) and the position of the attention area set by the attention area setting unit 20. Then, the zoom processing unit 26 (enlarged image generation unit) generates an enlarged surgical field image L(x,y) obtained by enlarging the estimated attention area at a predetermined magnification, and the display control unit 40 displays at least the enlarged surgical field image L(x,y). Therefore, the affected part can be continuously enlarged and observed.

[0082] In the medical observation system 10a, the endoscope 5001 incorporating the imaging device 42a may be equipped with an acceleration sensor such as a gyro sensor. By monitoring the output of the acceleration sensor, the position and orientation of the endoscope 5001 can be measured in real time. Therefore, the position and orientation of the endoscope 5001 can be measured without the imaging device 42a capturing two images at different times, and thereby the position of the attention area can be estimated.

[0083] (Second Embodiment) The configuration of the medical observation system 10a is not limited to the configuration described in the first embodiment, and various modifications can be realized. Hereinafter, another embodiment of the medical observation system will be described in sequence.

[0084] FIG. 9 is a diagram showing an example of the display form of the image output from the display control unit 40 to the display device 50. That is, in the first embodiment, an example in which the display control unit 40 outputs only the magnified surgical field image L(x, y) to the display device 50 and an example in which the surgical field image K(x, y) and the magnified surgical field image L(x, y) are output to different display devices 50a and 50b, respectively, were described. However, the display form of the output image is not limited to these.

[0085] FIG. 9A is an example in which the display control unit 40 displays the surgical field image K(x, y) and the magnified surgical field image L(x, y) adjacent to each other (side by side) on the display device 50. That is, the magnified surgical field image L(x, y) is displayed in the first display area 52a set on the display screen of the display device 50, and the surgical field image K(x, y) is displayed in the second display area 52b. By adopting such a display form, the surgeon 5061 can proceed with the surgery while observing the magnified surgical field image L(x, y), and the scopist 5062 can adjust the position of the endoscope 5001 while observing the surgical field image K(x, y).

[0086] FIG. 9B is an example in which the display control unit 40 superimposes the surgical field image K(x, y) on a part of the magnified surgical field image L(x, y) (in PinP) and displays it on the display device 50. In this case, the second display area 52b is superimposed on a part of the first display area 52a. By adopting such a display form, the surgeon 5061 can proceed with the surgery while observing the magnified surgical field image L(x, y), and the scopist 5062 can adjust the position of the endoscope 5001 while observing the surgical field image K(x, y). Note that the position where the surgical field image K(x, y) is superimposed is not limited to the example in FIG. 9B, and it may be superimposed on any of the upper left, upper right, and lower right positions of the magnified surgical field image L(x, y).

[0087] [Explanation of the effects of the second embodiment] Thus, according to the second embodiment, the display control unit 40 causes the single display device 50 to display the surgical field image K(x,y) and the magnified surgical field image L(x,y) adjacent to each other. Therefore, both the magnified surgical field image L(x,y) that the surgeon 5061 wants to view and the surgical field image K(x,y) that the scopist 5062 wants to view can be displayed.

[0088] Also, according to the second embodiment, the display control unit 40 causes the single display device 50 to display by superimposing the surgical field image K(x,y) on a part of the magnified surgical field image L(x,y). Therefore, both the magnified surgical field image L(x,y) that the surgeon 5061 wants to view and the surgical field image K(x,y) that the scopist 5062 wants to view can be displayed. In particular, the magnified surgical field image L(x,y) can be displayed as large as possible.

[0089] (Third Embodiment) FIG. 10 is a diagram showing an example of processing performed when the zoom frame reaches the edge of the surgical field image K(x,y) as the endoscope 5001 moves. The zoom frame is a frame indicating the display range of the magnified surgical field image L(x,y).

[0090] Here, since the endoscope 5001 has a tube shape with a circular cross-section, the surgical field image K(x,y) observed with the endoscope 5001 is accompanied by a circular kerfed area 130 around it. Since the kerfed area 130 is an area where light does not reach, it is observed as a black area as shown in FIG. 10A.

[0091] When the scopist 5062 moves the endoscope 5001, the zoom frame may reach the edge of the surgical field image K(x,y). In such a case, the medical observation system 10a takes one of three processing modes prepared in advance.

[0092] FIG. 10B is an example in which when the zoom frame 112a reaches the edge of the surgical field image K(x, y), an area without image information is painted black and displayed. That is, the zoom processing unit 26 generates an enlarged surgical field image L(x, y) in which a predetermined pixel value (for example, a pixel value 0 representing black) is stored in the area of the zoom frame 112a that exceeds the edge of the surgical field image K(x, y) and the area overlapping the cut-off area 130. Then, the display control unit 40 displays the generated enlarged surgical field image L(x, y).

[0093] Specifically, when the zoom frame 112a reaches the position in FIG. 10A, an enlarged surgical field image L(x, y) in which the pixel value 0 representing black is supplemented in the area where the image information inside the zoom frame 112a is missing is displayed. By adopting such a display form, the scopist 5062 can immediately recognize that the position of the endoscope 5001 has reached the edge of the surgical field image K(x, y) because the black area expands. Then, the scopist 5062 can regenerate the enlarged surgical field image L(x, y) without cut-off by adjusting the position of the endoscope 5001.

[0094] FIG. 10C is an example in which when the zoom frame 112a reaches the edge of the surgical field image K(x, y), the screen edge is continuously displayed as it is. That is, when the edge of the zoom frame 112a, that is, the edge of the enlarged surgical field image L(x, y), coincides with the edge of the surgical field image K(x, y), even when the endoscope 5001 moves further beyond the edge of the surgical field image K(x, y), the zoom processing unit 26 generates the enlarged surgical field image L(x, y) while maintaining the position of the zoom frame 112a. Then, the display control unit 40 displays the generated enlarged surgical field image L(x, y).

[0095] Specifically, when the zoom frame 112a reaches the position in FIG. 10A, the zoom frame 112a is moved to the position of the zoom frame 112b, and the image inside the moved zoom frame 112b is displayed as the magnified surgical field image L(x, y). That is, at this time, the left end of the zoom frame 112b coincides with the left end of the surgical field image K(x, y). By adopting such a display form, the display area of the magnified surgical field image L(x, y) can be held at the end of the surgical field image K(x, y) regardless of the movement of the endoscope 5001.

[0096] FIG. 10D is an example in which when the zoom frame 112a reaches the end of the surgical field image K(x, y), the zoom processing unit 26 stops generating the magnified surgical field image L(x, y). And at this time, the display control unit 40 displays the surgical field image K(x, y).

[0097] By adopting such a display form, since the display of the magnified surgical field image L(x, y) is cancelled, the scopist 5062 can immediately recognize that the imaging range of the endoscope 5001 has reached the end of the surgical field image K(x, y). Then, the scopist 5062 can reproduce the magnified surgical field image L(x, y) without blur by adjusting the position of the endoscope 5001.

[0098] In addition, when the zoom frame 112a reaches the end of the surgical field image K(x, y), which of the above-described processes is to be performed may be set in advance in the zoom processing unit 26.

[0099] [Explanation of the Effects of the Third Embodiment] As described above, according to the third embodiment, when the zoom frame 112a reaches the edge of the surgical field image K(x, y) or overlaps with the clipped area of the surgical field image K(x, y), the zoom processing unit 26 (enlarged image generation unit) generates an enlarged surgical field image L(x, y) in which a predetermined pixel value is stored in the area of the zoom frame 112a that exceeds the edge and the area that overlaps with the clipped area. Therefore, the scopist 5062 can immediately recognize that the zoom frame 112a has reached the edge of the surgical field image K(x, y). Then, the scopist 5062 can adjust the position of the endoscope 5001 so that no clipping occurs.

[0100] Also, according to the third embodiment, when the zoom frame 112a reaches the edge of the surgical field image K(x, y), the zoom processing unit 26 (enlarged image generation unit) generates an enlarged surgical field image L(x, y) in which the edge of the enlarged surgical field image L(x, y) coincides with the edge of the surgical field image K(x, y). Therefore, an enlarged surgical field image L(x, y) without clipping can be continuously displayed.

[0101] Also, according to the third embodiment, when the zoom frame 112a reaches the edge of the surgical field image K(x, y) or overlaps with the clipped area of the surgical field image K(x, y), the generation of the enlarged surgical field image L(x, y) is aborted. Therefore, the scopist 5062 can immediately recognize that the imaging range of the endoscope 5001 has reached the edge of the surgical field image K(x, y). Then, the scopist 5062 can adjust the position of the endoscope 5001 so that no clipping occurs.

[0102] (Fourth Embodiment) In the first embodiment, the medical observation system 10a has been described as having one imaging element 44a in the imaging device 42a. However, the configuration of the imaging device is not limited to this.

[0103] FIG. 11 is a diagram showing an example of a schematic configuration of a medical observation system 10b configured with an image sensor 44b in which an image plane phase difference sensor 46 is provided in an imaging device 42b. Note that FIG. 11 is drawn with a partial omission of FIG. 2, and unless otherwise specified, the omitted portions have the same configuration as FIG. 2.

[0104] The image plane phase difference sensor 46 has a configuration in which pixels for distance measurement are discretely arranged in the image sensor 44b. By using the medical observation system 10b configured as shown in FIG. 11, the map generation unit 15 can extract depth information (distance information) to the imaged object 100 from the image plane phase difference information output by the image plane phase difference sensor 46. Therefore, the SLAM technology can be effectively utilized. Note that the image plane phase difference sensor 46 can obtain depth information from only one captured image.

[0105] As described above, according to the fourth embodiment, since depth information can be obtained from one captured surgical field image K(x, y), even when the object is moving, the three-dimensional position of the object can be measured with high accuracy.

[0106] (Fifth Embodiment) FIG. 12 is a diagram showing an example of a schematic configuration of a medical observation system 10c configured with two image sensors 44c and 44d in an imaging device 42c. Note that the two image sensors 44c and 44d are arranged while maintaining a predetermined relative relationship, and different parts of the affected area are imaged so that a part overlaps. More specifically, the image sensors 44c and 44d respectively acquire image signals for the right eye and the left eye corresponding to stereoscopic vision. Note that FIG. 12 is drawn with a partial omission of FIG. 2, and unless otherwise specified, the omitted portions have the same configuration as FIG. 2.

[0107] In addition, in the medical observation system 10c, the camera control unit 12b includes a depth information generation unit 30 in addition to the configuration described in FIG. 2. The depth information generation unit 30 generates depth information by performing matching of two surgical field images respectively captured by the two image sensors 44c and 44d.

[0108] By using the medical observation system 10c configured as shown in FIG. 12, the map generation unit 15 can generate a three-dimensional map D(X, Y, Z) by utilizing the SLAM technology based on the depth information generated by the depth information generation unit 30 and the surgical field images captured by the imaging elements 44c and 44d, respectively. Further, since the two imaging elements 44c and 44d can perform imaging simultaneously, depth information can be obtained from the two images obtained in one imaging. Therefore, even when the object is moving, the three-dimensional position of the object can be measured with high accuracy.

[0109] As described above, according to the fifth embodiment, the imaging device 42c includes two imaging elements 44c and 44d that image different ranges with partial overlap, and the three-dimensional information generation unit 14 generates three-dimensional information of the surgical field based on two surgical field images K(x, y) captured by the two imaging elements 44c and 44d at the same time. Therefore, since depth information can be obtained from the two surgical field images K(x, y) obtained in one imaging, even when the surgical field is moving, the three-dimensional position of the surgical field can be measured with high accuracy.

[0110] (Sixth Embodiment) FIG. 13 is a diagram showing an example of a schematic configuration of a medical observation system 10d in which the imaging device 42c is composed of two imaging elements and the camera control unit 12c includes a tracking processing unit 34. Note that FIG. 13 is drawn by partially omitting FIG. 2, and unless otherwise specified, the omitted portions have the same configuration as FIG. 2.

[0111] The camera control unit 12c of the medical observation system 10d includes a depth information generation unit 30, a three-dimensional information generation unit 32, a tracking processing unit 34, and a zoom region calculation unit 36.

[0112] The three-dimensional information generation unit 32 is provided in place of the three-dimensional information generation unit 14 (FIG. 2), and generates three-dimensional information of the surgical field image K(x, y) based on the depth information generated by the depth information generation unit 30. The tracking processing unit 34 is provided in place of the three-dimensional map data storage unit 24 (FIG. 2), and calculates the difference in the position and orientation of the imaging device 42c by using, for example, the ICP (Iterative Closest Point) method, which is a method of overlapping two point clouds based on the three-dimensional information of the previous frame and the three-dimensional information of the current frame. The zoom area calculation unit 36 is provided in place of the attention area estimation unit 22 (FIG. 2), and calculates the coordinates on the screen of the attention area based on the difference value of the position and orientation of the imaging device 42c calculated by the tracking processing unit 34. Then, the zoom processing unit 26 (FIG. 2) performs zoom processing on the area calculated by the zoom area calculation unit 36 to generate an enlarged surgical field image L(x, y).

[0113] Thus, according to the sixth embodiment, the attention area in the surgical field image K(x, y) can be stably tracked (pursued) regardless of the movement of the imaging device 42c.

[0114] (Seventh Embodiment) FIG. 14 is a diagram showing an example of a schematic configuration of a medical observation system 10e in which the imaging device 42d is composed of an imaging element 44a and a depth sensor 48. Note that FIG. 14 is drawn with a partial omission of FIG. 2, and unless otherwise specified, the omitted parts have the same configuration as FIG. 2.

[0115] The depth sensor 48 is a so-called 3D sensor that measures the distance to the subject. The depth sensor 48 is a so-called ToF (Time of Flight) sensor that measures the distance to the subject by measuring the flight time of light by receiving reflected light such as infrared light irradiated toward the subject. Further, the depth sensor 48 is realized by a so-called pattern projection method (Structured Light) that measures the distance to the subject by imaging an image of projection light having a plurality of different geometric patterns irradiated on the subject.

[0116] The map generation unit 15 extracts depth information (distance information) to the imaged object 100 based on the surgical field image K(x, y) captured by the imaging element 44a and the distance output by the depth sensor 48. More specifically, the map generation unit 15 calculates which pixel in the surgical field image K(x, y) captured by the imaging element 44a corresponds to the point measured by the depth sensor 48. Then, the map generation unit 15 generates a three-dimensional map D(X, Y, Z) (three-dimensional information) of the surgical field. Therefore, the SLAM technology can be effectively utilized.

[0117] As described above, according to the seventh embodiment, the imaging device 42d includes one imaging element 44a and a depth sensor 48 (ranging device) that measures the distance to the object, and the three-dimensional information generation unit 14 generates a three-dimensional map D(X, Y, Z) (three-dimensional information) of the surgical field based on the image captured by the imaging element 44a and the distance measured by the depth sensor 48. Therefore, the distance to the surgical field can be easily and reliably measured.

[0118] (Eighth Embodiment) FIG. 15 is a diagram showing an example of a schematic configuration of a medical observation system 10f in which the imaging device 42d is composed of an imaging element 44a and a depth sensor 48, and the camera control unit 12d includes a tracking processing unit 34. Note that FIG. 15 is drawn by partially omitting FIG. 2, and unless otherwise specified, the omitted portions have the same configuration as FIG. 2.

[0119] The camera control unit 12d of the medical observation system 10f includes a three-dimensional information generation unit 32, a tracking processing unit 34, and a zoom region calculation unit 36.

[0120] The three-dimensional information generation unit 32 is provided in place of the three-dimensional information generation unit 14 (FIG. 2), and determines the movement state of the surgical field by matching two distance information (for example, a distance image in which pixel values corresponding to the distances to the subject are stored) measured by the depth sensor 48 from different positions. The tracking processing unit 34 is provided in place of the three-dimensional map data storage unit 24 (FIG. 2), and calculates the difference in the position and orientation of the imaging device 42c based on the movement state of the surgical field described above. The zoom area calculation unit 36 is provided in place of the attention area estimation unit 22 (FIG. 2), and calculates the coordinates on the screen of the attention area based on the difference value of the position and orientation of the imaging device 42c calculated by the tracking processing unit 34. Then, the zoom processing unit 26 (FIG. 2) performs zoom processing on the area calculated by the zoom area calculation unit 36 to generate an enlarged surgical field image L(x, y).

[0121] As described above, according to the eighth embodiment, it is possible to stably track (trail) the attention area in the surgical field image K(x, y) regardless of the movement of the imaging device 42d.

[0122] (Ninth Embodiment) FIG. 16 is a diagram showing an example in which a plurality of attention frames 110a and 110b are set in the surgical field image K(x, y).

[0123] As shown in FIG. 16, the attention area setting unit 20 (FIG. 2) may set a plurality of attention areas in the surgical field image K(x, y). For example, when it is necessary to pay attention to a plurality of affected parts, the attention area setting unit 20 sets attention frames 110a and 110b indicating each attention area based on the instruction of the scopist 5062. Then, the display control unit 40 causes the display device 50 to display two enlarged surgical field images L(x, y) in which the areas of the zoom frames corresponding to the respective attention frames 110a and 110b are enlarged.

[0124] As described above, according to the ninth embodiment, when there are a plurality of areas to be focused on in the surgical field, the attention area setting unit 20 sets a plurality of attention areas. Therefore, it is possible to display enlarged surgical field images L(x, y) in which a plurality of attention areas are enlarged.

[0125] (Embodiment 10) FIG. 17 is a diagram showing an example in which a region within a predetermined distance range is highlighted in the operative field image K(x, y).

[0126] When setting the region of interest, the region of interest setting unit 20 performs predetermined coloring and display on a region within a predetermined distance range in the operative field image K(x, y) as shown in FIG. 17. FIG. 17 shows an example of display in which different colorings are applied to a region R1 having a distance closer than d1 and a region R2 having a distance farther than d2, respectively. This is a process performed to limit the distance range to the region of interest between distance d1 and distance d2 for the purpose of facilitating the setting of the region of interest.

[0127] For example, as shown in FIG. 17, the values of distance d1 and distance d2 may be set by the region of interest setting unit 20 displaying a distance scale near the operative field image K(x, y) and the scopist 5062 operating an input device such as a mouse or a touch panel. Then, according to the set values of distance d1 and distance d2, the region of interest setting unit 20 performs real-time coloring display on regions R1 and R2 on the operative field image K(x, y). At this time, the operator points the input device at the position of the distance to be set on the distance scale to set distance d1 or distance d2. Then, while pointing the input device, the operator drags the input device in the far or near direction on the distance scale. By detecting this drag operation, the region of interest setting unit 20 displays, as shown in FIG. 17, the color to be colored in the dragged distance range on the distance scale. By adopting such a GUI (Graphical User Interface), it becomes easier for the operator to recognize the region corresponding to the distance range set by the operator in the operative field image K(x, y). Note that the method of displaying the set distance range on the distance scale is not limited to the method shown in FIG. 17, and other display forms may be adopted as long as the set distance range is clearly shown.

[0128] The display control unit 40 causes the display device 50 to display the surgical field image K(x, y) in which the regions R1 and R2 are colored. Then, the scopist 5062 sets the region of interest according to the above-described procedure (see FIG. 4) while viewing the surgical field image K(x, y) in which the regions R1 and R2 are colored.

[0129] As described above, according to the tenth embodiment, the region of interest setting unit 20 (setting unit) further includes a function of specifying the distance range in which the region of interest exists, and causes the region of interest to be set within the specified distance range. Therefore, the scopist 5062 can more easily set the region of interest.

[0130] (Eleventh Embodiment) FIG. 18 is a diagram showing an example of the display form of the regions of interest 110c to 110g set in the surgical field image K(x, y).

[0131] The display form of the region of interest is not limited to the rectangular frame shown in FIG. 4. FIG. 18A shows an example in which the region of interest 110c is displayed as a circular region. FIG. 18B shows an example in which the region of interest 110d is shown as a colored (highlighted) closed region. FIG. 18C shows an example in which the region of interest 110e is shown by a symbol. FIG. 18D shows an example in which the region of interest 110f is shown by a closed curve. FIG. 18E shows an example in which both the region of interest 110g and the region having the same distance as the position where the region of interest 110g is set are colored and displayed. In particular, according to the display form of FIG. 18E, the scopist 5062 can recognize that there is another region at a position equal to the distance of the region of interest. Therefore, the endoscope 5001 can be grasped more carefully so that the tracking of the region of interest is not lost when the endoscope 5001 accidentally turns in the direction of another region.

[0132] Note that the scopist 5062 may set in advance in the attention area setting unit 20 how to display the attention frame in any form. The setting method of the attention frames 110c to 110g may follow the method described in FIG. 4 or FIG. 5. In particular, as shown in FIGS. 18B, 18D, and 18E, when setting the attention frame as a closed area of an arbitrary shape, as described in FIG. 5, it is efficient to directly set the position and shape of the attention frame on the surgical field image K(x, y) displayed on the display device 50.

[0133] Thus, according to the eleventh embodiment, it is possible to display the attention frames 110c to 110g in a form that is easy for the operator to view in the set attention area.

[0134] (Twelfth Embodiment) FIG. 19 is a diagram showing an example of a method for setting the zoom frame 112.

[0135] At the start of use of the medical observation system 10a, the scopist 5062 may set the magnification when enlarging the surgical field image K(x, y). For example, the zoom processing unit 26 in FIG. 2 may cause the display control unit 40 to display a plurality of selectable zoom frames 112 (112c to 112f) superimposed on the surgical field image K(x, y) on the display device 50, and let the operator specify one of the zoom frames therefrom. FIG. 19 shows an example in which the zoom frame 112e indicating a magnification of 1.5 times is specified. The selection of the zoom frame 112 may be performed, for example, by operating an input device such as a switch provided at the hand of the endoscope 5001.

[0136] In addition, the zoom processing unit 26 may generate an enlarged surgical field image L(x, y) at a magnification corresponding to the distance to the target area. That is, the zoom processing unit 26 calculates, for example, the distance to the target area based on the three-dimensional map D(X, Y, Z) generated by the three-dimensional information generation unit 14 and stored in the three-dimensional map data storage unit 24. Then, according to the calculated distance to the target area, the magnification when generating the enlarged surgical field image L(x, y) is determined. Further, an AF (Auto Focus) function may be implemented in the imaging device 42a, and the imaging device 42a may calculate the distance to the target area by focusing on the position of the target area estimated by the target area estimation unit 22. For example, when the distance to the target area is far, the magnification can be set high, and when the distance to the target area is near, the magnification can be set low.

[0137] As described above, according to the twelfth embodiment, the scopist 5062 can easily set the magnification by selecting one zoom frame from among the plurality of zoom frames 112c to 112f displayed on the display device 50.

[0138] Also, according to the twelfth embodiment, the zoom processing unit 26 (enlarged image generation unit) generates an enlarged surgical field image L(x, y) at a magnification corresponding to the distance to the target area. Therefore, even when the endoscope 5001 moves in the front-rear direction with respect to the affected part, the affected part can be continuously observed at a constant size.

[0139] (The thirteenth embodiment) FIG. 20 is a diagram showing an example of a method of displaying the zoom frame 112 when the medical observation system 10a is in operation.

[0140] The zoom processing unit 26 may cause the display control unit 40 to display the zoom frame 112 superimposed on the surgical field image K(x, y) displayed on the display device 50.

[0141] FIG. 20A shows an example in which the zoom frame 112 is displayed in the surgical field image K(x, y) displayed superimposed on a part of the enlarged surgical field image L(x, y).

[0142] FIG. 20B shows an example in which a zoom frame 112 is displayed in the surgical field image K(x, y) displayed adjacent to the magnified surgical field image L(x, y).

[0143] FIG. 20C shows an example in which a zoom frame 112 is displayed in the surgical field image K(x, y) displayed on a display device 50a different from the display device 50b on which the magnified surgical field image L(x, y) is displayed.

[0144] Thus, according to the 13th embodiment, the scopist 5062 can easily confirm the position of the zoom frame 112. Therefore, since the scopist 5062 can predict in advance the reaching of the zoom frame 112 to the screen edge, it is possible to prevent the occurrence of glare caused by the display range of the magnified surgical field image L(x, y) exceeding the edge of the surgical field image K(x, y).

[0145] (14th Embodiment) In the medical observation system 10a described in the first embodiment, in order to make the surgical field image K(x, y) and the magnified surgical field image L(x, y) displayed on the display device 50 easier to view, shake correction processing and adjustment of the exposure amount may be performed on these images. The shake correction processing is performed by, for example, the zoom processing unit 26 in FIG. 2, and the adjustment of the exposure amount is performed by the development processing unit 18 in FIG. 2.

[0146] More specifically, the zoom processing unit 26 calculates the amount of movement and the direction of movement of an object or the like reflected in the image among a plurality of captured images for the surgical field image K(x, y) and the magnified surgical field image L(x, y). Then, according to the calculated amount of movement and the direction of movement, the captured image is electronically shifted to generate an image with shake correction. Since the magnified surgical field image L(x, y) is an image obtained by observing a narrower area than the surgical field image K(x, y), the blur of the image caused by shake becomes larger. Therefore, it is desirable that the zoom processing unit 26 perform shake correction with a higher correction effect on the magnified surgical field image L(x, y) than the shake correction on the surgical field image K(x, y).

[0147] In addition, the development processing unit 18 may separately adjust the exposure amounts of the surgical field image K(x, y) and the magnified surgical field image L(x, y) by separately setting digital gain and gamma curves for each of them.

[0148] As described above, according to the 14th embodiment, the zoom processing unit 26 (magnified image generation unit) performs shake correction on the surgical field image K(x, y) and the magnified surgical field image L(x, y). Therefore, even when shake occurs in the surgical field image K(x, y) captured by the endoscope 5001, it is possible to obtain an easy-to-see surgical field image K(x, y) and magnified surgical field image L(x, y) in which the shake is corrected.

[0149] (15th Embodiment) FIG. 21 is a diagram showing an example of a schematic configuration of a microscope surgery system 5300 to which the technology according to the present disclosure can be applied. Referring to FIG. 21, the microscope surgery system 5300 includes a microscope device 5301, a control device 5317, and a display device 50. In the following description of the microscope surgery system 5300, the "user" means any medical staff who uses the microscope surgery system 5300, such as a surgeon and an assistant.

[0150] The microscope device 5301 includes a microscope unit 5303 for magnifyingly observing an observation target (the surgical site of a patient), an arm unit 5309 that supports the microscope unit 5303 at its tip, and a base unit 5315 that supports the base end of the arm unit 5309.

[0151] The microscope unit 5303 includes a substantially cylindrical tubular portion 5305 and an imaging unit (not shown) provided inside the tubular portion 5305. The microscope unit 5303 is an electronically imaging type microscope unit (so-called video type microscope unit) that electronically captures an imaging image by the imaging unit. The imaging unit is an example of the imaging device in the present disclosure.

[0152] A cover glass for protecting the internal imaging unit is provided on the opening surface at the lower end of the cylindrical portion 5305. Light from the observation target (hereinafter also referred to as observation light) passes through the cover glass and enters the imaging unit inside the cylindrical portion 5305. Note that a light source composed of, for example, an LED (Light Emitting Diode) or the like may be provided inside the cylindrical portion 5305, and during imaging, light may be irradiated from the light source to the observation target through the cover glass.

[0153] The imaging unit is composed of an optical system that condenses the observation light and an image sensor that receives the observation light condensed by the optical system. The optical system is configured by combining a plurality of lenses including a zoom lens and a focus lens, and its optical characteristics are adjusted so as to form an image of the observation light on the light receiving surface of the image sensor. The image sensor generates a signal corresponding to the observation light, that is, an image signal corresponding to the observation image, by receiving the observation light and performing photoelectric conversion. As the image sensor, for example, one capable of color photography having a Bayer array is used. The image sensor may be various known image sensors such as a CMOS image sensor or a CCD image sensor. The image signal generated by the image sensor is transmitted to the control device 5317 as RAW data. Here, the transmission of this image signal may preferably be performed by optical communication. In the operating room, since the surgeon performs the operation while observing the state of the affected part from the imaging image, for a safer and more reliable operation, it is required that the moving image of the surgical field be displayed in real time as much as possible. By transmitting the image signal by optical communication, it becomes possible to display the imaging image with low latency.

[0154] Note that the imaging unit may have a drive mechanism that moves the zoom lens and the focus lens of its optical system along the optical axis. By appropriately moving the zoom lens and the focus lens by the drive mechanism, the magnification of the imaging image and the focal length at the time of imaging can be adjusted. In addition, the imaging unit may be equipped with various functions that can generally be provided in an electronically imaging type microscope unit, such as an AE (Auto Exposure) function and an AF function.

[0155] Further, the imaging unit may be configured as a so-called single-plate imaging unit having one imaging element, or may be configured as a so-called multi-plate imaging unit having a plurality of imaging elements. When the imaging unit is configured as a multi-plate type, for example, image signals corresponding to RGB respectively may be generated by each imaging element, and a color image may be obtained by synthesizing them. Alternatively, the imaging unit may be configured to have a pair of imaging elements for respectively acquiring right-eye and left-eye image signals corresponding to stereoscopic vision (3D display). By performing 3D display, the surgeon can more accurately grasp the depth of the biological tissue in the surgical field. When the imaging unit is configured as a multi-plate type, a plurality of optical systems may be provided corresponding to each imaging element.

[0156] The arm unit 5309 is configured by a plurality of links (the first link 5313a to the sixth link 5313f) being rotatably connected to each other by a plurality of joint parts (the first joint part 5311a to the sixth joint part 5311f).

[0157] The first joint part 5311a has a substantially cylindrical shape, and at its tip (lower end), the upper end of the cylindrical part 5305 of the microscope unit 5303 is rotatably supported around a rotation axis (the first axis O1) parallel to the central axis of the cylindrical part 5305. Here, the first joint part 5311a can be configured such that the first axis O1 coincides with the optical axis of the imaging unit of the microscope unit 5303. Thereby, by rotating the microscope unit 5303 around the first axis O1, it becomes possible to change the visual field so as to rotate the captured image.

[0158] The first link 5313a fixedly supports the first joint part 5311a at its tip. Specifically, the first link 5313a is a rod-shaped member having a substantially L shape, and one side on the tip side extends in a direction orthogonal to the first axis O1, and the end of the one side abuts against the upper end part of the outer periphery of the first joint part 5311a, and is connected to the first joint part 5311a. The second joint part 5311b is connected to the end of the other side of the substantially L-shaped base end side of the first link 5313a.

[0159] The second joint portion 5311b has a substantially cylindrical shape, and at its tip, the base end of the first link 5313a is rotatably supported about a rotation axis (second axis O2) orthogonal to the first axis O1. At the base end of the second joint portion 5311b, the tip of the second link 5313b is fixedly connected.

[0160] The second link 5313b is a rod-shaped member having a substantially L-shaped configuration, and one side on the tip side thereof extends in a direction orthogonal to the second axis O2, and the end of the one side is fixedly connected to the base end of the second joint portion 5311b. The third joint portion 5311c is connected to the other side on the substantially L-shaped base end side of the second link 5313b.

[0161] The third joint portion 5311c has a substantially cylindrical shape, and at its tip, the base end of the second link 5313b is rotatably supported about a rotation axis (third axis O3) orthogonal to the first axis O1 and the second axis O2. At the base end of the third joint portion 5311c, the tip of the third link 5313c is fixedly connected. By rotating the tip-side configuration including the microscope unit 5303 about the second axis O2 and the third axis O3, the microscope unit 5303 can be moved so as to change the position of the microscope unit 5303 in the horizontal plane. That is, by controlling the rotation about the second axis O2 and the third axis O3, the field of view of the captured image can be moved within the plane.

[0162] The third link 5313c is configured such that its tip side has a substantially cylindrical shape, and at the tip of the cylindrical shape, the base end of the third joint portion 5311c is fixedly connected so that both have substantially the same central axis. The base end side of the third link 5313c has a prismatic shape, and the fourth joint portion 5311d is connected to the end thereof.

[0163] The fourth joint portion 5311d has a substantially cylindrical shape, and at its tip, the base end of the third link 5313c is rotatably supported about a rotation axis (fourth axis O4) orthogonal to the third axis O3. At the base end of the fourth joint portion 5311d, the tip of the fourth link 5313d is fixedly connected.

[0164] The fourth link 5313d is a rod-shaped member extending in a substantially straight line. While extending so as to be orthogonal to the fourth axis O4, the end portion of its tip contacts the substantially cylindrical side surface of the fourth joint portion 5311d, and it is fixedly connected to the fourth joint portion 5311d. A fifth joint portion 5311e is connected to the proximal end of the fourth link 5313d.

[0165] The fifth joint portion 5311e has a substantially cylindrical shape, and on its tip side, supports the proximal end of the fourth link 5313d so as to be rotatable about a rotation axis (fifth axis O5) parallel to the fourth axis O4. The tip of the fifth link 5313e is fixedly connected to the proximal end of the fifth joint portion 5311e. The fourth axis O4 and the fifth axis O5 are rotation axes capable of moving the microscope unit 5303 in the vertical direction. By rotating the tip-side configuration including the microscope unit 5303 about the fourth axis O4 and the fifth axis O5, the height of the microscope unit 5303, that is, the distance between the microscope unit 5303 and the observation object can be adjusted.

[0166] The fifth link 5313e is composed of a combination of a first member having a substantially L-shaped with one side extending in the vertical direction and the other side extending in the horizontal direction, and a rod-shaped second member extending vertically downward from the portion of the first member extending in the horizontal direction. The proximal end of the fifth joint portion 5311e is fixedly connected near the upper end of the portion of the first member of the fifth link 5313e extending in the vertical direction. A sixth joint portion 5311f is connected to the proximal end (lower end) of the second member of the fifth link 5313e.

[0167] The sixth joint portion 5311f has a substantially cylindrical shape, and on its tip side, supports the proximal end of the fifth link 5313e so as to be rotatable about a rotation axis (sixth axis O6) parallel to the vertical direction. The tip of the sixth link 5313f is fixedly connected to the proximal end of the sixth joint portion 5311f.

[0168] The sixth link 5313f is a rod-shaped member extending in the vertical direction, and its proximal end is fixedly connected to the upper surface of the base portion 5315.

[0169] The rotatable ranges of the first joint portion 5311a to the sixth joint portion 5311f are appropriately set so that the microscope unit 5303 can make a desired movement. Thereby, in the arm unit 5309 having the configuration described above, with respect to the movement of the microscope unit 5303, a total of six degrees of freedom of movement, namely three translational degrees of freedom and three rotational degrees of freedom, can be realized. By configuring the arm unit 5309 so that six degrees of freedom are realized with respect to the movement of the microscope unit 5303 in this way, it becomes possible to freely control the position and orientation of the microscope unit 5303 within the movable range of the arm unit 5309. Therefore, it becomes possible to observe the surgical field from any angle, and the surgery can be carried out more smoothly.

[0170] Note that the configuration of the illustrated arm unit 5309 is merely an example, and the number and shape (length) of the links constituting the arm unit 5309, as well as the number, arrangement position, and direction of the rotation axis of the joint portions, etc., may be appropriately designed so that the desired degrees of freedom can be realized. For example, as described above, in order to freely move the microscope unit 5303, it is preferable that the arm unit 5309 is configured to have six degrees of freedom, but the arm unit 5309 may be configured to have a larger degree of freedom (i.e., redundant degrees of freedom). When there are redundant degrees of freedom, in the arm unit 5309, it becomes possible to change the posture of the arm unit 5309 with the position and orientation of the microscope unit 5303 fixed. Therefore, more convenient control for the surgeon can be realized, such as controlling the posture of the arm unit 5309 so that the arm unit 5309 does not interfere with the field of view of the surgeon looking at the display device 50.

[0171] Here, the first joint part 5311a to the sixth joint part 5311f may be provided with an actuator equipped with a drive mechanism such as a motor and an encoder for detecting the rotation angle of each joint part. Then, by appropriately controlling the driving of each actuator provided in the first joint part 5311a to the sixth joint part 5311f by the control device 5317, the posture of the arm part 5309, that is, the position and posture of the microscope part 5303 can be controlled. Specifically, the control device 5317 can grasp the current posture of the arm part 5309 and the current position and posture of the microscope part 5303 based on the information about the rotation angle of each joint part detected by the encoder. The control device 5317 uses the grasped information to calculate control values (for example, rotation angle or generated torque, etc.) for each joint part so that the microscope part 5303 realizes a desired movement, and drives the drive mechanism of each joint part according to the control values. At this time, the control method of the arm part 5309 by the control device 5317 is not limited, and various known control methods such as force control or position control may be applied.

[0172] For example, when the operator appropriately performs an operation input through an input device (not shown), the driving of the arm part 5309 is appropriately controlled by the control device 5317 according to the operation input, and the position and posture of the microscope part 5303 may be controlled. By this control, after moving the microscope part 5303 from an arbitrary position to an arbitrary position, it can be fixedly supported at the moved position. As the input device, considering the convenience of the operator, for example, a foot switch or the like that can be operated even when the operator has a surgical instrument in hand is preferably applied. Also, based on gesture detection or gaze detection using a wearable device or a camera provided in the operating room, the operation input may be performed non-contact. Thereby, even a user belonging to the clean area can operate the equipment belonging to the unclean area with higher freedom. Alternatively, the arm part 5309 may be operated in a so-called master-slave method. In this case, the arm part 5309 can be remotely operated by the user through an input device installed at a location away from the operating room.

[0173] Also, when force control is applied, the actuators of the first joint part 5311a to the sixth joint part 5311f are driven so as to receive an external force from the user and smoothly move the arm part 5309 following the external force, and so-called power assist control may be performed. Thereby, when the user tries to grip the microscope part 5303 and directly move its position, the microscope part 5303 can be moved with a relatively light force. Therefore, it becomes possible to move the microscope part 5303 more intuitively and with a simpler operation, and the convenience of the user can be improved.

[0174] Also, the driving of the arm part 5309 may be controlled so as to perform a pivot operation. Here, the pivot operation is an operation of moving the microscope part 5303 so that the optical axis of the microscope part 5303 always faces a predetermined point in space (hereinafter referred to as the pivot point). According to the pivot operation, since it becomes possible to observe the same observation position from various directions, it becomes possible to observe the affected part in more detail. When the microscope part 5303 is configured such that its focal length cannot be adjusted, it is preferable that the pivot operation be performed with the distance between the microscope part 5303 and the pivot point fixed. In this case, the distance between the microscope part 5303 and the pivot point may be adjusted to the fixed focal length of the microscope part 5303. Thereby, the microscope part 5303 moves on a hemispherical surface (schematically illustrated in FIG. 21) having a radius corresponding to the focal length centered on the pivot point, and a clear captured image can be obtained even when the observation direction is changed. On the other hand, when the microscope part 5303 is configured such that its focal length can be adjusted, the pivot operation may be performed with the distance between the microscope part 5303 and the pivot point variable. In this case, for example, the control device 5317 may calculate the distance between the microscope part 5303 and the pivot point based on information about the rotation angle of each joint part detected by the encoder, and automatically adjust the focal length of the microscope part 5303 based on the calculation result. Alternatively, if the microscope part 5303 is provided with an AF function, every time the distance between the microscope part 5303 and the pivot point changes due to the pivot operation, the AF function may automatically adjust the focal length.

[0175] The control device 5317 comprehensively controls the operation of the microscope surgery system 5300 by controlling the operations of the microscope device 5301 and the display device 50. For example, the control device 5317 controls the drive of the arm unit 5309 by operating the actuators of the first joint unit 5311a to the sixth joint unit 5311f according to a predetermined control method. Also, for example, the control device 5317 changes the operation mode of the arm unit 5309 by controlling the operation of the brakes of the first joint unit 5311a to the sixth joint unit 5311f. Further, the control device 5317 has the functions of the camera control unit 12a described in the first embodiment. Then, the control device 5317 generates an enlarged surgical field image L(x, y) obtained by enlarging a region of interest from the surgical field image K(x, y) captured by the imaging unit of the microscope unit 5303, and displays it on the display device 50. Note that the control device 5317 may perform various known signal processes, such as development processing (demosaicing processing), image quality improvement processing (band enhancement processing, super-resolution processing, NR (Noise reduction) processing, and / or hand shake correction processing, etc.), on the surgical field image K(x, y) acquired by the imaging unit of the microscope unit 5303 of the microscope device 5301.

[0176] The communication between the control device 5317 and the microscope unit 5303, and the communication between the control device 5317 and the first joint unit 5311a to the sixth joint unit 5311f may be wired communication or wireless communication. In the case of wired communication, communication by an electrical signal or optical communication may be performed. In this case, the transmission cable used for wired communication can be configured as an electrical signal cable, an optical fiber, or a composite cable thereof according to the communication method. On the other hand, in the case of wireless communication, since it is not necessary to lay a transmission cable in the operating room, a situation where the movement of medical staff in the operating room is hindered by the transmission cable can be resolved.

[0177] The control device 5317 can be a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a microcomputer or a control board in which a processor and a storage element such as a memory are mounted together. By operating the processor of the control device 5317 according to a predetermined program, the various functions described above can be realized. In the illustrated example, the control device 5317 is provided as a device separate from the microscope device 5301. However, the control device 5317 may be installed inside the base portion 5315 of the microscope device 5301 and configured integrally with the microscope device 5301. Alternatively, the control device 5317 may be composed of a plurality of devices. For example, a microcomputer or a control board or the like may be disposed in each of the microscope unit 5303 and the first joint portion 5311a to the sixth joint portion 5311f of the arm unit 5309, and the same functions as those of the control device 5317 may be realized by connecting them to be communicable with each other.

[0178] The display device 50 is provided in the operating room and displays an image corresponding to the image data generated by the control device 5317 under the control of the control device 5317. That is, at least the magnified surgical field image L(x, y) among the surgical field image K(x, y) and the magnified surgical field image L(x, y) captured by the microscope unit 5303 is displayed on the display device 50. Note that the display device 50 may display various types of information related to the surgery, such as, for example, the patient's body information and information about the surgical procedure, instead of or together with the surgical field image K(x, y). In this case, the display of the display device 50 may be appropriately switched by an operation by the user. Alternatively, a plurality of display devices 50 may be provided, and the surgical field image K(x, y), the magnified surgical field image L(x, y), and various types of information related to the surgery may be respectively displayed on each of the plurality of display devices 50. Note that as the display device 50, various known display devices such as a liquid crystal display device or an EL display device may be applied.

[0179] FIG. 22 is a diagram showing a surgical procedure using the microsurgical system 5300 shown in FIG. 21. In FIG. 22, an operator 5061 is schematically shown performing a surgery on a patient 5071 on a patient bed 5069 using the microsurgical system 5300. In FIG. 22, for simplicity, illustration of the control device 5317 among the components of the microsurgical system 5300 is omitted, and the microscope device 5301 including the microscope unit 5303 (FIG. 21) is shown in a simplified manner.

[0180] As shown in FIG. 22, during the surgery, at least the magnified surgical field image L(x, y) among the surgical field image K(x, y) and the magnified surgical field image L(x, y) captured by the microscope device 5301 using the microsurgical system 5300 is enlarged and displayed on a display device 50 installed on the wall surface of the operating room. The display device 50 is installed at a position facing the operator 5061, and the operator 5061 observes the state of the surgical site through the video projected on the display device 50 and performs various treatments on the surgical site, such as resection of the affected part.

[0181] FIG. 23 is a diagram showing an example of a control state in which the zoom frame 112 included in the microsurgical system 5300 is held at the center of the screen.

[0182] The control device 5317 constantly monitors the position of the zoom frame 112 in the surgical field image K(x, y). Specifically, the control device 5317 constantly monitors whether the zoom frame 112 protrudes from a control determination frame 114 set at approximately the center of the surgical field image K(x, y).

[0183] When it is detected that the zoom frame 112 has protruded from the control determination frame 114, the control device 5317 controls the angles of the first joint part 5311a to the sixth joint part 5311f so that the zoom frame 112 remains inside the control determination frame 114, thereby controlling the position and posture of the microscope unit 5303.

[0184] In the example of FIG. 23, the zoom frame 112 protrudes beyond the right side of the control determination frame 114. In this case, the control device 5317 controls the position and orientation of the microscope unit 5303 so that the zoom frame 112 stays inside the control determination frame 114. That is, in the example of FIG. 23, the control device 5317 moves the position and orientation of the microscope unit 5303 to the right to keep the zoom frame 112 inside the control determination frame 114. Similarly, when the zoom frame 112 protrudes in a direction other than the right side of the control determination frame 114, the control device 5317 controls the position and orientation of the microscope unit 5303 so that the zoom frame 112 stays inside the control determination frame 114.

[0185] Thus, according to the 15th embodiment, the control device 5317 controls the position and orientation of the microscope unit 5303 so that the zoom frame 112 stays inside the control determination frame 114. Therefore, when the operator 5061 performs the operation alone, since the operator 5061 does not need to hold the microscope unit 5303, the operator can concentrate on the operation. Note that the control of the position and orientation of the microscope unit 5303 described in the 15th embodiment can also be applied to, for example, the medical observation system 10a described in the 1st embodiment. That is, in the medical observation system 10a, the position and orientation of the endoscope 5001 can be controlled so that the zoom frame 112 always stays at a predetermined position of the display device 50.

[0186] Also, according to the 15th embodiment, the imaging unit is mounted on the microscope unit 5303. Therefore, when performing an operation or the like using the microscope, the operator 5061 can magnify and stably observe the affected part.

[0187] The above has described an example of a microsurgery system 5300 to which the technology according to the present disclosure can be applied. Here, although the microsurgery system 5300 has been described as an example, the system to which the technology according to the present disclosure can be applied is not limited to such an example. For example, the microscope device 5301 can also function as a support arm device that supports another observation device or another surgical instrument instead of the microscope unit 5303 at its tip. As the other observation device, for example, an endoscope can be applied. As the other surgical instrument, forceps, tweezers, a pneumoperitoneum tube for pneumoperitoneum, or an energy treatment instrument that incises tissue or seals blood vessels by cauterization can be applied. By supporting these observation devices and surgical instruments with the support arm device, it is possible to fix the position more stably and reduce the burden on medical staff compared to the case where medical staff hold them manually. The technology according to the present disclosure may be applied to a support arm device that supports configurations other than such a microscope unit.

[0188] Note that the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0189] Also, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure.

[0190] Note that the present disclosure can also take the following configuration. (1) An imaging device that images the surgical field to obtain a surgical field image, A three-dimensional information generation unit that generates three-dimensional information of the surgical field from the surgical field image imaged by the imaging device, A setting unit that sets at least one region of interest based on at least one surgical field image imaged by the imaging device at a predetermined timing, An estimation unit that estimates the existence position of the region of interest from among surgical field images imaged at a timing different from the predetermined timing based on the three-dimensional information and the position of the region of interest set by the setting unit, An enlarged image generation unit that generates an enlarged surgical field image obtained by enlarging a presumed region of interest at a predetermined magnification, A display control unit that outputs at least the enlarged surgical field image, A medical observation system comprising the same. (2) The display control unit displays the surgical field image and the enlarged surgical field image. The medical observation system according to (1) above. (3) The display control unit displays the surgical field image and the enlarged surgical field image adjacent to each other on one display device. The medical observation system according to (2) above. (4) The display control unit displays the surgical field image superimposed on a part of the enlarged surgical field image on one display device. The medical observation system according to (2) above. (5) The display control unit displays the surgical field image and the enlarged surgical field image on two display devices respectively. The medical observation system according to (2) above. (6) The setting unit designates the specific position as the region of interest on the condition that a setting signal for instructing the setting of the region of interest is generated while the specific position of the surgical field image displayed by the display control unit is made to coincide with a predetermined position. The medical observation system according to any one of (1) to (5) above. (7) The setting unit sets the region of interest at the position indicated by the input device in the surgical field image displayed by the display control unit. The medical observation system according to any one of (1) to (5) above. (8) When the region of interest reaches the edge of the surgical field image or overlaps with the clipped area of the surgical field image, the enlarged image generation unit generates an enlarged surgical field image in which a predetermined pixel value is stored in the region of the region of interest that exceeds the edge and the region that overlaps with the clipped area. The medical observation system according to any one of (1) to (7) above. (9) When the region of interest reaches the edge of the surgical field image, the enlarged image generation unit generates an enlarged surgical field image in which the edge of the enlarged surgical field image coincides with the edge of the surgical field image. The medical observation system according to any one of (1) to (7) above. (10) When the region of interest reaches the edge of the surgical field image or overlaps with the keratosed region of the surgical field image, the enlarged image generation unit aborts the generation of the enlarged surgical field image. The medical observation system according to any one of (1) to (7) above. (11) The imaging device includes one imaging element, and the three-dimensional information generation unit generates three-dimensional information of the surgical field based on at least two surgical field images captured by the imaging device at different times. The medical observation system according to any one of (1) to (10) above. (12) The imaging device includes two imaging elements that image different ranges with partial overlap, and the three-dimensional information generation unit generates three-dimensional information of the surgical field based on two surgical field images captured by the imaging elements at the same time. The medical observation system according to any one of (1) to (10) above. (13) The imaging device includes one imaging element and a distance measuring device that measures the distance to the object, and the three-dimensional information generation unit generates three-dimensional information of the surgical field based on the image captured by the imaging element and the distance measured by the distance measuring device. The medical observation system according to any one of (1) to (10) above. (14) The setting unit further has a function of designating a distance range in which the region of interest exists, and sets the region of interest within the designated distance range. The medical observation system according to any one of (1) to (13) above. (15) The enlarged image generation unit generates the enlarged surgical field image at a magnification corresponding to the distance to the region of interest. The medical observation system according to any one of (1) to (14) above. (16) The enlarged image generation unit performs shake correction on the surgical field image and the enlarged surgical field image. The medical observation system according to any one of (1) to (15) above. (17) The imaging device is mounted on an endoscope. The medical observation system according to any one of (1) to (16) above. (18) The imaging device is mounted on a microscope. The medical observation system according to any one of (1) to (16) above. (19) A three-dimensional information generation unit that generates three-dimensional information of the surgical field from a surgical field image obtained by imaging the surgical field, A setting unit that sets at least one region of interest based on at least one of the surgical field images captured at a predetermined timing, An estimation unit that estimates the position where the region of interest exists from among the surgical field images captured at a timing different from the predetermined timing based on the three-dimensional information and the position of the region of interest set by the setting unit, An enlarged image generation unit that generates an enlarged surgical field image obtained by enlarging the estimated region of interest at a predetermined magnification, A display control unit that outputs at least the enlarged surgical field image, A medical observation device comprising: (20) A step of generating three-dimensional information of the surgical field from a surgical field image obtained by imaging the surgical field, A step of setting at least one region of interest based on at least one of the surgical field images captured at a predetermined timing, A step of estimating the position where the region of interest exists from among the surgical field images captured at a timing different from the predetermined timing based on the three-dimensional information and the position of the region of interest, Generating a magnified operative field image by magnifying a presumed region of interest at a predetermined magnification factor; Outputting at least the magnified operative field image; A medical observation method comprising the above.

Explanation of Signs

[0191] 10a, 10b, 10c, 10d, 10e, 10f Medical observation system 12a, 12b, 12c Camera control unit (medical observation device) 14 Three-dimensional information generation unit 15 Map generation unit 16 Self-position estimation unit 18 Development processing unit 20 Region of interest setting unit (setting unit) 22 Region of interest estimation unit (estimation unit) 24 Three-dimensional map data storage unit 26 Zoom processing unit (magnified image generation unit) 40 Display control unit 42a, 42b, 42c, 42d Imaging device 44a, 44b, 44c, 44d Image sensor 46 Image plane phase difference sensor 48 Depth sensor (distance measurement device) 50, 50a, 50b Display device 52a First display area 52b Second display area 110 Region of interest frame 112, 112a, 112b Zoom frame 5001 Endoscope 5061 Surgeon 5062 Scopist 5063 Assistant 5300 Microscopic surgery system 5303 Microscope unit 5317 Control device D(X,Y,Z) Three-dimensional map (three-dimensional information) K(x,y), K(x,y,t) Operative field image L(x,y) Magnified operative field image

Claims

1. a three-dimensional information generating unit that generates three-dimensional information of an observation object from an endoscopic image captured by an endoscope; a setting unit that sets at least one region of interest based on at least one of the endoscopic images captured by the endoscope at a predetermined timing; an estimation unit that estimates an existing position of the region of interest from the endoscopic image captured at a timing different from the predetermined timing, based on the three-dimensional information and the position of the region of interest set by the setting unit; an image generating unit that generates a display image in which a frame of interest indicating the position of the region of interest is superimposed on the endoscopic image captured at a timing different from the predetermined timing based on the estimated position; a display control unit that outputs at least the generated display image; Equipped with the estimation unit estimates an existing position of the region of interest based on a position and an orientation of the endoscope at the predetermined timing, which are estimated based on the three-dimensional information and the endoscopic image, and a position and an orientation of the endoscope at a timing different from the predetermined timing. Medical observation system.

2. the three-dimensional information generating unit estimates a position and an attitude of the endoscope, the attitude and the position of which change during a predetermined time interval, based on a plurality of feature points in at least two of the endoscopic images captured by the endoscope at different times separated by a predetermined time interval, and generates three-dimensional information of the observation target. The medical observation system according to claim 1 .

3. the setting unit, under a condition that a setting signal instructing to set the region of interest is generated in a state in which a specific position of the endoscopic image captured at the predetermined timing and displayed by the display control unit is matched with a predetermined position, specifies the specific position as the region of interest.

3. The medical observation system according to claim 1 or 2.

4. the setting unit sets the region of interest at a position designated by an input device in the endoscopic image captured at the predetermined timing and displayed by the display control unit.

3. The medical observation system according to claim 1 or 2.

5. 5. The medical observation system according to claim 1, wherein the setting unit sets a plurality of the regions of interest.

6. 6. The medical observation system according to claim 1, wherein the image generating section generates the display image in which the region of interest surrounded by the frame of interest is colored in a predetermined color.

7. 7. The medical observation system according to claim 6, wherein the image generating unit generates the display image in which an area whose distance from the endoscope is equal to a distance from the endoscope to the area of ​​interest is colored in the predetermined color.

8. 8. The medical observation system according to claim 1, wherein the image generating section generates an enlarged image by enlarging the region of interest at a predetermined magnification.

9. The display control unit displays the display image and the enlarged image. The medical observation system according to claim 8.

10. The endoscope has a distance measuring device that measures a distance to an object, the three-dimensional information generating unit generates the three-dimensional information based on the endoscopic image and the distance measured by the distance measuring device. The medical observation system according to any one of claims 1 to 9.

11. the image generation unit performs camera shake correction on the display image and the enlarged image.

10. The medical observation system according to claim 8 or 9.

12. a three-dimensional information generating unit that generates three-dimensional information of an observation object from an endoscopic image captured by an endoscope; a setting unit that sets at least one region of interest based on at least one of the endoscopic images captured by the endoscope at a predetermined timing; an estimation unit that estimates an existing position of the region of interest from the endoscopic image captured at a timing different from the predetermined timing, based on the three-dimensional information and the position of the region of interest set by the setting unit; an image generating unit that generates a display image in which a frame of interest indicating the position of the region of interest is superimposed on the endoscopic image captured at a timing different from the predetermined timing based on the estimated position; a display control unit that outputs at least the generated display image; Equipped with the estimation unit estimates an existing position of the region of interest based on a position and an orientation of the endoscope at the predetermined timing, which are estimated based on the three-dimensional information and the endoscopic image, and a position and an orientation of the endoscope at a timing different from the predetermined timing. Medical observation equipment.

13. Medical observation equipment, generating three-dimensional information of an observation target from an endoscopic image captured by an endoscope; setting at least one region of interest based on at least one endoscopic image captured by the endoscope at a predetermined timing; estimating a location of the region of interest from the endoscopic image captured at a timing different from the predetermined timing, based on the three-dimensional information and the set position of the region of interest; generating a display image in which a frame of interest indicating the position of the region of interest is superimposed on the endoscopic image captured at a timing different from the predetermined timing based on the estimated position; outputting at least the generated image for display; Equipped with In the step of estimating a position of the region of interest, estimating an existing position of the region of interest based on a position and orientation of the endoscope at the predetermined timing and a position and orientation of the endoscope at a timing different from the predetermined timing, the position and orientation being estimated based on the three-dimensional information and the endoscopic image; Medical observation methods.

Images