Medical imaging system, medical imaging apparatus, and control method

Abstract

The present disclosure relates to a medical imaging system, a medical imaging device, and a control method capable of appropriately outputting an image according to surgery. This medical imaging system is provided with a surgery mode setting unit for setting a surgery mode; and a selection processing unit which performs selection of switching a display image on the basis of the surgery mode from among a Near image having a focus on a near point, a Mid image having a focus on a middle point, and a Far image having a focus on a far point which are captured from one imaging lens by three imaging elements different in optical path length, and an EDOF image having a depth of field extended by combining the images. This technology is applicable to, for example, a medical imaging system capable of capturing an EDOF image.

Description

[Technical Field]

[0001] The present disclosure relates to a medical imaging system, a medical imaging apparatus, and a control method, and more particularly to a medical imaging system, a medical imaging apparatus, and a control method that are capable of outputting an image appropriately according to surgery. [Background technology]

[0002] Conventionally, in the medical field, when performing procedures such as surgery while observing an affected area as a region of interest in an image captured through a lens, the image becomes out of focus in the near point or far point areas that are outside the depth of field. Therefore, whenever the region of interest changes to the near point or far point, it is necessary to adjust the focus so that the region of interest is in focus, and there is a concern that images with a shallow depth of field will reduce work efficiency. Therefore, there is a demand for a medical imaging system that can capture images with a deep depth of field.

[0003] For example, Patent Document 1 discloses a medical observation device capable of acquiring an extended depth of field (EDOF) image with an extended depth of field. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-158764 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the depth of field and focus conditions required vary depending on the type of surgery, so there is a demand for outputting images with depth of field and focus adjusted appropriately according to the surgery.

[0006] The present disclosure has been made in view of such circumstances, and makes it possible to output images appropriately according to the surgery. [Means for solving the problem]

[0007] A medical imaging system and a medical imaging apparatus according to one aspect of the present disclosure include a surgery mode setting unit that sets a surgery mode, and at least one imaging lens having different optical path lengths from one imaging lens. 3 sheets The image is captured by the image sensor , a Near image focused on a near point, a Mid image focused on a midpoint, and a Far image focused on a far point; a selection processing unit that selects an image to be displayed from among EDOF (Extended Depth of Field) images in which the depth of field is extended by combining these images, based on the surgery mode; and a color-coded image generating unit that uses one of the near image, the mid image, and the far image as a base image, calculates contrast for each pixel of the near image, the mid image, and the far image, and generates a color-coded image by superimposing a color corresponding to the image for which the highest contrast is calculated on the base image; and Equipped with When a user selects a desired color portion of the color-coded image, the image corresponding to the selected color among the Near image, the Mid image, and the Far image is used as the displayed image.

[0008] A control method according to one aspect of the present disclosure includes a step of controlling a medical imaging system by setting a surgical mode and controlling at least one imaging lens having a different optical path length. 3 sheets The image is captured by the image sensor , a Near image focused on a near point, a Mid image focused on a midpoint, and a Far image focused on a far point; and selecting an EDOF (Extended Depth of Field) image, which has an extended depth of field by combining these images, to switch the displayed image based on the surgery mode. one of the near image, the mid image, and the far image is used as a base image, contrast is calculated for each pixel of the near image, the mid image, and the far image, and a color-coded image is generated by superimposing a color corresponding to the image for which the highest contrast is calculated on the base image; Including When a user selects a desired color portion of the color-coded image, the color-coded image is used to select an image from the Near image, the Mid image, and the Far image that corresponds to the selected color as the displayed image.

[0009] In one aspect of the present disclosure, a surgical mode is set, and at least one imaging lens having different optical path lengths from one imaging lens is used. 3 sheets The image is captured by the image sensor , a Near image focused on a near point, a Mid image focused on a midpoint, and a Far image focused on a far point; The depth of field is expanded by combining these images, and the displayed image is selected based on the surgical mode. The system uses one of the near image, mid image, and far image as a base image, calculates the contrast for each pixel of the near image, mid image, and far image, and generates a color-coded image by superimposing the color corresponding to the image with the highest contrast on the base image. When a user selects a desired color portion of the color-coded image, the image corresponding to the selected color from the near image, mid image, and far image is used as the display image. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of an embodiment of a medical imaging system to which the present technology is applied. [Figure 2]FIG. 2 is a diagram illustrating the configuration of an endoscope and a device unit. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of an imaging module. [Figure 4] 1A and 1B are diagrams for explaining a Mid image, a Near image, a Far image, an EDOF image, and a color-coded image. [Figure 5] FIG. 10 is a diagram showing the relationship between focus positions in a cataract surgery mode and a vitreous surgery mode. [Figure 6] FIG. 10 is a diagram illustrating an example of the configuration of an image selection function of a CCU. [Figure 7] 10 is a flowchart illustrating an image selection process. [Figure 8] 10 is a flowchart illustrating a display image output process. [Figure 9] 1 is a block diagram illustrating an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <Configuration example of medical imaging system> FIG. 1 is a diagram showing an example of the configuration of an embodiment in which a medical imaging system to which the present technology is applied is applied to endoscopic surgery.

[0013] The medical imaging system 11 shown in FIG. 1 is configured to include an endoscope 12, an energy treatment tool 13, a display device 14, and a device unit 15.

[0014] For example, in a surgery using the medical imaging system 11, the endoscope 12 and the energy treatment tool 13 are inserted into the patient's body, and the forceps 16 is also inserted into the patient's body. In the medical imaging system 11, an image of an affected area such as a tumor taken by the endoscope 12 is displayed in real time on the display device 14, and a doctor can treat the affected area using the energy treatment tool 13 and the forceps 16 while viewing the image.

[0015] As shown in Fig. 2, the endoscope 12 is configured by attaching a cylindrical lens barrel 22 incorporating an optical system such as an objective lens to a camera head 21 incorporating an imaging module (see Fig. 3) having a plurality of imaging elements. For example, the lens barrel 22 is a scope formed into a cylindrical shape using a hard or soft material, and a light guide extending inside the lens barrel 22 guides light to the tip, enabling the light to be irradiated into the patient's body cavity. The camera head 21 is configured so that an optical element such as a birefringent mask (BM) can be inserted between the lens barrel 22 and the camera head, and images of the inside of the patient's body cavity can be captured via the optical system of the lens barrel 22.

[0016] The energy treatment device 13 is a medical instrument used in endoscopic surgery, for example, to remove an affected area or seal a blood vessel by using heat generated by a high-frequency current.

[0017] The display device 14 can display the image captured by the endoscope 12 as is, or can display the image that has been subjected to image processing in the device unit 15.

[0018] The equipment unit 15 is configured by incorporating various devices required for performing endoscopic surgery using the medical imaging system 11. For example, as shown in Fig. 2, the equipment unit 15 can be configured to include a light source device 31, a CCU (Camera Control Unit) 32, a recording device 33, and an output device 34.

[0019] The light source device 31 supplies light to the endoscope 12 via an optical fiber or the like, with which the affected area is irradiated when the endoscope 12 captures an image.

[0020] The CCU 32 controls imaging by the endoscope 12 and performs image processing on the images captured by the endoscope 12. The CCU 32 also has, for example, a focus control function for appropriately controlling the focus when capturing an image by the endoscope 12 in accordance with the surgical mode, and an image selection function for appropriately outputting the images captured by the endoscope 12 in accordance with the surgical mode.

[0021] The recording device 33 records on a recording medium the image output from the CCU 32. The output device 34 prints and outputs the image output from the CCU 32, or outputs it via a communication network.

[0022] <Configuration example of imaging module> FIG. 3 is a diagram showing an example of the configuration of an imaging module incorporated into the camera head 21 of the endoscope 12. As shown in FIG.

[0023] 3, the imaging module 41 is configured to include a branching optical system 51 and three imaging elements 52-1 to 52-3. In addition, an imaging lens 42 is arranged on the optical axis of light incident on the imaging module 41. The imaging lens 42 is configured with one or more lenses, and focuses light toward the imaging elements 52-1 to 52-3 so that imaging is performed using light entering the barrel portion 22 of the endoscope 12, and causes the light to enter the branching optical system 51.

[0024] The branching optical system 51 branches the light incident via the imaging lens 42 toward each of the imaging elements 52-1 to 52-3. The branching optical system 51 is composed of a first prism 61, a second prism 62, a third prism 63, a first dichroic mirror 64, and a second dichroic mirror 65.

[0025] The first prism 61, the second prism 62, and the third prism 63 constitute a prism block that is bonded together so that no air gaps are formed between the first prism 61 and the second prism 62 and between the second prism 62 and the third prism 63. By employing a prism block with a so-called gapless structure in this way, the branching optical system 51 can avoid the inclusion of process dust and the seepage of sealing material. Therefore, the branching optical system 51 can eliminate the reflection of foreign matter and suppress degradation of image quality even in a lens system with a relatively large F-number, such as that of the endoscope 12.

[0026] The first dichroic mirror 64 is an optical thin film made of a dielectric multilayer film formed on the exit surface of the first prism 61 facing the second prism 62, and splits the light into light amounts such that, for example, the average reflectance:average transmittance=1:2.

[0027] The second dichroic mirror 65 is an optical thin film made of a dielectric multilayer film formed on the exit surface of the second prism 62 facing the third prism 63, and splits the light into light amounts such that, for example, the average reflectance:average transmittance=1:1.

[0028] The image pickup elements 52-1 to 52-3 are, for example, CMOS image sensors having RGB filters in a Bayer array. The image pickup element 52-1 is disposed at a position where the distance (optical path length) from the principal point of the imaging lens 42 is a reference intermediate distance. The image pickup element 52-2 is disposed at a position farther away from the branching optical system 51 by the shift amount ΔZ so that the distance from the principal point of the imaging lens 42 is longer than the reference distance. The image pickup element 52-3 is disposed at a position closer to the branching optical system 51 by the shift amount ΔZ so that the distance from the principal point of the imaging lens 42 is shorter than the reference distance.

[0029] As a result, when the focal length of the imaging lens 42 is adjusted so that the imaging element 52-1 captures an image focused on the region of interest, the imaging element 52-2 captures an image focused on a point closer to the region of interest. Similarly, in this case, the imaging element 52-3 captures an image focused on a point farther from the region of interest. Therefore, hereinafter, the image captured by the imaging element 52-1 will be referred to as the "Mid image," the image captured by the imaging element 52-2 will be referred to as the "Near image," and the image captured by the imaging element 52-3 will be referred to as the "Far image."

[0030] Therefore, the imaging module 41 is configured to be able to output a near image, a mid image, and a far image to the CCU 32.

[0031] The medical imaging system 11 can switch between Near images, Mid images, and Far images and output them to the display device 14, as well as switch between EDOF images and color-coded images that have been image-processed in the CCU 32 and output them to the display device 14.

[0032] FIG. 4 shows images of a Near image, a Mid image, a Far image, an EDOF image, and a color-coded image that are switched and displayed in the medical imaging system 11.

[0033] For example, a Near image is captured so that the focus is on the near point, and the image becomes increasingly blurred as it moves toward the far point. A Mid image is captured so that the focus is on the middle point, and the image becomes blurred on both the near point and far point sides. A Far image is captured so that the focus is on the far point, and the image becomes increasingly blurred as it moves toward the near point.

[0034] An EDOF image is an image that has been processed to expand the depth of field so that the image is in focus from the near point to the far point by calculating the contrast for each pixel in the near image, mid image, and far image, and then selecting and combining the pixels with the highest contrast.

[0035] The color-coded image is an image in which the contrast is calculated for each pixel in the Near, Mid, and Far images, and the image is color-coded with the color corresponding to the image with the highest contrast, and is used to select areas. For example, the color-coded image is color-coded so that the pixel with the highest contrast in the Near image is red (solid line in Figure 4), the pixel with the highest contrast in the Mid image is green (dash line in Figure 4), and the pixel with the highest contrast in the Far image is blue (dashed line in Figure 4).

[0036] In the medical imaging system 11, when a user inputs a surgical mode using, for example, a user interface displayed on the display device 14, the input surgical mode is set in the CCU 32. Then, the CCU 32 can appropriately select and output a near image, a mid image, a far image, an EDOF image, and a color-coded image according to the set surgical mode.

[0037] For example, the user can select which image to display by looking at the near image, mid image, far image, EDOF image, and color-coded image arranged as shown in Fig. 4 as options for display images displayed on the display device 14. Note that the display image may be determined by default depending on the surgical mode.

[0038] Here, an example will be described in which a cataract surgery mode, a vitreous surgery mode, and a laparoscopic mode are adopted as selectable surgery modes in the medical imaging system 11. Alternatively, a corneal surgery mode, a vitreous surgery mode, or a fundus surgery mode may be adopted.

[0039] The relationship between focus positions in the cataract surgery mode and the vitreous surgery mode will be described with reference to FIG.

[0040] FIG. 5A shows an example of the focus positions of the near image, the mid image, and the far image in the cataract surgery mode.

[0041] As shown in the figure, in the cataract surgery mode, the near image is captured so that the surface of the cornea is the focus position, the mid image is captured so that the surface of the lens is the focus position, and the far image is captured so that the back surface of the lens is the focus position.

[0042] FIG. 5B shows an example of the focus positions of the Mid image, the Near image, and the Far image in the vitreous surgery mode.

[0043] As shown in the figure, in the vitreous surgery mode, the near image is captured so that the instrument insertion portion on the outside of the eye is the focus position, the mid image is captured so that the center of the vitreous body is the focus position, and the far image is captured so that the bottom of the eye is the focus position. Here, when performing vitreous surgery, surgery is performed on the vitreous body only without damaging the lens, so the inside of the vitreous body is viewed using the mid and far images, and the instrument insertion portion on the outside of the eye is viewed using the near image.

[0044] When the medical imaging system 11 is used for ophthalmic surgery, it is preferable to set the difference in optical path length between the near point and the midpoint small and between the midpoint and the far point large. That is, the difference in optical path length between the image sensor 52-1 and the image sensor 52-2 (ΔZ) is set smaller than the difference in optical path length between the image sensor 52-1 and the image sensor 52-3 (-ΔZ).

[0045] This makes it possible to easily perform an operation of switching between the Near image and the Mid image near the cornea during ophthalmic surgery. At this time, it is preferable that the Near image is focused near the corneal surface (instrument insertion part) and the Mid image is focused on the bottom surface of the cornea (the deeper side of the cornea in the eyeball). Specifically, since the thickness of the lens is 4 to 5 mm, the difference in optical path lengths between the imaging element 52-1 and the imaging element 52-2 is preferably set within 5 mm. Since the size of the eyeball from the cornea to the fundus is 22 to 23 mm, the difference in optical path lengths between the imaging element 52-1 and the imaging element 52-3 is preferably set within 25 mm. Note that the optical path lengths are small in the order of the imaging element 52-3, the imaging element 52-1, and the imaging element 52-2.

[0046] <Image selection function of CCU> FIG. 6 is a block diagram for explaining the image selection function of the CCU 32.

[0047] As shown in FIG. 6, the CCU 32 includes a surgical mode setting unit 71, a display image selection processing unit 72, an imaging signal acquisition unit 73, an EDOF image output unit 74, and a color-separated image output unit 75.

[0048] When the surgical mode is input by the user using, for example, an input unit (not shown) to the surgical mode setting unit 71, the surgical mode setting unit 71 sets the surgical mode for the display image selection processing unit 72.

[0049] The display image selection processing unit 72 presents the user with display image options to be displayed on the display device 14 based on the surgery mode set by the surgery mode setting unit 71, and when the user selects a desired image to be displayed on the display device 14, determines the selected image as the display image to be displayed on the display device 14. Furthermore, the display image selection processing unit 72 sets an EDOF-use image from the near image, mid image, and far image to be used by the EDOF image output unit 74 to generate an EDOF image based on the surgery mode. Furthermore, the display image selection processing unit 72 sets a base image from the near image, mid image, and far image to be used as a basis when the color-coded image output unit 75 generates a color-coded image based on the surgery mode.

[0050] The imaging signal acquisition unit 73 acquires imaging signals of the near image, mid image, and far image output from the imaging module 41. Then, the imaging signal acquisition unit 73 outputs the near image when the near image is determined as the display image, outputs the mid image when the mid image is determined as the display image, and outputs the far image when the far image is determined as the display image. Furthermore, when an EDOF-use image is determined as the display image, the imaging signal acquisition unit 73 supplies the EDOF-use image set based on the surgery mode to the EDOF image output unit 74. Furthermore, when a color-coded image is determined as the display image, the imaging signal acquisition unit 73 supplies the near image, mid image, and far image to the color-coded image output unit 75 and instructs the color-coded image output unit 75 on the image set as the base image.

[0051] The EDOF image output unit 74 uses the EDOF-using images supplied from the imaging signal acquisition unit 73 to generate and output an EDOF image. For example, if a near image, a mid image, and a far image are set as EDOF-using images, the EDOF image output unit 74 calculates the contrast for each pixel in the near image, the mid image, and the far image. The EDOF image output unit 74 then selects and combines the pixels with the highest contrast to generate an EDOF image with an extended depth of field so that the image is in focus from the near point to the far point.

[0052] The color-coded image output unit 75 generates and outputs a color-coded image using the near image, mid image, and far image supplied from the imaging signal acquisition unit 73, based on the base image set by the display image selection processing unit 72. For example, when the mid image is specified as the base image, the color-coded image output unit 75 converts the base image to black and white, finds the contrast for each pixel of the near image, mid image, and far image, and superimposes the color corresponding to the image with the highest contrast, thereby generating a color-coded image.

[0053] In the CCU 32 configured as above, when the user inputs a surgery mode, the surgery mode setting unit 71 sets the surgery mode in the display image selection processing unit 72. Then, the user can set on the screen "which area to focus on" and "whether to display EDOF images," etc.

[0054] For example, when the cataract surgery mode is set, the display image selection processing unit 72 sets the color-coded image, EDOF image, near image, mid image, and far image as display image options. The color-coded image is an option that allows the user to select which image is best from the multiple images displayed, that is, which image is best in focus.

[0055] When the cataract surgery mode is set, the display image options may be displayed as "Selection," "Surgical Instrument," "Cornea," "Anterior Capsule," and "Posterior Capsule," and the names of the options may be changed depending on the surgery mode. However, even if the name of the option is changed depending on the surgery mode (for example, an image named "Cornea" in the cataract surgery mode is changed to "Vitreous" in the vitreous surgery mode), the optical path length remains the same.

[0056] In the medical imaging system 11, it is preferable that the user can change the selection using a switch (for example, a foot pedal) or voice input, etc. This allows the focus to be changed more quickly than by adjusting the focus during surgery, thereby improving the efficiency of surgery.

[0057] Furthermore, when the cataract surgery mode is set, the color-coded image generated by the color-coded image output unit 75 is used to select an area, and the Mid image is set as the base image from which the color-coded image is generated. That is, the image displayed when selecting an area is the Mid image, and the user specifies which area of ​​the Mid image to focus on, and the color-coded image output unit 75 selects the image that is estimated to be most in focus in the specified area (the image with the largest edge).

[0058] Furthermore, when the cataract surgery mode is set, the EDOF image output unit 74 sets the near image, mid image, and far image as the EDOF images used to generate the EDOF image. This is because cataract surgery is often performed by observing the instrument insertion site, the vicinity of the cornea, and the vicinity of the vitreous body. That is, the near image is used to observe the instrument insertion site, the mid image is used to observe the vicinity of the cornea, which is the surgical target, and the far image is used to monitor whether the vitreous body is being affected. Note that the images used for the EDOF images for each surgery mode may be changeable by the user.

[0059] When the display image selection processing unit 72 presents the display image options to the user, the image currently being displayed on the display device 14 or an image recommended based on a preset may be highlighted with a red frame or the like. When presenting the display image options to the user, the display image options may be displayed on a display separate from the display device 14 as a user interface for selecting a display image.

[0060] For example, the user can look at the near image, mid image, far image, EDOF image, and color-coded image (image for region selection) arranged side by side as shown in Figure 4 and decide which image to display. Note that the displayed image may be determined by default depending on the surgical mode. In the color-coded image (image for region selection), each region is color-coded to correspond to the near image, mid image, and far image, and the user can select the image in which the region in focus they want to view by, for example, touching the corresponding colored part in the color-coded image.

[0061] Furthermore, the CCU 32 may use image recognition to recognize a region such as the cornea, and focus the Mid or Near image on the cornea. When an EDOF image is selected, some areas on the screen may be EDOF-enabled and others may not. For example, the Mid image may always be output in the center of the screen, and EDOF of the Mid and Far images may be performed around the center. The extent of the center of the screen may be determined by prior settings or image recognition. In this way, the user may set the EDOF settings in advance.

[0062] Even when the vitreous surgery mode is set, the expressions of the options presented to the user and the images to be synthesized by EDOF can be changed. Also, the image set as the base image can be changed as desired by the user.

[0063] Furthermore, it is preferable that the surgical mode be created from the perspective of the surgical target, such as the cornea, iris, conjunctiva, lens, ciliary body, vitreous body, optic disc, macula, etc., or the type of surgery (surgical procedure) (for example, tumor resection, pterygium resection, ophthalmic ptosis correction surgery, strabismus surgery, corneal transplantation, corneal refractive surgery, cataract surgery, glaucoma surgery, retinal detachment surgery, vitreous surgery, IOL (Intraocular Lens) insertion surgery, etc.). The user then determines the surgical mode by selecting the surgical target or surgical procedure.

[0064] The image selection function of the CCU 32 is configured as described above, and can appropriately switch images and output them to the display device 14 when the required depth of field conditions differ depending on the surgical mode, or when it is necessary to switch between multiple imaging signals with different focus positions.

[0065] For example, in anterior segment surgery, a depth of field that allows the anterior segment to be in focus is sufficient, but there are also situations where the port or fundus is to be viewed temporarily, which conventionally requires time-consuming and laborious focus adjustment each time. Also, in anterior segment surgery, the cornea, lens, etc. are transparent, so it is necessary to switch between image signals that are in focus for each, and conventionally, if the depth of field is shallow, focus adjustment is required each time.

[0066] In contrast, the medical imaging system 11 simultaneously acquires near images, mid images, and far images with different focus positions, and can switch the display image output to the display device 14 depending on the surgical mode. Also, the medical imaging system 11 can generate more optimal EDOF images by switching the EDOF-using image depending on the surgical mode.

[0067] As a result, in an anterior segment surgery, for example, the medical imaging system 11 simultaneously acquires images focused on the anterior segment, the port, and the fundus, and allows the user to quickly view the images without adjusting the focus by switching the display image output to the display device 14. Therefore, the medical imaging system 11 can realize the required depth of field extension and switching of imaging signals with different focus positions without the need for focus adjustment depending on the surgery mode.

[0068] In addition, conventional endoscopes have been unable to deliver light to the far end of an object, resulting in poor visibility even when the depth of field is large. In contrast, the medical imaging system 11 performs EDOF using three image sensors 52-1 to 52-3, thereby improving visibility compared to EDOF using a phase mask, and achieving better color and depth of field extension.

[0069] <Example of image selection processing> The image selection process will be described with reference to the flowcharts shown in FIGS.

[0070] Fig. 7 is a flowchart illustrating the image selection process executed by the CCU 32. Note that Fig. 7 describes a processing example in which one of the cataract surgery mode, vitreous surgery mode, and laparoscopic surgery mode is set as the surgery mode, but similar processing is performed when the various surgery modes described above are set.

[0071] For example, when a surgery mode is input by the user, the process starts. In step S11, the surgery mode setting unit 71 sets the surgery mode input by the user to the display image selection processing unit 72.

[0072] In step S12, the display image selection processing unit 72 determines whether the surgery mode set in step S11 is the cataract surgery mode, the vitreous surgery mode, or the laparoscopic surgery mode.

[0073] If the display image selection processing unit 72 determines in step S12 that the cataract surgery mode has been set, the process proceeds to step S13.

[0074] In step S13, the display image selection processor 72 sets the color-coded image, EDOF image, Near image, Mid image, and Far image as display image options for the cataract surgery mode. In step S14, the display image selection processor 72 sets the Mid image as the base image that serves as the base for generating the color-coded image, and in step S15, sets the Near image, Mid image, and Far image as images to be used for EDOF.

[0075] On the other hand, if the display image selection processing unit 72 determines in step S12 that the vitreous surgery mode has been set, the process proceeds to step S16.

[0076] In step S16, the display image selection processor 72 sets the color-coded image, EDOF image, Near image, Mid image, and Far image as display image options for the vitreous surgery mode. In step S17, the display image selection processor 72 sets the Mid image as the base image that serves as the base for generating the color-coded image, and in step S18, sets the Mid image and the Far image as images to be used for EDOF.

[0077] On the other hand, if the display image selection processing unit 72 determines in step S12 that the laparoscopic surgery mode has been set, the process proceeds to step S19.

[0078] In step S19, the display image selection processing unit 72 sets the color-coded image, EDOF image, and Mid image as display image options for the laparoscopic surgery mode. In step S20, the display image selection processing unit 72 sets the Mid image as the base image that serves as the base for generating the color-coded image, and in step S18, sets the Near image, Mid image, and Far image as images to be used for EDOF.

[0079] After processing step S15, step S18, or step S21, the process proceeds to step S22, where the display image selection processing unit 72 displays the display image options set in step S13, step S16, or step S19 and presents them to the user.

[0080] In step S23, when the user selects the desired image to be displayed on the display device 14 from the display image options presented in step S22, the display image selection processing unit 72 determines the selected image as the display image to be displayed on the display device 14.

[0081] In step S24, a display image output process (see FIG. 8, which will be described later) is executed to output the display image determined in step S23.

[0082] In step S25, the image output in the display image output process in step S24 is displayed on the display device 14, and then the process ends.

[0083] FIG. 8 is a flowchart illustrating the display image output process executed in step S24 of FIG.

[0084] In step S31, the imaging signal acquisition unit 73 acquires imaging signals of the near image, the mid image, and the far image output from the imaging module 41.

[0085] In step S32, the imaging signal acquisition unit 73 resets the parameter i (i=0) that identifies the pixel to be processed.

[0086] In step S33, the imaging signal acquisition unit 73 determines whether it has been decided in step S23 of Fig. 7 to output the Near image as the display image, and if it has been decided to output the Near image, the processing proceeds to step S34. In step S34, the imaging signal acquisition unit 73 outputs pixel i of the Near image.

[0087] On the other hand, if in step S33 the imaging signal acquisition unit 73 determines that it has not been decided to output a Near image as the display image, that is, if an image other than a Near image has been decided as the display image, the processing proceeds to step S35.

[0088] In step S35, the imaging signal acquisition unit 73 determines whether it has been determined in step S23 of Fig. 7 that the Mid image should be output as the display image, and if it has been determined that it has been determined that the Mid image should be output, the process proceeds to step S36. In step S36, the imaging signal acquisition unit 73 outputs pixel i of the Mid image.

[0089] On the other hand, if in step S35 the imaging signal acquisition unit 73 determines that it has not been decided to output the Mid image as the display image, that is, if an image other than the Mid image has been decided as the display image, processing proceeds to step S37.

[0090] In step S37, the imaging signal acquisition unit 73 determines whether it has been determined in step S23 of Fig. 7 that the far image should be output as the display image, and if it has been determined that the far image should be output, the process proceeds to step S38. In step S38, the imaging signal acquisition unit 73 outputs pixel i of the far image.

[0091] On the other hand, if in step S37 the imaging signal acquisition unit 73 determines that it has not been decided to output a far image as the display image, that is, if an image other than a far image has been decided as the display image, the processing proceeds to step S39.

[0092] In step S39, the imaging signal acquisition unit 73 determines whether it has been decided in step S23 of Figure 7 to output an EDOF image as a display image, and if it is determined that it has been decided to output an EDOF image, the processing proceeds to step S40.

[0093] In step S40, the imaging signal acquisition unit 73 supplies the EDOF-using image set in step S15, step S18, or step S21 of Fig. 7 to the EDOF image output unit 74. The EDOF image output unit 74 calculates the contrast of pixel i for the EDOF-using image supplied from the imaging signal acquisition unit 73.

[0094] In step S41, the EDOF image output unit 74 outputs the pixel i of the image for which the highest contrast has been found out of the contrasts of the pixels i of the EDOF-using images found in step S40.

[0095] On the other hand, if the imaging signal acquisition unit 73 determines in step S39 that it has not been determined in step S23 of Fig. 7 that an EDOF image is to be output as the display image, the process proceeds to step S42. In this case, taking into consideration the determinations in steps S33, S35, and S37, it has been determined in step S23 of Fig. 7 that a color-coded image is to be output as the display image.

[0096] In step S 42 , the color-coded image output unit 75 obtains the contrast for each pixel i of the near image, mid image, and far image supplied from the imaging signal acquisition unit 73 .

[0097] In step S43, the color-coded image output unit 75 associates the pixel i with an identification number that identifies the image with the highest contrast among the contrasts of pixel i in the near image, mid image, and far image obtained in step S42.

[0098] In step S44, the color-coded image output unit 75 converts pixel i of the base image into black and white.

[0099] In step S45, the color-coded image output unit 75 outputs the color according to the identification number associated with pixel i in step S43, superimposed on pixel i that was converted to black and white in step S44.

[0100] In step S46, the selection map generating unit 77 increments (i++) the parameter i that identifies the pixel to be processed.

[0101] In step S47, the imaging signal acquisition unit 73 determines whether or not output has been performed for all pixels. For example, if the parameter i matches the number of pixels, the imaging signal acquisition unit 73 can determine that output has been performed for all pixels.

[0102] If the imaging signal acquisition section 73 determines in step S47 that output has not been performed for all pixels, the process returns to step S33, and the same processes are repeated thereafter.

[0103] On the other hand, if the imaging signal acquisition section 73 determines in step S47 that output has been performed for all pixels, the process ends.

[0104] As described above, the medical imaging system 11 can appropriately switch and output images depending on the surgery mode.

[0105] Although the present embodiment has been described with reference to an example configuration using three image sensors 52-1 to 52-3, the present technology can also be applied to a configuration using at least two image sensors, for example, 52-1 and 52-2. In such a configuration, for example, three types of images, a Near image, a Mid image, and an EDOF image generated from the Near image and the Mid image, can be selectively output according to the surgical mode.

[0106] <Example of computer configuration> Next, the above-described series of processes (control method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0107] FIG. 9 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0108] The program can be recorded in advance on the hard disk 105 or ROM 103 as a recording medium built into the computer.

[0109] Alternatively, the program can be stored (recorded) on a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0110] The program can be installed into the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0111] The computer includes a CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0112] When a user inputs a command by operating input unit 107 via input / output interface 110, CPU 102 executes a program stored in ROM (Read Only Memory) 103 in accordance with the command. Alternatively, CPU 102 loads a program stored on hard disk 105 into RAM (Random Access Memory) 104 and executes it.

[0113] As a result, CPU 102 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. CPU 102 then outputs the processing results from output unit 106 via input / output interface 110, transmits them from communication unit 108, or records them on hard disk 105, as necessary.

[0114] The input unit 107 is made up of a keyboard, a mouse, a microphone, etc. The output unit 106 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0115] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

[0116] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0117] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0118] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0119] Furthermore, for example, this technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0120] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0121] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0122] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0123] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0124] <Configuration combination example> The present technology can also be configured as follows. (1) a surgery mode setting unit that sets a surgery mode; a selection processing unit that selects, based on the surgery mode, to switch the displayed image from among two or more types of images captured by at least two image capturing elements with different optical path lengths from one imaging lens, and an EDOF (Extended Depth of Field) image in which the depth of field is extended by combining these images; and A medical imaging system comprising: (2) Images are taken from one imaging lens using three imaging elements with different optical path lengths, and a Near image focused on the near point, a Mid image focused on the midpoint, and a Far image focused on the far point are used. The medical imaging system according to (1) above. (3) The selection processing unit selects to switch the display image from among the near image, the mid image, the far image, the EDOF image obtained by combining the near image and the mid image, the EDOF image obtained by combining the mid image and the far image, and the EDOF image obtained by combining the near image, the mid image, and the far image, based on the surgery mode. The medical imaging system according to (2) above. (4) Regarding the optical path lengths from the imaging lenses of a first imaging element that captures the Mid image, a second imaging element that captures the Near image, and a third imaging element that captures the Far image, the difference in optical path length between the first imaging element and the second imaging element is smaller than the difference in optical path length between the first imaging element and the third imaging element. A medical imaging system according to (2) or (3) above. (5) An EDOF (Extended Depth of Field) image generating unit that calculates the contrast for each pixel of the near image, the mid image, and the far image, selects the pixel with the highest contrast, and generates a synthesized EDOF image. The medical imaging system according to any one of (2) to (4) above, further comprising: (6) a color-coded image generating unit that uses one of the near image, the mid image, and the far image as a base image, calculates contrast for each pixel of the near image, the mid image, and the far image, and generates a color-coded image by superimposing a color corresponding to the image for which the highest contrast is calculated on the base image; The medical imaging system according to any one of (2) to (5) above, further comprising: (7) The surgery modes include a cataract surgery mode, a vitreous surgery mode, and a laparoscopic mode. A medical imaging system according to any one of (1) to (6) above. (8) a surgery mode setting unit that sets a surgery mode; a selection processing unit that selects, based on the surgery mode, to switch the displayed image from among two or more types of images captured by at least two image capturing elements with different optical path lengths from one imaging lens, and an EDOF (Extended Depth of Field) image in which the depth of field is extended by combining these images; and A medical imaging device comprising: (9) Medical imaging systems Setting a surgery mode; Selecting and switching the displayed image based on the surgery mode from two or more types of images captured by at least two imaging elements with different optical path lengths from one imaging lens, and an EDOF (Extended Depth of Field) image in which the depth of field is extended by combining these images. A control method comprising:

[0125] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained. [Explanation of symbols]

[0126] 11 medical imaging system, 12 endoscope, 13 energy treatment tool, 14 display device, 15 device unit, 16 forceps, 21 camera head, 22 lens barrel section, 31 light source device, 32 CCU, 33 recording device, 34 output device, 41 imaging module, 42 imaging lens, 51 branching optical system, 52-1 to 52-3 imaging element, 61 first prism, 62 second prism, 63 third prism, 64 first dichroic mirror, 65 second dichroic mirror, 71 surgical mode setting section, 72 region of interest setting section, 73 imaging signal acquisition section, 74 EDOF image output section, 75 color-coded image output section

Claims

1. a surgery mode setting unit that sets a surgery mode; a selection processing unit that selects, based on the surgery mode, to switch the displayed image from among a Near image focused on a near point, a Mid image focused on a middle point, a Far image focused on a far point, and an EDOF (Extended Depth of Field) image in which the depth of field is extended by combining these images, all of which are captured by at least three image capturing elements with different optical path lengths from one imaging lens; a color-coded image generating unit that uses one of the near image, the mid image, and the far image as a base image, calculates contrast for each pixel of the near image, the mid image, and the far image, and generates a color-coded image by superimposing a color corresponding to the image for which the highest contrast is calculated on the base image; Equipped with When a user selects a desired color portion of the color-coded image, the image corresponding to the selected color among the near image, the mid image, and the far image is used as the displayed image. Medical imaging systems.

2. The selection processing unit selects to switch the display image from among the near image, the mid image, the far image, the EDOF image obtained by combining the near image and the mid image, the EDOF image obtained by combining the mid image and the far image, and the EDOF image obtained by combining the near image, the mid image, and the far image, based on the surgery mode. The medical imaging system of claim 1 .

3. Regarding the optical path lengths from the imaging lenses of a first imaging element that captures the Mid image, a second imaging element that captures the Near image, and a third imaging element that captures the Far image, the difference in the optical path lengths between the first imaging element and the second imaging element is smaller than the difference in the optical path lengths between the first imaging element and the third imaging element. The medical imaging system of claim 1 .

4. An EDOF (Extended Depth of Field) image generating unit that calculates the contrast for each pixel of the near image, the mid image, and the far image, selects the pixel with the highest contrast, and generates a synthesized EDOF image. The medical imaging system of claim 1 further comprising:

5. The surgery modes include a cataract surgery mode, a vitreous surgery mode, and a laparoscopic mode. The medical imaging system of claim 1 .

6. a surgery mode setting unit that sets a surgery mode; a selection processing unit that selects, based on the surgery mode, to switch the displayed image from among a Near image focused on a near point, a Mid image focused on a middle point, a Far image focused on a far point, and an EDOF (Extended Depth of Field) image in which the depth of field is extended by combining these images, all of which are captured by at least three image capturing elements with different optical path lengths from one imaging lens; a color-coded image generating unit that uses one of the near image, the mid image, and the far image as a base image, calculates contrast for each pixel of the near image, the mid image, and the far image, and generates a color-coded image by superimposing a color corresponding to the image for which the highest contrast is calculated on the base image; Equipped with When a user selects a desired color portion of the color-coded image, the image corresponding to the selected color among the near image, the mid image, and the far image is used as the displayed image. A medical imaging device comprising:

7. Medical imaging systems Setting a surgery mode; Selecting to switch the displayed image based on the surgery mode from among a Near image focused on a near point, a Mid image focused on a middle point, a Far image focused on a far point, and an EDOF (Extended Depth of Field) image in which the depth of field is extended by combining these images, all of which are captured by at least three imaging elements with different optical path lengths from one imaging lens; one of the near image, the mid image, and the far image is used as a base image, contrast is calculated for each pixel of the near image, the mid image, and the far image, and a color-coded image is generated by superimposing a color corresponding to the image for which the highest contrast is calculated on the base image; Including, When a user selects a desired color portion of the color-coded image, the image corresponding to the selected color among the near image, the mid image, and the far image is used as the displayed image. Control method.

Images