Medical imaging system, medical imaging apparatus, and control method

Abstract

The present disclosure relates to a medical imaging system, a medical imaging device, and a control method capable of appropriately displaying an in-focus image according to a surgery. The medical imaging system comprises a surgical mode setting unit, an area-of-interest setting unit, and a focus processing unit. The surgical mode setting unit sets a surgical mode. On the basis of the surgical mode, the area-of-interest setting unit sets, from among a near image, a mid image, and a far image, an ROI image to be used in implementing an AF process, and sets an area of interest in the ROI image, with the near, mid, and far images being captured by three imaging elements having different optical path lengths from one imaging lens, the near image being focused on a near point, the mid image being focused on a mid point, and the far image being focused on a far point. The focus processing unit determines an evaluation value from the area of interest of the ROI image and adjusts the focus. This technology is applicable to, for example, a medical imaging system capable of capturing an EDOF image.

Description

[Technical Field]

[0001] The present disclosure relates to a medical imaging system, a medical imaging apparatus, and a control method, and more particularly to a medical imaging system, a medical imaging apparatus, and a control method that are capable of displaying images that are appropriately focused according to surgery. [Background technology]

[0002] Conventionally, in the medical field, when performing procedures such as surgery while observing an affected area as a region of interest in an image captured through a lens, the image becomes out of focus in the near or far region outside the depth of field. Therefore, whenever the region of interest changes to the near or far side, it is necessary to adjust the focus so that the region of interest is in focus, and there is a concern that images with a shallow depth of field will reduce work efficiency. Therefore, there is a demand for a medical imaging system that can capture images with a deep depth of field.

[0003] For example, Patent Document 1 discloses a medical observation device capable of acquiring an extended depth of field (EDOF) image with an extended depth of field. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-158764 Summary of the Invention [Problem to be solved by the invention]

[0005] Depending on the type of surgery, it may be necessary to appropriately control AF (Auto Focus) to obtain the EDOF effect. For example, if the EDOF effect cannot be obtained, an image may be displayed that appears to be focused on the far point of the depth of field relative to the near point.

[0006] The present disclosure has been made in view of the above circumstances, and makes it possible to display an image that is appropriately focused according to the surgery. [Means for solving the problem]

[0007] A medical imaging system and a medical imaging device according to one aspect of the present disclosure include a surgical mode setting unit that sets a surgical mode, a focus area setting unit that sets an ROI image to be used for AF processing based on the surgical mode from among two or more types of images captured by at least two image sensors with different optical path lengths from a single imaging lens, and sets a focus area in the ROI image, which is an area in which a contrast AF evaluation value is obtained, and a focus processing unit that obtains an evaluation value from the focus area of ​​the ROI image and adjusts the focus.

[0008] A control method according to one aspect of the present disclosure includes setting a surgical mode, setting an ROI image to be used for AF processing based on the surgical mode from among two or more types of images captured by at least two image sensors having different optical path lengths from a single imaging lens, setting a region of interest in the ROI image as an area for obtaining a contrast AF evaluation value, and obtaining the evaluation value from the region of interest in the ROI image to adjust the focus.

[0009] In one aspect of the present disclosure, a surgical mode is set, and an ROI image to be used for AF processing is set based on the surgical mode from among two or more types of images captured by at least two image sensors with different optical path lengths from one imaging lens, and a region of interest, which is an area in which a contrast AF evaluation value is obtained, is set in the ROI image, and the evaluation value is obtained from the region of interest in the ROI image to adjust the focus. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of an embodiment of a medical imaging system to which the present technology is applied. [Figure 2]FIG. 2 is a diagram illustrating the configuration of an endoscope and a device unit. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of an imaging module. [Figure 4] 1A and 1B are diagrams for explaining a Mid image, a Near image, a Far image, an EDOF image, and a color-coded image. [Figure 5] FIG. 10 is a diagram illustrating an example of the configuration of a focus control function of a CCU. [Figure 6] 10 is a flowchart illustrating a focus control process. [Figure 7] 10 is a flowchart illustrating an AF process. [Figure 8] 10 is a flowchart illustrating a selection map generation process. [Figure 9] 1 is a block diagram illustrating an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <Configuration example of medical imaging system> FIG. 1 is a diagram showing an example of the configuration of an embodiment in which a medical imaging system to which the present technology is applied is applied to endoscopic surgery.

[0013] The medical imaging system 11 shown in FIG. 1 is configured to include an endoscope 12, an energy treatment tool 13, a display device 14, and a device unit 15.

[0014] For example, in a surgery using the medical imaging system 11, the endoscope 12 and the energy treatment tool 13 are inserted into the patient's body, and the forceps 16 is also inserted into the patient's body. In the medical imaging system 11, an image of an affected area such as a tumor taken by the endoscope 12 is displayed in real time on the display device 14, and a doctor can treat the affected area using the energy treatment tool 13 and the forceps 16 while viewing the image.

[0015] As shown in Fig. 2, the endoscope 12 is configured by attaching a cylindrical lens barrel 22 incorporating an optical system such as an objective lens to a camera head 21 incorporating an imaging module (see Fig. 3) having a plurality of imaging elements. For example, the lens barrel 22 is a scope formed into a cylindrical shape using a hard or soft material, and a light guide extending inside the lens barrel 22 guides light to the tip, enabling the light to be irradiated into the patient's body cavity. The camera head 21 is configured so that an optical element such as a birefringent mask (BM) can be inserted between the lens barrel 22 and the camera head, and images of the inside of the patient's body cavity can be captured via the optical system of the lens barrel 22.

[0016] The energy treatment device 13 is a medical instrument used in endoscopic surgery, for example, to remove an affected area or seal a blood vessel by using heat generated by a high-frequency current.

[0017] The display device 14 can display the image captured by the endoscope 12 as is, or can display the image that has been subjected to image processing in the device unit 15.

[0018] The equipment unit 15 is configured by incorporating various devices required for performing endoscopic surgery using the medical imaging system 11. For example, as shown in Fig. 2, the equipment unit 15 can be configured to include a light source device 31, a CCU (Camera Control Unit) 32, a recording device 33, and an output device 34.

[0019] The light source device 31 supplies light to the endoscope 12 via an optical fiber or the like, with which the affected area is irradiated when the endoscope 12 captures an image.

[0020] The CCU 32 controls imaging by the endoscope 12 and performs image processing on the images captured by the endoscope 12. The CCU 32 also has, for example, a focus control function for appropriately controlling the focus when capturing an image by the endoscope 12 in accordance with the surgical mode, and an image selection function for appropriately selecting an image captured by the endoscope 12 in accordance with the surgical mode.

[0021] The recording device 33 records on a recording medium the image output from the CCU 32. The output device 34 prints and outputs the image output from the CCU 32, or outputs it via a communication network.

[0022] <Configuration example of imaging module> FIG. 3 is a diagram showing an example of the configuration of an imaging module incorporated into the camera head 21 of the endoscope 12. As shown in FIG.

[0023] 3, the imaging module 41 is configured to include a branching optical system 51 and three imaging elements 52-1 to 52-3. In addition, an imaging lens 42 is arranged on the optical axis of light incident on the imaging module 41. The imaging lens 42 is configured with one or more lenses, and focuses light toward the imaging elements 52-1 to 52-3 so that imaging is performed using light entering the barrel portion 22 of the endoscope 12, and causes the light to enter the branching optical system 51.

[0024] The branching optical system 51 branches the light incident via the imaging lens 42 toward each of the imaging elements 52-1 to 52-3. The branching optical system 51 is composed of a first prism 61, a second prism 62, a third prism 63, a first dichroic mirror 64, and a second dichroic mirror 65.

[0025] The first prism 61, the second prism 62, and the third prism 63 constitute a prism block that is bonded together so that no air gaps are formed between the first prism 61 and the second prism 62 and between the second prism 62 and the third prism 63. By employing a prism block with a so-called gapless structure in this way, the branching optical system 51 can avoid the inclusion of process dust and the seepage of sealing material. Therefore, the branching optical system 51 can eliminate the reflection of foreign matter and suppress degradation of image quality even in a lens system with a relatively large F-number, such as that of the endoscope 12.

[0026] The first dichroic mirror 64 is an optical thin film made of a dielectric multilayer film formed on the exit surface of the first prism 61 facing the second prism 62, and splits the light into light amounts such that, for example, the average reflectance:average transmittance=1:2.

[0027] The second dichroic mirror 65 is an optical thin film made of a dielectric multilayer film formed on the exit surface of the second prism 62 facing the third prism 63, and splits the light into light amounts such that, for example, the average reflectance:average transmittance=1:1.

[0028] The image pickup elements 52-1 to 52-3 are, for example, CMOS image sensors having RGB filters in a Bayer array. The image pickup element 52-1 is disposed at a position where the distance (optical path length) from the principal point of the imaging lens 42 is a reference intermediate distance. The image pickup element 52-2 is disposed at a position farther away from the branching optical system 51 by the shift amount ΔZ so that the distance from the principal point of the imaging lens 42 is longer than the reference distance. The image pickup element 52-3 is disposed at a position closer to the branching optical system 51 by the shift amount ΔZ so that the distance from the principal point of the imaging lens 42 is shorter than the reference distance.

[0029] As a result, when the focal length of the imaging lens 42 is adjusted so that the imaging element 52-1 captures an image focused on the region of interest, the imaging element 52-2 captures an image focused on a point closer to the region of interest. Similarly, in this case, the imaging element 52-3 captures an image focused on a point farther from the region of interest. Therefore, hereinafter, the image captured by the imaging element 52-1 will be referred to as the "Mid image," the image captured by the imaging element 52-2 will be referred to as the "Near image," and the image captured by the imaging element 52-3 will be referred to as the "Far image."

[0030] Therefore, the imaging module 41 is configured to be able to output a near image, a mid image, and a far image to the CCU 32.

[0031] The medical imaging system 11 can switch between Near images, Mid images, and Far images and output them to the display device 14, as well as switch between EDOF images and color-coded images that have been image-processed in the CCU 32 and output them to the display device 14.

[0032] FIG. 4 shows images of a Near image, a Mid image, a Far image, an EDOF image, and a color-coded image that are switched and displayed in the medical imaging system 11.

[0033] For example, a Near image is captured so that the focus is on the near point, and the image becomes increasingly blurred as it moves toward the far point. A Mid image is captured so that the focus is on the middle point, and the image becomes blurred on both the near point and far point sides. A Far image is captured so that the focus is on the far point, and the image becomes increasingly blurred as it moves toward the near point.

[0034] An EDOF image is an image that has been processed to expand the depth of field so that the image is in focus from the near point to the far point by calculating the contrast for each pixel in the near image, mid image, and far image, and then selecting and combining the pixels with the highest contrast.

[0035] The color-coded image is an image in which the contrast is calculated for each pixel in the Near, Mid, and Far images, and the image is color-coded with the color corresponding to the image with the highest contrast, and is used to select areas. For example, the color-coded image is color-coded so that the pixel with the highest contrast in the Near image is red (solid line in Figure 4), the pixel with the highest contrast in the Mid image is green (dash line in Figure 4), and the pixel with the highest contrast in the Far image is blue (dashed line in Figure 4).

[0036] In the medical imaging system 11, when a user inputs a surgical mode using, for example, a user interface displayed on the display device 14, the input surgical mode is set in the CCU 32. Then, the CCU 32 can perform focus control in accordance with the set surgical mode so that an appropriately focused image is captured.

[0037] For example, in anterior segment surgery, it is desirable to set the ROI image to the Near image, set a region of interest on the cornea captured in the ROI image, and perform focus control so that the cornea is in focus in the Near image and the crystalline lens is in focus in the Mid image. In fundus surgery, it is desirable to set the ROI image to the Far image, set the region of interest on the fundus captured in the ROI image, and perform focus control so that the fundus is in focus in the Far image. In crystalline lens surgery, it is desirable to set the ROI image to the Mid image, set a region of interest on the crystalline lens captured in the ROI image, and perform focus control so that the crystalline lens is in focus in the Mid image. In laparoscopic surgery, it is desirable to set the ROI image to the Mid image, set a region of interest in the center of the ROI image, and perform focus control so that the center of the Mid image is in focus.

[0038] In this way, focus control is performed so that the Near image, the Mid image, and the Far image each have a different focus position according to the surgical mode. Further, in the EDOF image, since the region with the highest contrast among the Near image, the Mid image, and the Far image is synthesized, it is necessary to set the attention region to be focused on for focus adjustment to the region to be synthesized respectively.

[0039] <Focus control function of CCU> FIG. 5 is a block diagram for explaining the focus control function of the CCU 32.

[0040] As shown in FIG. 5, the CCU 32 includes a surgical mode setting unit 71, an attention region setting unit 72, an imaging signal acquisition unit 73, an EDOF image output unit 74, a color-separated image output unit 75, a focus processing unit 76, and a selection map generation unit 77.

[0041] When the surgical mode is input by the user using, for example, an input unit (not shown), the surgical mode setting unit 71 sets the surgical mode to the attention region setting unit 72.

[0042] Based on the surgical mode set by the surgical mode setting unit 71, the attention region setting unit 72 sets which of the Near image, the Mid image, and the Far image to be used for the ROI (Region of Interest) image, which is the region for obtaining the evaluation value of the contrast AF and is the target for setting the attention region, to the imaging signal acquisition unit 73. Further, the attention region setting unit 72 sets a predetermined location based on the surgical mode to the attention region in the ROI image.

[0043] For example, the region of interest setting unit 72 sets the ROI image to the Mid image in the cataract surgery mode, the Mid image in the vitreous surgery mode, and the Mid image in the laparoscopic surgery mode. The region of interest setting unit 72 also sets the ROI image to the Near image in the retinal detachment surgery mode, the Near image in the corneal transplant surgery mode, and the Far image in the macular disease surgery mode. For example, an image to be used as the ROI image is set for each surgery mode in a preset, but it is preferable that the user be able to change the preset and set any image as the ROI image. Note that if no surgery mode is set, the ROI image may be set to the Mid image.

[0044] In the cataract surgery mode, the attention area setting unit 72 sets the area of ​​interest at the center of the screen where the iris is captured. For example, in ophthalmic surgery, the position of the eye where the imaging area is set is conventionally determined by medical staff, so it is preferable to prepare a table of attention areas for each surgery mode in advance. Note that a configuration may be adopted in which the detection target and image recognition algorithm are determined based on the surgery mode by image recognition to detect the attention area.

[0045] The imaging signal acquisition unit 73 acquires imaging signals of the near image, mid image, and far image output from the imaging module 41. Then, the imaging signal acquisition unit 73 sets an image from the near image, mid image, and far image set by the region-of-interest setting unit 72 as an ROI image, and sets a region of interest for the ROI image. The imaging signal acquisition unit 73 supplies the ROI image in which the region of interest is set to the focus processing unit 76. The imaging signal acquisition unit 73 supplies the near image, mid image, and far image to the EDOF image output unit 74, the color-coded image output unit 75, and the selection map generation unit 77. The imaging signal acquisition unit 73 can switch between the near image, mid image, and far image and output it to the display device 14.

[0046] The EDOF image output unit 74 calculates the contrast for each pixel of the Near image, Mid image, and Far image, selects the pixel with the highest contrast, and combines them to output an EDOF image with an expanded depth of field so that the image is in focus from the near point to the far point.

[0047] The color-coded image output unit 75 calculates the contrast for each pixel in the near image, mid image, and far image, and outputs a color-coded image that is color-coded with the color corresponding to the image with the highest contrast. For example, the color-coded image can be used to select a region.

[0048] The focus processing unit 76 controls the optical system of the endoscope 12 so that the center of the ROI image supplied from the imaging signal acquisition unit 73 is most in focus, and adjusts the focus when imaging is performed by the imaging module 41. The focus processing unit 76 may also roughly calculate and adjust the focus position in the ROI image by wobbling. Furthermore, the focus processing unit 76 performs fine adjustment of the focus using a selection map generated by a selection map generation unit 77.

[0049] The selection map generating unit 77 generates a selection map used for finely adjusting the focus when imaging is performed by the imaging module 41, and supplies the selection map to the focus processing unit .

[0050] For example, the selection map generation unit 77 calculates the contrast for each pixel in the near image, mid image, and far image, and selects the image with the highest calculated contrast from the near image, mid image, and far image. That is, the selection map generation unit 77 compares the contrast with that of surrounding pixels within the same image and selects the image with the pixel with the largest difference (highest contrast). The selection map generation unit 77 associates an identification number (a number assigned to each of the near image, mid image, and far image) that identifies the selected image with each pixel. Then, the selection map generation unit 77 can generate a selection map by associating an identification number with every pixel.

[0051] Therefore, in the CCU 32, the focus processing unit 76 can perform fine adjustment of the focus using the selection map generated by the selection map generating unit 77.

[0052] For example, each time the focus processing unit 76 repeats focus adjustment, it determines whether the identification number associated with the pixel of the region of interest in the selection map generated from the image captured with the previously adjusted focus matches the image set as the ROI image. If the identification number of the region of interest in the selection map does not match the image set as the ROI image, the focus processing unit 76 adjusts the focus so that the identification number of the region of interest in the selection map matches the image set as the ROI image. At this time, the focus processing unit 76 may move the focus lens by a predetermined amount in a predetermined direction, or may determine the direction and amount of movement of the focus lens based on statistics of the contrast of the region of interest in the selection map.

[0053] Thereafter, when the identification number of the region of interest in the selection map matches the image set as the ROI image, the focus processing unit 76 performs contrast AF on the region of interest in the ROI image. This allows the region of interest in the ROI image to be focused on in the EDOF image. Note that the determination of whether the identification number of the region of interest in the selection map matches the image set as the ROI image may be based not only on a perfect match (100%) but also on a predetermined percentage match (e.g., 80%).

[0054] As described above, the focus control function of the CCU32 is configured to simultaneously acquire near, mid, and far images, and by changing the autofocus evaluation value calculation method depending on the surgical mode, it is possible to acquire the optimal EDOF image depending on the surgical mode. This makes it possible to obtain the optimal EDOF effect, which is the extended depth of field when performing surgery, thereby improving work efficiency.

[0055] Contrast AF is similar to focus evaluation in general AF, and focus processing unit 76 sets multiple focus frames on the image, evaluates the luminance values ​​in the focus frames, and determines the amount and direction of movement of the focus lens by looking at the correlation between each focus frame. The focus lens can be configured as part of imaging lens 42 in Figure 3.

[0056] The focus processing unit 76 then determines the amount and direction of movement of the focus lens using the focus evaluation of the focus frame that overlaps the region of interest. Focus evaluation is always performed for all focus frames, and the evaluation result of which focus frame to use is selected depending on the surgical mode. While contrast detection alone is generally used in the medical field, phase difference detection may also be used in combination.

[0057] Furthermore, the CCU 32 may be configured to adjust the focus after quickly changing the focus position as the user switches between, for example, a near image, a mid image, a far image, or an EDOF image. That is, it is preferable to switch the ROI image from the near image to the mid image as the user switches from the near image to the mid image. At this time, if the regions of interest for the near image, mid image, and far image are preset according to the surgical mode, the regions of interest are also switched at the same time. This allows the focus conditions to be changed when switching from the near image to the mid image, thereby achieving more appropriate autofocus.

[0058] <Example of focus control processing> The focus control process will be described with reference to the flowcharts shown in FIGS.

[0059] Fig. 6 is a flowchart illustrating the focus control process executed by the CCU 32. Note that Fig. 6 describes a process example in which one of the cataract surgery mode, vitreous surgery mode, and laparoscopic surgery mode is set as the surgery mode, but similar processes are performed when the various surgery modes described above are set.

[0060] For example, when a surgery mode is input by the user, the process starts. In step S11, the surgery mode setting unit 71 sets the surgery mode input by the user in the region-of-interest setting unit 72.

[0061] In step S12, the region-of-interest setting unit 72 determines whether the surgery mode set in step S11 is the cataract surgery mode, the vitreous surgery mode, or the laparoscopic surgery mode.

[0062] If the attention region setting unit 72 determines in step S12 that the cataract surgery mode is set, the process proceeds to step S13. In step S13, the attention region setting unit 72 sets the ROI image as the Mid image, and in step S14, sets the area in the center of the screen of the ROI image where the iris is reflected as the attention region.

[0063] On the other hand, if the region-of-interest setting unit 72 determines in step S12 that the vitreous surgery mode is set, the process proceeds to step S15. In step S15, the region-of-interest setting unit 72 sets the ROI image to the Mid image, and in step S16, sets the area in the center of the screen of the ROI image where the retina is displayed as the region of interest.

[0064] On the other hand, if the region-of-interest setting unit 72 determines in step S12 that the laparoscopic surgery mode has been set, the process proceeds to step S17. In step S17, the region-of-interest setting unit 72 sets the ROI image to the Mid image, and in step S18, sets the center of the screen of the ROI image as the region of interest. For example, if there is an affected area such as a tumor that is the target of surgery, the endoscope 12 is operated so that the affected area is displayed in the center of the screen of the ROI image, and the region-of-interest setting unit 72 can set the location where the affected area is displayed as the region of interest.

[0065] After processing step S14, step S16, or step S18, the process proceeds to step S19, where AF processing (see FIG. 7 described later) is performed to adjust the focus so as to focus on the region of interest set in the ROI image according to the surgical mode.

[0066] In step S20, an image selected from the Near image, Mid image, and Far image according to the surgical mode, which has been adjusted to be in focus by the AF processing in step S19, is output from the CCU 32 and displayed on the display device 14.

[0067] FIG. 7 is a flowchart illustrating the AF process executed in step S19 of FIG.

[0068] In step S31, the imaging signal acquisition unit 73 acquires imaging signals of the near image, the mid image, and the far image output from the imaging module 41.

[0069] 6 to the focus processing unit 76. Then, the focus processing unit 76 controls the optical system of the endoscope 12 so that the ROI image is best focused on the center of the ROI image. As a result, the focus is adjusted to focus on the imaging element 52 that captures the image set as the ROI image, among the imaging elements 52-1 to 52-3 of the imaging module 41, and imaging signals of the near image, mid image, and far image captured at that focus are supplied from the imaging signal acquisition unit 73 to the selection map generation unit 77.

[0070] In step S33, the selection map generation unit 77 executes a selection map generation process (see FIG. 8 described later) to generate a selection map using the imaging signals of the Near image, Mid image, and Far image supplied from the imaging signal acquisition unit 73 in step S32.

[0071] In step S34, the selection map generation unit 77 determines whether the identification number associated with the pixel of the region of interest in the selection map generated in the selection map generation process of step S33 matches the image set as the ROI image.

[0072] In step S34, if the selection map generation unit 77 determines that the identification number associated with the pixel in the region of interest of the selection map does not match the image set as the ROI image, the process returns to step S31, and the same process is repeated thereafter.

[0073] On the other hand, if the selection map generating unit 77 determines in step S34 that the identification number associated with the pixel in the region of interest of the selection map matches the image set as the ROI image, the process proceeds to step S35.

[0074] In step S35, the imaging signal acquisition unit 73 acquires an ROI image captured with the final focus adjusted by repeatedly performing the processes of steps S31 to S33, and supplies the ROI image to the focus processing unit 76. Then, the focus processing unit 76 performs contrast AF by calculating an evaluation value so as to focus on the region of interest set in the ROI image, and then the AF processing is terminated.

[0075] FIG. 8 is a flowchart illustrating the selection map generation process executed in step S33 of FIG.

[0076] In step S41, the selection map generating unit 77 resets the parameter i that identifies the pixel to be processed (i=0).

[0077] In step S42, the selection map generating unit 77 obtains the contrast of pixel i in each of the near image, mid image, and far image supplied from the imaging signal acquiring unit 73 in step S32 of FIG.

[0078] In step S43, the selection map generation unit 77 identifies the image with the highest contrast among the contrast of pixel i in the Near image, the contrast of pixel i in the Mid image, and the contrast of pixel i in the Far image obtained in step S42. Then, the selection map generation unit 77 associates an identification number that identifies the image with the highest contrast (Near image, Mid image, or Far image) with the pixel i being processed.

[0079] In step S44, the selection map generating unit 77 increments (i++) the parameter i that identifies the pixel to be processed.

[0080] In step S45, the selection map generation unit 77 determines whether or not identification numbers have been associated with all pixels in the near image, mid image, and far image supplied from the imaging signal acquisition unit 73 in step S32 of Fig. 7. For example, if the parameter i matches the number of pixels, the selection map generation unit 77 can determine that identification numbers have been associated with all pixels.

[0081] In step S45, if the selection map generation unit 77 determines that identification numbers have not been assigned to all pixels in the Near image, Mid image, and Far image, the process returns to step S42, and the same process is repeated thereafter.

[0082] On the other hand, if selection map generator 77 determines in step S45 that an identification number has been assigned to all pixels in the Near, Mid, and Far images, the process ends. That is, in this case, a selection map has been generated in which an identification number that identifies one of the Near, Mid, and Far images is assigned to all pixels.

[0083] As described above, the medical imaging system 11 can control AF so as to obtain the EDOF effect according to the surgery mode, and can display an appropriately focused image.

[0084] Although the present embodiment has been described with reference to an example configuration using three image sensors 52-1 to 52-3, the present technology can also be applied to a configuration using at least two image sensors, 52-1 and 52-2. In such a configuration, for example, three types of images, a Near image, a Mid image, and an EDOF image, can be selectively output according to the surgical mode.

[0085] <Computer configuration example> Next, the above-described series of processes (control method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0086] FIG. 9 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0087] The program can be recorded in advance on the hard disk 105 or ROM 103 as a recording medium built into the computer.

[0088] Alternatively, the program can be stored (recorded) on a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0089] The program can be installed into the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed on the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0090] The computer includes a CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0091] When a user inputs a command by operating input unit 107 via input / output interface 110, CPU 102 executes a program stored in ROM (Read Only Memory) 103 in accordance with the command. Alternatively, CPU 102 loads a program stored on hard disk 105 into RAM (Random Access Memory) 104 and executes it.

[0092] As a result, CPU 102 performs processing according to the flowchart described above or processing performed by the configuration of the block diagram described above. CPU 102 then outputs the processing results from output unit 106 via input / output interface 110, transmits them from communication unit 108, or records them on hard disk 105, as necessary.

[0093] The input unit 107 is made up of a keyboard, a mouse, a microphone, etc. The output unit 106 is made up of an LCD (Liquid Crystal Display), a speaker, etc.

[0094] In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

[0095] The program may be processed by a single computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to and executed on a remote computer.

[0096] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0097] Also, for example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0098] Furthermore, for example, this technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0099] Furthermore, for example, the above-described program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0100] Also, for example, each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0101] In addition, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0102] It should be noted that the present technologies described in this specification can be implemented independently and singly, unless a contradiction arises. Of course, any two or more of the present technologies can also be implemented in combination. For example, part or all of the present technologies described in any embodiment can be implemented in combination with part or all of the present technologies described in other embodiments. Furthermore, part or all of any of the present technologies described above can also be implemented in combination with other technologies not described above.

[0103] <Configuration combination example> The present technology can also be configured as follows. (1) a surgery mode setting unit that sets a surgery mode; a region of interest setting unit that sets a region of interest (ROI) image to be used for AF (Auto Focus) processing based on the surgery mode from two or more types of images captured by at least two image sensors with different optical path lengths from one imaging lens, and sets a region of interest in the ROI image, which is an area where a contrast AF evaluation value is to be obtained; a focus processing unit that calculates an evaluation value from a region of interest of the ROI image and adjusts the focus; A medical imaging system comprising: (2) Images are taken from one imaging lens using three imaging elements with different optical path lengths, and a Near image focused on the near point, a Mid image focused on the midpoint, and a Far image focused on the far point are used. The medical imaging system according to (1) above. (3) a selection map generating unit that calculates contrast for each pixel of the near image, the mid image, and the far image, selects the image with the highest contrast among the near image, the mid image, and the far image, and generates a selection map in which an identification number that identifies the selected image is associated with every pixel; Furthermore, The focus adjustment is repeatedly performed by the focus processing unit so that the identification number in the region of interest of the selection map matches the image set as the ROI image. The medical imaging system according to (2) above. (4) an EDOF (Extended Depth of Field) image generating unit that calculates contrast for each pixel of the near image, the mid image, and the far image captured by the focus processing unit with the focus adjusted on the region of interest of the ROI image, and generates an EDOF (Extended Depth of Field) image by selecting and combining pixels with the highest contrast; The medical imaging system according to (2) or (3) above, further comprising: (5) When a surgical mode for performing an anterior eye surgery is set, the region-of-interest setting unit sets the ROI image to the near image and sets the region of interest to the cornea captured in the ROI image. A medical imaging system according to any one of (2) to (4) above. (6) When a surgical mode for performing surgery on the fundus is set, the region-of-interest setting unit sets the ROI image to the far image and sets the region of interest on the fundus captured in the ROI image. A medical imaging system according to any one of (2) to (5) above. (7) When a surgical mode for performing a crystalline lens surgery is set, the region-of-interest setting unit sets the ROI image to the Mid image and sets the region of interest to the crystalline lens captured in the ROI image. A medical imaging system according to any one of (2) to (6) above. (8) When a surgery mode for performing surgery using a laparoscope is set, the region-of-interest setting unit sets the ROI image to the Mid image and sets the region of interest at the center of the ROI image. A medical imaging system according to any one of (2) to (7) above. (9) a surgery mode setting unit that sets a surgery mode; a region of interest setting unit that sets a region of interest (ROI) image to be used for AF (Auto Focus) processing based on the surgery mode from two or more types of images captured by at least two image sensors with different optical path lengths from one imaging lens, and sets a region of interest in the ROI image, which is an area where a contrast AF evaluation value is to be obtained; a focus processing unit that calculates an evaluation value from a region of interest of the ROI image and adjusts the focus; A medical imaging device comprising: (10) Medical imaging systems Setting a surgery mode; From among two or more types of images captured by at least two image sensors with different optical path lengths from one imaging lens, a ROI (Region of Interest) image to be used for AF (Auto Focus) processing is set based on the surgery mode, and a region of interest is set in the ROI image, which is an area for obtaining an evaluation value of contrast AF; obtaining an evaluation value from the region of interest of the ROI image and adjusting the focus; A control method comprising:

[0104] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained. [Explanation of symbols]

[0105] 11 medical imaging system, 12 endoscope, 13 energy treatment tool, 14 display device, 15 device unit, 16 forceps, 21 camera head, 22 lens barrel section, 31 light source device, 32 CCU, 33 recording device, 34 output device, 41 imaging module, 42 imaging lens, 51 branching optical system, 52-1 to 52-3 imaging element, 61 first prism, 62 second prism, 63 third prism, 64 first dichroic mirror, 65 second dichroic mirror, 71 surgical mode setting section, 72 region of interest setting section, 73 imaging signal acquisition section, 74 EDOF image output section, 75 color-coded image output section, 76 focus processing section, 77 selection map generation section

Claims

1. a surgery mode setting unit that sets a surgery mode; a region of interest (ROI) setting unit that sets a region of interest (ROI) image to be used for AF (Auto Focus) processing based on the surgery mode from two or more types of images captured by at least two image sensors with different optical path lengths from one imaging lens, and sets a region of interest in the ROI image, which is an area for obtaining an evaluation value of contrast AF; a focus processing unit that calculates an evaluation value from a region of interest of the ROI image and adjusts the focus; A medical imaging system comprising:

2. Images are taken from one imaging lens using three imaging elements with different optical path lengths, and a Near image focused on the near point, a Mid image focused on the middle point, and a Far image focused on the far point are used. The medical imaging system of claim 1 .

3. a selection map generating unit that calculates contrast for each pixel of the near image, the mid image, and the far image, selects the image with the highest contrast among the near image, the mid image, and the far image, and generates a selection map in which an identification number that identifies the selected image is associated with every pixel; Furthermore, The focus adjustment is repeatedly performed by the focus processing unit so that the identification number in the region of interest of the selection map matches the image set as the ROI image. The medical imaging system of claim 2 .

4. an EDOF (Extended Depth of Field) image generating unit that calculates contrast for each pixel of the near image, the mid image, and the far image captured by the focus processing unit with the focus adjusted on the region of interest of the ROI image, and generates an EDOF (Extended Depth of Field) image by selecting and combining pixels with the highest contrast; The medical imaging system of claim 2 further comprising:

5. When a surgical mode for performing an anterior eye surgery is set, the region-of-interest setting unit sets the ROI image to the near image and sets the region of interest to the cornea captured in the ROI image. The medical imaging system of claim 2 .

6. When a surgical mode for performing surgery on the fundus is set, the region-of-interest setting unit sets the ROI image to the far image and sets the region of interest on the fundus captured in the ROI image. The medical imaging system of claim 2 .

7. When a surgical mode for performing a crystalline lens surgery is set, the region-of-interest setting unit sets the ROI image to the Mid image and sets the region of interest to the crystalline lens captured in the ROI image. The medical imaging system of claim 2 .

8. When a surgery mode for performing surgery using a laparoscope is set, the region-of-interest setting unit sets the ROI image to the Mid image and sets the region of interest at the center of the ROI image. The medical imaging system of claim 2 .

9. a surgery mode setting unit that sets a surgery mode; a region of interest (ROI) setting unit that sets a region of interest (ROI) image to be used for AF (Auto Focus) processing based on the surgery mode from two or more types of images captured by at least two image sensors with different optical path lengths from one imaging lens, and sets a region of interest in the ROI image, which is an area for obtaining an evaluation value of contrast AF; a focus processing unit that calculates an evaluation value from a region of interest of the ROI image and adjusts the focus; A medical imaging device comprising:

10. Medical imaging systems Setting a surgery mode; From among two or more types of images captured by at least two image sensors having different optical path lengths from one imaging lens, a ROI (Region of Interest) image to be used for AF (Auto Focus) processing is set based on the surgery mode, and a region of interest is set in the ROI image, which is an area for obtaining an evaluation value of contrast AF; obtaining an evaluation value from the region of interest of the ROI image and adjusting the focus; A control method comprising:

Images