Signal processing devices, signal processing methods, programs, and medical image processing systems

Abstract

This disclosure relates to a signal processing apparatus, signal processing method, program, and medical image processing system capable of performing special light observation with lower noise. An EDOF signal processing unit performs EDOF signal processing as a recovery process on a special light image, which is obtained by imaging an observation target with special light. A setting unit sets the degree of recovery in the EDOF signal processing performed by the EDOF signal processing unit. The degree of recovery parameter for the EDOF signal processing of the special light image is set such that the degree of recovery in the EDOF signal processing of the special light image is lower than the degree of recovery in the EDOF signal processing as a recovery process for a normal light image, which is obtained by imaging an observation target with normal light. This technology can be applied, for example, to a medical image processing system performing special light observation.

Description

Technical Field

[0001] This disclosure relates to signal processing apparatus, signal processing methods, procedures, and medical image processing systems, and more particularly to signal processing apparatus, signal processing methods, procedures, and medical image processing systems capable of performing special light observations with low noise. Background Technology

[0002] Traditionally, medical observation devices such as endoscopes or microscopes typically produce images with shallow depth of field. In contrast, since surgical fields of view usually have depth, medical observation devices with deeper depth of field are required.

[0003] Therefore, in order to increase the depth of field, endoscopes, microscopes and other devices with extended depth of field (EDOF) optical systems have been proposed.

[0004] For example, the image processing apparatus disclosed in Patent Document 1 is provided with an EDOF optical system through a birefringent mask and a control unit that uses a function to adjust the amount of blur according to conditions, and can observe the target image in a more preferred manner according to the state, situation, etc. related to the observed target image.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-158764 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] However, in the past, when using EDOF optical systems to perform special light observations using special light with wavelengths different from normal light, there were concerns about increased noise due to EDOF signal processing. Therefore, even for images that have undergone EDOF signal processing, there is a need to be able to perform special light observations with lower noise.

[0008] This disclosure is made in view of this situation, and its purpose is to enable special optical observations to be performed with lower noise.

[0009] Solution for solving the problem

[0010] According to one aspect of the present disclosure, a signal processing apparatus includes: an EDOF signal processing unit that performs extended depth-of-field (EDOF) signal processing as a restoration process on a special light image, the special light image being obtained by imaging an observation target illuminated with special light; and a setting unit that sets the restoration degree of the EDOF signal processing of the EDOF signal processing unit, wherein the setting unit sets a parameter for the restoration degree in the EDOF signal processing of the special light image such that the restoration degree in the EDOF signal processing of the special light image is lower than the restoration degree in the EDOF signal processing as a restoration process for a normal light image, the normal light image being obtained by imaging an observation target illuminated with normal light.

[0011] A signal processing method or procedure according to one aspect of this disclosure includes performing extended depth-of-field (EDOF) signal processing as a restoration process on a special light image, which is obtained by imaging an observation target with special light; and setting the restoration degree of the EDOF signal processing, wherein a parameter is set for the restoration degree in the EDOF signal processing of the special light image such that the restoration degree in the EDOF signal processing of the special light image is lower than the restoration degree in the EDOF signal processing as a restoration process of a normal light image, which is obtained by imaging an observation target with normal light.

[0012] A medical image processing system according to one aspect of this disclosure includes: a light source for illuminating an observation target with special light or normal light; an imaging unit for imaging the observation target illuminated by special light or normal light; an extended depth-of-field (EDOF) optical system arranged on the optical axis of the light incident on the imaging unit; and a signal processing unit for performing signal processing on the image captured by the imaging unit, wherein the signal processing unit includes: an EDOF signal processing unit for performing EDOF signal processing as a recovery process on the special light image, the special light image being obtained by imaging the observation target with special light; and a setting unit for setting the recovery level of the EDOF signal processing of the EDOF signal processing unit, and the setting unit setting a parameter for the recovery level in the EDOF signal processing of the special light image such that the recovery level in the EDOF signal processing of the special light image is lower than the recovery level in the EDOF signal processing as a recovery process for the normal light image, the normal light image being obtained by imaging the observation target with normal light.

[0013] In one aspect of this disclosure, a parameter is set for the degree of restoration in EDOF signal processing of a special light image, such that the degree of restoration in EDOF signal processing of the special light image is lower than the degree of restoration in EDOF signal processing of a normal light image, which is obtained by imaging an observation target illuminated by normal light. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating a configuration example of a medical image processing system applying the present technology.

[0015] Figure 2 This is a diagram used to illustrate the characteristics of a birefringent mask.

[0016] Figure 3 This is a diagram illustrating an example of the modulation transfer function variation of a series of optical systems accompanying the insertion of a birefringent mask.

[0017] Figure 4 This is a diagram used to illustrate the definition of weakened EDOF signal processing.

[0018] Figure 5 This is a graph showing the relationship between the effectiveness of EDOF signal processing and noise.

[0019] Figure 6 It is a diagram used to illustrate the black areas of an image.

[0020] Figure 7 This is a block diagram illustrating an example configuration of a signal processing unit.

[0021] Figure 8 It is a flowchart used to illustrate signal processing.

[0022] Figure 9 This is a diagram used to illustrate time-division imaging.

[0023] Figure 10 This is a diagram showing an example of a settings screen for setting the recovery level of EDOF signal processing.

[0024] Figure 11 This is a block diagram illustrating a configuration example of a computer applying the present technology. Detailed Implementation

[0025] In the following, specific embodiments of the application of this technology are described in detail with reference to the accompanying drawings.

[0026] <Configuration Example of a Medical Image Processing System>

[0027] Figure 1 This is a block diagram illustrating a configuration example of a medical image processing system applying the present technology.

[0028] like Figure 1 As shown, the medical image processing system 11 is formed by mounting various devices for endoscopic surgery on a trolley 13 on which a display device 12 is placed, and can perform image processing, for example, on images obtained by an endoscope 14 for endoscopic surgery.

[0029] Display device 12 displays images obtained by endoscope 14, as well as images obtained by applying image processing to those images. For example, display device 12 can display images as described below. Figure 10 The settings screen shown has a user interface.

[0030] exist Figure 1 In the example shown, the light source device 21, the camera control unit (CCU) 22, and the signal processing device 23 are mounted on the trolley 13.

[0031] The light source device 21 is provided with, for example, an LED, a xenon lamp, a halogen lamp, a laser light source, or a combination thereof, and provides the illumination light to be applied to the target of observation to the endoscope 14 via a light guide. For example, see below. Figure 9 The light source device 21 can apply normal light and special light, and switch between them at a predetermined time.

[0032] CCU 22 controls imaging via an imaging element built into camera 32 and provides an image obtained by imaging the observed target via the imaging element. For example, see below. Figure 9 The CCU 22 allows the imaging element to image at a predetermined time.

[0033] The signal processing device 23 performs signal processing based on the image obtained by the endoscope 14, and performs image processing on the image. Note that reference will be made later. Figure 7 Describe the detailed configuration of the signal processing device 23.

[0034] The endoscope 14 includes a barrel 31 and a camera 32, and an optical element insertion unit 33 may be disposed between the barrel 31 and the camera 32. Note that, for example, in addition to a structure in which the optical element insertion unit 33 can be attached to and detached from the barrel 31 and the camera 32, a structure in which the optical element insertion unit 33 is part of the barrel 31, or a structure in which the optical element insertion unit 33 is part of the camera 32, etc., may be adopted.

[0035] The endoscope tube 31 is a tubular body formed from a rigid or flexible material, and a predetermined length thereof is inserted into the patient's body cavity. For example, an opening for fitting the objective lens is provided at the front end of the endoscope tube 31. Furthermore, an introduction unit for introducing light generated by the light source device 21 into the endoscope tube 31 is provided on the side surface of the endoscope tube 31, and the introduction unit is connected to the light source device 21 via a light guide. The light introduced into the endoscope tube 31 is then guided by the light guide extending into the endoscope tube 31 to the front end of the endoscope tube 31, and illuminates the target for observation within the patient's body cavity via the objective lens.

[0036] The camera 32 has a built-in imaging element for capturing images, an optical system for focusing light onto the imaging element, an aperture for adjusting the amount of light, etc., and captures images under the control of the CCU 22 and provides the images to the CCU 22.

[0037] The optical element insertion unit 33 is configured to allow optical elements such as a birefringent mask (BM) 41 to be inserted between the lens barrel 31 and the camera 32. Note that, in addition to this, there are also optical elements such as cubic phase masks that can be inserted by the optical element insertion unit 33.

[0038] For example, in the medical image processing system 11, by inserting various optical elements between the lens barrel 31 and the camera 32 using the optical element insertion unit 33, the optical characteristics of a series of optical systems that form the image of the subject on the imaging element in the camera 32 can be changed, and the amount of blur in the captured image can be adjusted (e.g., to control the depth of field).

[0039] Here, the configuration of the optical element in the optical element insertion unit 33 is described. The optical element insertion unit 33 is disposed between the endoscope barrel 31 and the camera 32 of the endoscope 14.

[0040] In recent years, the resolution of imaging elements (so-called image sensors) used in imaging devices such as cameras has tended to be higher, and not only "HD (1280×720)" but also "4K UHD (3840×2160)" and "8K UHD (7680×4320)" have been proposed. Therefore, similarly, in medical observation devices (imaging devices) such as the endoscope 14 according to this embodiment, it is desirable to improve the resolution of the captured image. Conversely, as the resolution increases, the pixel size of the imaging element tends to be smaller, and the amount of light gathered by each pixel tends to be relatively smaller. In this case, for example, there is a situation where insufficient light can be compensated by further opening the aperture (i.e., by making the f-value smaller), but conversely, there is a situation where the depth of field becomes narrower as the aperture is opened.

[0041] In view of the above, for example, there are cases where a technique called EDOF (Extended Depth of Field) is applied. In the endoscope 14 according to this embodiment, the depth of field of the captured image can be further extended by applying EDOF technology using a birefringence mask. Specifically, the endoscope 14 according to this embodiment is configured such that an optical element can be inserted through the optical element insertion unit 33 disposed between the endoscope barrel 31 and the camera 32 as described above, and the depth of field of the captured image can be controlled by inserting a birefringence mask as an optical element.

[0042] For example, Figure 1This illustrates an example configuration where the birefringent mask 41 is seen from the optical axis direction of camera 32 as an example of a birefringent mask 41 inserted by optical element insertion unit 33. Note that for Figure 1 The birefringent mask 41 shown in the figure, within the area enclosed by the dashed line, has the horizontal direction set as the x-direction, the vertical direction set as the y-direction, and the depth direction (i.e., the optical axis direction of the camera 32) set as the z-direction. Furthermore, in the following description, unless otherwise defined, the optical axis direction (in other words, the depth direction) of the imaging element built into the camera 32 is set as the z-direction, and the horizontal and vertical directions (i.e., the directions perpendicular to the optical axis) of the image captured by the imaging element are set as the x-direction and the y-direction, respectively.

[0043] In the birefringent mask 41, multiple polarization elements 42 are arranged concentrically from near the center outwards, and... Figure 1 In the example shown, three polarizing elements 42a to 42c are arranged. That is, in the birefringent mask 41, the three polarizing elements 42a to 42c are concentrically arranged on the xy plane perpendicular to the optical axis. Note that in Figure 1 In the birefringent mask 41, the arrows schematically indicate the polarization directions of the polarization elements 42 with arrows. That is, the polarization elements 42a to 42c are configured such that the polarization directions of adjacent elements are different from each other (approximately orthogonal).

[0044] For example, in Figure 1 In the example shown, the polarization direction of polarizing element 42a is the x-direction. Conversely, the polarization direction of polarizing element 42b, which is adjacent to polarizing element 42a, is the y-direction, i.e., a direction rotated 90 degrees relative to the polarization direction (x-direction) of polarizing element 42a. Similarly, the polarization direction of polarizing element 42c, which is adjacent to polarizing element 42b, is the x-direction, i.e., a direction rotated 90 degrees relative to the polarization direction (y-direction) of polarizing element 42b.

[0045] Using this configuration, light converged by the lens barrel 31 is incident on any one of the polarization elements 42a to 42c of the birefringent mask 41, depending on the position on the xy plane perpendicular to the optical axis (z direction), and light polarized by the polarization elements 42a to 42c is incident on the camera 32.

[0046] Here, refer to Figure 2 describe Figure 1 The characteristics of the birefringent mask 41 shown.

[0047] Figure 2 This is an explanatory diagram illustrating the characteristics of the birefringent mask 41 according to this embodiment. Specifically, Figure 2A is an example of the case where the birefringent mask 41 is not inserted between the lens barrel 31 and the camera 32, schematically showing the optical path of light converged by the lens barrel 31 and guided to the camera 32. Furthermore, Figure 2 B is an example of the case where the birefringent mask 41 is inserted between the lens barrel 31 and the camera 32, which schematically shows the light path of light converged by the lens barrel 31 and guided to the camera 32 via the birefringent mask 41.

[0048] like Figure 2 As shown in Figure A, the optical path of the light converged by the lens barrel 31 and guided to the camera 32 is controlled by the image forming optical system of the camera 32 to form an image on the image surface of the imaging element. Figure 2 In Figure A, the image indicated by reference numeral v11 schematically shows the subject image formed at the position indicated by reference numeral p11. Furthermore, the image indicated by reference numeral v13 schematically shows the subject image formed at the position indicated by reference numeral p13.

[0049] On the contrary, such as Figure 2 As shown in Figure B, the light converged by the lens barrel 31 is guided to the camera 32 via the birefringent mask 41, and its optical routing is controlled by the image forming optical system of the camera 32. Figure 2 In Figure B, the image indicated by reference numeral v21 schematically shows the subject image formed at the position indicated by reference numeral p11. Furthermore, the image indicated by reference numeral v23 schematically shows the subject image formed at the position indicated by reference numeral p13.

[0050] The comparison shows that the characteristics of the series of optical systems (hereinafter also referred to as "the series of optical systems") used to form the subject image on the imaging element of the camera 32 change with the insertion of the birefringent mask 41. Specifically, compared with before the insertion of the birefringent mask 41, the change in the image forming shape (i.e., the point spread function (PSF)) of the subject image between positions p11 and p13 is smaller with the insertion of the birefringent mask 41.

[0051] For example, Figure 3 This is an explanatory diagram illustrating an example of the characteristics of a birefringent mask 41 applied to an endoscope 14 according to this embodiment, showing an example of the variation of the modulation transfer function (MTF) of a series of optical systems accompanying the insertion of the birefringent mask 41.

[0052] exist Figure 3 In the diagram, the offset (i.e., defocusing amount) relative to the optical axis of a series of image-forming surfaces (in other words, focal positions) of the optical system is plotted along the horizontal axis, and the modulation transfer function (MTF) is plotted along the vertical axis. Furthermore, in... Figure 3In the figure, the curve indicated by reference numeral g11 shows the curve in the figure. Figure 2 Figure A shows an example of the modulation transfer function (MTF) of a series of optical systems when the birefringent mask 41 is not inserted between the lens barrel 31 and the camera 32. Furthermore, the curve indicated by reference numeral g13 shows the modulation transfer function (MTF) of such optical systems. Figure 2 The example shown in Figure B illustrates the modulation transfer function (MTF) of a series of optical systems in the case where a birefringent mask 41 is inserted between a lens barrel 31 and a camera 32.

[0053] like Figure 3 As shown, by applying the birefringent mask 41, the characteristics of a series of optical systems are altered in such a way that the modulation transfer function (MTF) is distributed over a wider range along the optical axis compared to before the application of the birefringent mask 41. That is, the depth of field can be further extended by applying the birefringent mask 41.

[0054] Conversely, refer to Figure 3 It can be seen that by applying the birefringence mask 41, the modulation transfer function (MTF) value at the focal position is smaller than the value before applying the birefringence mask 41. Therefore, in the medical image processing system 1 according to this embodiment, as Figure 2 As shown in B, restoration processing (image processing) is performed on the image captured by camera 32 to recover the image from image degradation (so-called blurring) of the subject image caused by the reduction of the modulation transfer function (MTF). For example, in Figure 2 In section B, the image indicated by reference numeral v25 shows an example of a subject image after restoration processing following the aforementioned restoration processing performed on the subject image v23. Note that, for example, there is a process called deconvolution as a restoration process (e.g., a process for adjusting the amount of blur). Needless to say, the restoration processing applied to the subject image is not necessarily limited to deconvolution, as long as it can be recovered from the aforementioned image degradation.

[0055] Through the aforementioned controls, for example, it is possible to obtain an image where the depth of field is expanded and the observed target is presented more clearly (i.e., a clearer image).

[0056] <Research on EDOF technology using birefringent masks>

[0057] This describes the technical problems encountered when EDOF technology using birefringent masks is applied in a medical image processing system according to this embodiment.

[0058] In the case of extending depth of field through a combination of birefringent masks and image processing (restoration processing), assuming that the optical properties of the birefringent masks and other optical systems are known, the design is performed as the content of image processing.

[0059] Here, in EDOF technology using birefringent masks, as described above, for example, a process called deconvolution is used to remove blur from an image captured via an optical system including a birefringent mask. In deconvolution, blur caused by the insertion of the birefringent mask is removed by appropriately switching the filter coefficients of the applied filter according to the optical characteristics of the optical system including the birefringent mask.

[0060] Note that, as deconvolution filters, there exist, for example, inverse filters, Wiener filters, etc. An inverse filter corresponds to a filter designed, for example, based on the optical characteristics (e.g., modulation transfer function (MTF)) of an optical system including a birefringent mask (e.g., the tube of an endoscope). That is, an inverse filter can be designed to have, for example, the inverse characteristics of the modulation transfer function of an optical system.

[0061] Furthermore, the spatial frequency characteristic WF of the Wiener filter is expressed by the calculation formula described in equation (1) below.

[0062] [Mathematical Expression 1]

[0063]

[0064] In Equation (1), when the directions parallel to and orthogonal to each other with respect to the image plane are the x and y directions, u and v represent the spatial frequencies in the x and y directions, respectively. Furthermore, H(u, v) represents the optical transfer function (OTF). Note that H*(u, v) represents the complex conjugate of H(u, v). Additionally, S... f (u, v) and S n (u, v) represent the power spectra of the original image and the noise, respectively.

[0065] Conversely, medical devices used in medical settings such as surgery, including cameras and endoscopes (e.g., rigid or flexible bodies), undergo high-pressure (hyperbaric steam) sterilization before each use. Therefore, for example, through repeated high-pressure sterilization of the endoscope, the optical characteristics of its optical system (i.e., rigid or flexible body), particularly the optical characteristics of the birefringent mask, gradually change. In this case, a discrepancy arises between the optical characteristics of the birefringent mask (i.e., the altered optical characteristics) and the optical characteristics assumed by image signal processing, thus affecting the image quality of the output image.

[0066] Specifically, as an optical property of a birefringent mask, there exists a parameter representing the optical phase difference, known as the retardation. The retardation is represented by the product Δn (=ne-no) and Δnd (the thickness d of the birefringent material). When a birefringent mask is subjected to a so-called autoclave treatment (high-pressure steam treatment), such as autoclave sterilization, the change in the characteristics of the refractive index difference Δn can be estimated (i.e., a decrease in the refractive index difference Δn). For example, due to the decrease in the value of the refractive index difference Δn, the effect on depth-of-field extension becomes smaller.

[0067] Conversely, despite the delay change of the birefringent mask as described above, when similar image processing (restoration processing) to that before the delay change is applied to an image captured using the birefringent mask, the image quality of the output image is likely to be so-called overemphasis. Therefore, there are cases where it is desirable to be able to output an image of better image quality by appropriately switching the content of image processing according to the delay change of the birefringent mask.

[0068] Note that the optical properties (delay) of a birefringent mask can be measured individually with respect to that birefringent mask. However, there are cases where the birefringent mask is incorporated into the optical system of an endoscope (e.g., a rigid mirror), a camera, etc., and in these cases, it is difficult to measure the optical properties of the birefringent mask individually.

[0069] Furthermore, restoration processing based on PSF information can be applied. By using an image captured by camera 32 as input, the subject image can be recovered from image degradation (i.e., blurring). The PSF information is obtained based on a measurement of the point image distribution function of the input image. For example, a process known as deconvolution can be used as a restoration process. As a more specific example, image degradation (blurring) occurring according to the optical properties represented by the obtained PSF information can be recovered by applying image processing (e.g., filtering) based on the inverse characteristics of the obtained PSF information to the input image. Needless to say, the restoration processing performed on the input image is not limited to deconvolution; any restoration processing that can improve the image degradation of the subject image based on the PSF information, such as using machine learning, can also be employed.

[0070] Here, in the restoration process using machine learning, for example, image degradation can be recovered by performing the following learning process. First, training data is prepared in pairs: the image before restoration and the image after restoration. Next, this training data is fed into a predetermined learning model to perform learning, thereby generating parameters for estimating the restored image from the image before restoration. Then, the restored image is generated by feeding the image before restoration into a restoration image generation model adjusted using these parameters. Note that the training model and the restoration image generation model are preferably computational models using multi-layer neural networks, and more preferably computational models based on reinforcement learning methods using multi-layer neural networks.

[0071] In medicine, there is an increasing demand for surgical observation using special light. Furthermore, when the target is illuminated by special light with a wavelength different from normal light, the EDOF effect produced by the EDOF optical system may differ from the EDOF effect when the target is illuminated by normal light.

[0072] For example, suppose we primarily use IR light as the special light source. Since IR light has a longer wavelength than normal light (visible light, white light), the refraction effect is reduced when IR light passes through a birefringent mask, thus diminishing the depth-of-field extension. Therefore, even if the same EDOF signal processing (MTF restoration processing) is performed on the image viewed through the special light source as it is on the image viewed through visible light, it only results in increased noise, and the depth-of-field extension is reduced.

[0073] That is, by performing EDOF signal processing, the noise in the original image may increase, or partial instability of the MTF may occur as a trade-off with depth-of-field extension. Therefore, in special lighting observation using IR light, not only the increase in noise and MTF instability are emphasized, but the effect of extending the depth of field is also reduced. Furthermore, in special lighting observation, the degree of restoration is made lower than that under visible light.

[0074] Therefore, the medical image processing system 11 of this disclosure proposes a technique that enables the provision of surgical field images with a more suitable EDOF effect (EDOF signal processing) even under special lighting conditions. For example, the medical image processing system 11 can reduce noise by making the EDOF signal processing in special lighting conditions weaker than that in normal lighting conditions.

[0075] Reference Figure 4 This describes the definition of reducing the degree of recovery through EDOF signal processing.

[0076] Figure 4Figure A illustrates the relationship between the modulation transfer function (MTF) and the amount of defocus when the degree of restoration in EDOF signal processing is appropriate. For example, it is considered that the degree of emphasis on matching the modulation transfer function (MTF) at the focused position is appropriate between the case without optical elements and the case with optical elements and restoration processing performed through EDOF signal processing.

[0077] Figure 4 Figure B illustrates the relationship between the modulation transfer function (MTF) and the amount of defocus when the degree of restoration in EDOF signal processing is low. For example, reducing the emphasis in EDOF signal processing is to allow a smaller MTF at the focused position when optical elements are present and restoration processing is performed through EDOF signal processing compared to the case without optical elements.

[0078] Figure 4 C illustrates the relationship between the modulation transfer function (MTF) and the amount of defocus when the degree of restoration through EDOF signal processing is high. For example, the emphasis in EDOF signal processing is enhanced to allow a larger modulation transfer function (MTF) at the focused position when optical elements are present and restoration processing is performed through EDOF signal processing compared to the case without optical elements.

[0079] Note that, for example, with respect to filters that correct the modulation transfer function (MTF), such as deconvolution filters, various mathematical formulas can be applied, and correction can be performed by changing their coefficients. Furthermore, any coefficient can be changed (increased / decreased), and this can, for example, be relative to the modulation transfer function (MTF) at the focus position when EDOF signal processing is not performed.

[0080] Reference Figure 5 This describes the balance between the degree of restoration achieved by EDOF signal processing and the noise generated in the image.

[0081] exist Figure 5 In the graph, the degree of restoration processing is plotted along the horizontal axis. Assuming the center of the horizontal axis (the intersection of the MTF curve and the noise curve) is appropriate, this means that relative to an appropriate degree of restoration processing, the degree of restoration processing decreases to the left and increases to the right.

[0082] Furthermore, the modulation transfer function (MTF) at the focus position is plotted along the left vertical axis; as shown by the MTF curve, the MTF at the focus position increases with increasing restoration processing and decreases with decreasing restoration processing. Conversely, the noise level is plotted along the right vertical axis; as shown by the noise curve, noise increases with increasing restoration processing and decreases with decreasing restoration processing.

[0083] In this way, a trade-off exists between the modulation transfer function (MTF) and noise at the focused position, depending on the degree of restoration processing. Therefore, when the degree of restoration processing is reduced to an appropriate level, noise decreases, but the MTF at the focused position also decreases. Thus, the extent to which the degree of restoration processing is reduced can be determined by referring to the balance between the EDOF effect and noise emphasis.

[0084] For example, regarding the extent of reduction in the degree of restoration processing, in a pre-hypothesized use case, the relationship with the noise reduction rate can be pre-formatted in a table. Then, referring to this table, the extent of reduction in the degree of restoration processing can be selected.

[0085] Furthermore, for example, the amount of noise can be measured from the image obtained by endoscope 14, and the degree of recovery processing can be reduced until the desired amount of noise is obtained. For example, as Figure 6 As shown, in the image obtained by endoscope 14, the amount of noise can be measured based on the brightness changes in the black image areas (mask areas, mechanical vignetting areas) where the subject is not imaged.

[0086] In addition, the medical image processing system 11 can analyze brightness, edge intensity, noise level, etc., from the image being captured and dynamically switch the degree of restoration. Furthermore, it can select the extent to which the degree of restoration processing is reduced, from the degree of normal light image synthesis to disabling EDOF signal processing.

[0087] Furthermore, in the medical image processing system 11, since a more blurred image is obtained by reducing the degree of restoration, other image processing techniques such as edge enhancement, contrast enhancement, and gamma correction can be combined to complement this. At this time, edge information from the normal light image can be applied to the specially captured light image to perform image processing that emphasizes edges.

[0088] Figure 7 This is a block diagram illustrating an example configuration of a signal processing device 23 that enables special optical observation with low noise.

[0089] like Figure 7 As shown, the signal processing device 23 includes an image acquisition unit 51, a noise estimation unit 52, an emphasis setting unit 53, and an EDOF signal processing unit 54.

[0090] The image acquisition unit 51 acquires a normal light image captured by the imaging element of the camera 32 during the period when the light source device 21 illuminates the observed target with normal light, and acquires a special light image captured by the imaging element of the camera 32 during the period when the light source device 21 illuminates the observed target with special light.

[0091] In the normal light image obtained by the image acquisition unit 51, the noise estimation unit 52, such as... Figure 6 As shown, the noise level is estimated based on the brightness variation in the black image regions (mask regions, mechanical vignetting regions) where the subject is not imaged.

[0092] The emphasis setting unit 53 sets the emphasis level for the EDOF signal processing unit 54 in the EDOF signal processing of the special light image based on the noise level estimated by the noise level estimation unit 52. For example, a predetermined value for setting the emphasis level to be reduced is preset in the emphasis setting unit 53. Then, when the noise level is equal to or greater than the predetermined value, the emphasis setting unit 53 sets the EDOF signal processing unit 54 to operate at a low emphasis level (see reference). Figure 4 B) Perform EDOF signal processing. Conversely, if the noise level is less than a specified value, the emphasis setting unit 53 sets the EDOF signal processing unit 54 to an appropriate degree of emphasis (refer to...). Figure 4 A) Perform EDOF signal processing. For example, the emphasis setting unit 53 can set parameters for adjusting the degree of emphasis in the EDOF signal processing of the EDOF signal processing unit 54.

[0093] The EDOF signal processing unit 54 performs EDOF signal processing on the normal light image and the special light image according to the settings of the emphasis setting unit 53. Then, the normal light image and the special light image after EDOF signal processing by the EDOF signal processing unit 54 are output from the signal processing device 23 and displayed on the display device 12.

[0094] Reference Figure 8 The flowchart shown describes the signal processing performed by the signal processing device 23.

[0095] For example, when a normal light image and a special light image are captured by the imaging element of camera 32, processing begins, and at step S11, image acquisition unit 51 obtains the normal light image and the special light image provided by camera 32. Then, image acquisition unit 51 provides the normal light image to noise estimation unit 52 and provides the special light image to EDOF signal processing unit 54.

[0096] At step S12, the noise estimation unit 52 estimates the noise level based on the black image region of the normal light image provided by the image acquisition unit 51 at step S11 (refer to...). Figure 6 The noise estimation unit 52 estimates the amount of noise generated in the black image area by measuring the brightness changes in the image. Then, the noise estimation unit 52 provides the estimated noise amount to the emphasis setting unit 53.

[0097] At step S13, the setting unit 53 determines whether the noise quantity provided by the noise quantity estimation unit 52 at step S12 is equal to or greater than a predetermined value set in advance.

[0098] In step S13, if the setting unit 53 determines that the noise level is equal to or greater than a specified value, the process proceeds to step S14.

[0099] At step S14, the emphasis setting unit 53 performs settings on the EDOF signal processing unit 54 to reduce the emphasis in the EDOF signal processing of the special light image.

[0100] After the processing in step S14, or if it is determined in step S13 that the noise level is not equal to or greater than the specified value (less than the specified value), the processing proceeds to step S15.

[0101] In step S15, the EDOF signal processing unit 54 performs EDOF signal processing on the special light image provided by the image acquisition unit 51 in step S11. At this time, if the noise level is equal to or greater than a predetermined value, the EDOF signal processing unit 54 applies the signal with a low emphasis level (refer to...). Figure 4 B) Perform EDOF signal processing, and, if the noise level is less than a specified value, apply an appropriate level of emphasis (refer to...). Figure 4 A) Perform EDOF signal processing.

[0102] Then, after the processing in step S15, the signal processing ends.

[0103] As described above, the signal processing device 23 can reduce the emphasis in EDOF signal processing of special light images based on the noise estimated from the normal light image. Therefore, the medical image processing system 11 can perform special light observation with lower noise.

[0104] Next, refer to Figure 9 Describes the timing for acquiring normal and special lighting images frame by frame. Note that... Figure 9 Examples are shown of applying strong EDOF signal processing to a normal light image and applying weak EDOF signal processing to a special light image.

[0105] like Figure 9 As shown, for example, the time periods during which the light source device 21 applies normal light and special light are set as a group, and this group is repeated frame by frame.

[0106] Then, the CCU 22 controls the imaging element of the camera 32 to perform imaging during normal light illumination to obtain a normal light image, and the EDOF signal processing unit 54 performs strong EDOF signal processing on the normal light image. Subsequently, the CCU 22 controls the imaging element of the camera 32 to perform imaging during special light illumination to obtain a special light image, and the EDOF signal processing unit 54 performs weak EDOF signal processing on the special light image.

[0107] In this way, the medical image processing system 11 can, when switching to a special light observation mode, make the recovery degree of EDOF signal processing performed on the special light image lower than that of the normal light image.

[0108] For example, in a configuration where an imaging element is provided in camera 32, imaging can be performed in a time-division (frame-sequence) manner, alternating between acquiring normal light images and special light images. In this case, since the focus position shifts due to the different wavelengths of normal light and special light, it is considered difficult to continuously focus on both the normal light image and the special light image. Therefore, in the medical image processing system 11, it is preferable to focus on the normal light image to avoid difficulties in determining the process and situation during endoscopic surgery. Thus, the special light image is out of focus, but since the EDOF effect is limited, this situation can be addressed by further reducing the degree of recovery through EDOF signal processing.

[0109] Note that in the configuration of the camera 32 with two imaging elements, normal light images and special light images can be captured continuously by each imaging element.

[0110] Figure 10 An example of a settings screen is shown for setting the degree of recovery of EDOF signals for normal light images and special light images.

[0111] like Figure 10 As shown, a slider is displayed below the preview image 62 shown on the settings screen 61. This slider serves as a user interface for setting the degree of recovery of the EDOF signal processing performed on the normal light image and the special light image.

[0112] That is, when slider 63 of the normal light image slider moves to the left, the recovery level of the EDOF signal processing performed on the normal light image is set to a lower level, and when slider 63 moves to the right, the recovery level of the EDOF signal processing performed on the normal light image is set to a higher level. Similarly, when slider 64 of the special light image slider moves to the left, the recovery level of the EDOF signal processing performed on the special light image is set to a lower level, and when slider 64 moves to the right, the recovery level of the EDOF signal processing performed on the special light image is set to a higher level.

[0113] Furthermore, when the restoration level of the normal light image changes, the restoration level of the special light image can also automatically change accordingly. For example, preferably, the restoration level of the special light image is always lower than that of the normal light image.

[0114] Note that, in addition to Figure 10 In addition to the slider shown, the degree of EDOF signal processing recovery can be selected by operating the buttons, and various user interfaces can be used. For example, a noise reduction button and a depth-of-field extension button can be used, and pressing the noise reduction button weakens the EDOF signal processing, while pressing the depth-of-field extension button strengthens the EDOF signal processing. In this case, preferably, noise can be reduced or depth of field extended while maintaining the relationship that EDOF signal processing for special light images is always weaker than EDOF signal processing for normal light images.

[0115] Furthermore, the medical image processing system 11 can be configured to display two special light images side by side (i.e., the image before the restoration level of the special light image is reduced and the image after the restoration level of the special light image is reduced), and allow the user to select the desired image.

[0116] <Computer Configuration Example>

[0117] Next, the above series of processes (signal processing methods) can be executed by hardware or software. In the case of executing the series of processes by software, the program forming the software is installed on a general-purpose computer or similar device.

[0118] Figure 11 This is a block diagram illustrating a configuration example of a computer equipped with a program that performs the series of processes described above.

[0119] The program can be pre-recorded in the hard disk 105 and ROM 103, which are recording media embedded in the computer.

[0120] Alternatively, the program can be stored (recorded) in a removable recording medium 111 driven by drive 109. This removable recording medium 111 can be configured as so-called packaged software. Here, the removable recording medium 111 includes, for example, a floppy disk, an optical disc read-only memory (CD-ROM), a magneto-optical (MO) disc, a digital multifunction disc (DVD), a magnetic disk, semiconductor memory, etc.

[0121] Note that the program can be installed on the computer from the removable recording medium 111 described above, or it can be downloaded to the computer via a communication network and a broadcast network for installation on the embedded hard disk 105. That is, for example, the program can be wirelessly transmitted to the computer from the download site via a satellite used for digital satellite broadcasting, or it can be wired to the computer via a network such as a local area network (LAN) and the Internet.

[0122] The central processing unit (CPU) 102 is embedded in the computer, and the input / output interface 110 is connected to the CPU 102 via the bus 101.

[0123] When a user inputs instructions by operating the input unit 107 through the input / output interface 110, the CPU 102 executes the program stored in the read-only memory (ROM) 103 in response. Alternatively, the CPU 102 loads the program stored in the hard disk 105 onto the random access memory (RAM) 104 for execution.

[0124] Therefore, CPU 102 performs processing according to the flowchart above, or performs processing through the configuration of the block diagram above. Then, for example, CPU 102 allows output unit 106 to output, or allows communication unit 108 to send, and further allows hard disk 105 to record processing results as needed through input / output interface 110, etc.

[0125] Note that the input unit 107 includes a keyboard, mouse, microphone, etc. Furthermore, the output unit 106 includes a liquid crystal display (LCD), speaker, etc.

[0126] Here, in this specification, the processing performed by the computer according to the program does not necessarily need to be executed sequentially in the order described in the flowchart. That is, the processing performed by the computer according to the program also includes parallel processing or independently executed processing (e.g., parallel processing or processing performed by an object).

[0127] Furthermore, the program can be processed by a single computer (processor) or by multiple computers in a distributed manner. Additionally, the program can be transferred to a remote computer for execution.

[0128] Furthermore, in this specification, the system is intended to represent the assembly of multiple components (devices, modules (components), etc.), and it is not important whether all components are in the same housing. Therefore, multiple devices stored in different housings and connected via a network, and a single device obtained by storing multiple modules in one housing, are both systems.

[0129] Furthermore, for example, a configuration described as one device (or processing unit) can be divided into multiple devices (or processing units). Conversely, the above-described configurations as multiple devices (or processing units) can be combined into a single device (or processing unit). Moreover, needless to say, configurations other than those described above can be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, a portion of the configuration of one device (or processing unit) can also be included in the configuration of another device (or another processing unit).

[0130] Furthermore, for example, this technology can be configured as cloud computing, where one function is shared by multiple devices over a network for collaborative processing.

[0131] Furthermore, for example, the above procedure can be executed by any device. In this case, only the device is required to have the necessary functions (function blocks, etc.) so that the necessary information can be obtained.

[0132] Furthermore, for example, each step described in the flowchart above can be executed by one device or by multiple devices in a shared manner. Additionally, when a step includes multiple processes, those processes can be executed by one or more devices in a shared manner. In other words, multiple processes included in one step can be executed as processes for multiple steps. Conversely, processes described as multiple steps can be executed uniformly as a single step.

[0133] Note that a program executed by a computer can cause the processes in the steps of the described program to be executed sequentially in the order described in this specification, or these processes to be executed in parallel or individually at required timings, such as when a call is made. That is, provided there is no inconsistency, the processes of the individual steps can be executed in a different order than described above. Furthermore, the processes in the steps of the described program can be executed in parallel with the processes of another program, or they can be executed in combination with the processes of another program.

[0134] Note that, provided there are no inconsistencies, each of the various techniques described in this specification can be implemented independently as a single unit. Needless to say, it can also be implemented by combining various arbitrary prior art techniques. For example, part or all of the techniques described in any embodiment can be implemented by combining part or all of the techniques described in other embodiments. Furthermore, any part or all of the above-described techniques can be implemented by combining with other techniques not described above.

[0135] <Configuration Combination Example>

[0136] Note that this technology can also have the following configurations. (1)

[0138] A signal processing device, comprising:

[0139] The EDOF signal processing unit performs extended depth-of-field (EDOF) signal processing on the special light image as a recovery process. This special light image is obtained by imaging the observed target with special light.

[0140] The setting unit configures the recovery level of the EDOF signal processing for the EDOF signal processing unit.

[0141] The setting unit sets parameters for the degree of restoration in EDOF signal processing of special light images, such that the degree of restoration in EDOF signal processing of special light images is lower than the degree of restoration in EDOF signal processing of normal light images, which are obtained by imaging an observation target illuminated by normal light. (2)

[0143] According to the signal processing device of (1) above, wherein

[0144] The setting unit sets parameters for the degree of restoration based on the noise level estimated from the normal light image, so as to reduce the degree of restoration of the special light image when the noise level is equal to or greater than a predetermined value. (3)

[0146] According to the signal processing device of (1) or (2) above, wherein

[0147] Special light images and normal light images are captured in frame order. (4)

[0149] The signal processing apparatus according to any one of (1) to (3) above, wherein

[0150] When the restoration level of the normal light image is changed using a predetermined user interface, the setting unit sets the parameters for the restoration level of the normal light image so that the restoration level of the special light image is always lower with this change. (5)

[0152] A signal processing method, comprising:

[0153] Extended Depth of Field (EDOF) signal processing is performed on the special light image as a recovery process using a signal processing device. This special light image is obtained by imaging the observed target with special light.

[0154] The recovery level of the EDOF signal is set using a signal processing device, wherein...

[0155] The parameters for the degree of restoration in EDOF signal processing of special light images are set such that the degree of restoration in EDOF signal processing of special light images is lower than the degree of restoration in EDOF signal processing of normal light images, which are obtained by imaging an observed target with normal light. (6)

[0157] A program that causes a computer in a signal processing device to perform the following processing:

[0158] Extended Depth-of-Field (EDOF) signal processing is performed on the special light image, which is obtained by imaging the observed target with special light; and

[0159] Set the recovery level of the EDOF signal processing, where

[0160] The parameters for the degree of restoration in EDOF signal processing of special light images are set such that the degree of restoration in EDOF signal processing of special light images is lower than the degree of restoration in EDOF signal processing of normal light images, which are obtained by imaging an observed target with normal light. (7)

[0162] A medical image processing system, comprising:

[0163] Light source: Illuminate the target being observed with special or normal light;

[0164] The imaging unit images the target being observed, whether illuminated by special light or normal light.

[0165] An extended depth-of-field (EDOF) optical system is positioned along the optical axis of the light incident on the imaging unit; and

[0166] The signal processing unit performs signal processing on the image captured by the imaging unit, wherein...

[0167] The signal processing unit includes:

[0168] The EDOF signal processing unit performs EDOF signal processing on a special light image as a recovery process. This special light image is obtained by imaging a target illuminated with special light.

[0169] The setting unit sets the recovery level of the EDOF signal processing for the EDOF signal processing unit, and

[0170] The setting unit sets parameters for the degree of restoration in EDOF signal processing of special light images, such that the degree of restoration in EDOF signal processing of special light images is lower than the degree of restoration in EDOF signal processing of normal light images, which are obtained by imaging an observed target with normal light.

[0171] Note that this embodiment is not limited to the above embodiment, and various changes can be made without departing from the spirit of this disclosure. Furthermore, the effects described in this specification are illustrative only and not restrictive; other effects may also exist.

[0172] Explanation of reference numerals in the attached figures

[0173] 11 Medical Image Processing System

[0174] 12 Display devices

[0175] 13. Stroller

[0176] 14. Endoscope

[0177] 21 Light source device

[0178] 22 CCU

[0179] 23 Signal processing device

[0180] 31 Lens tube

[0181] 32 cameras

[0182] 33 Optical Component Insertion Unit

[0183] 41 Birefringent mask

[0184] 42a to 42c polarization elements

[0185] 51 Image Acquisition Unit

[0186] 52 Noise Estimation Unit

[0187] 53. Emphasis on setting up units

[0188] 54 EDOF signal processing unit.

Claims

1. A signal processing apparatus, comprising: An extended depth-of-field signal processing unit performs extended depth-of-field signal processing as a recovery process on a special light image, wherein the special light image is obtained by imaging an observed object illuminated by special light, wherein the special light is infrared light; and The setting unit sets the recovery level of the extended depth-of-field signal processing of the extended depth-of-field signal processing unit, wherein... The setting unit sets a parameter for the degree of restoration in the extended depth-of-field signal processing performed on the special light image, such that the degree of restoration in the extended depth-of-field signal processing performed on the special light image is lower than the degree of restoration in the extended depth-of-field signal processing performed on the normal light image as a restoration process, the normal light image being obtained by imaging the observed object illuminated by normal light.

2. The signal processing apparatus according to claim 1, wherein, The setting unit sets the parameter for the degree of restoration based on the noise level estimated from the normal light image, so as to reduce the degree of restoration of the special light image when the noise level is equal to or greater than a predetermined value.

3. The signal processing apparatus according to claim 1, wherein, The special light image and the normal light image are captured in frame order.

4. The signal processing apparatus according to claim 1, wherein, When the restoration level of the normal light image is changed using a predetermined user interface, the setting unit sets a parameter for the restoration level of the normal light image such that the restoration level of the special light image is always lower with this change.

5. A signal processing method, comprising: The signal processing device performs extended depth-of-field signal processing on the special light image as a recovery process. The special light image is obtained by imaging an observed object illuminated by special light, wherein the special light is infrared light. The degree of recovery of the extended depth-of-field signal processing is set, wherein... A parameter is set for the degree of restoration in the extended depth-of-field signal processing performed on the special light image, such that the degree of restoration in the extended depth-of-field signal processing performed on the special light image is lower than the degree of restoration in the extended depth-of-field signal processing performed on a normal light image as a restoration process, the normal light image being obtained by imaging the observed object illuminated by normal light.

6. A computer-readable storage medium having a program stored thereon, the program, when executed by a computer of a signal processing apparatus, causing the computer of the signal processing apparatus to perform the following processes: Extended depth-of-field signal processing is performed on the special light image as a recovery process. The special light image is obtained by imaging an observed object illuminated by special light. The special light is infrared light; and The degree of recovery of the extended depth-of-field signal processing is set, wherein... A parameter is set for the degree of restoration in the extended depth-of-field signal processing performed on the special light image, such that the degree of restoration in the extended depth-of-field signal processing performed on the special light image is lower than the degree of restoration in the extended depth-of-field signal processing performed on a normal light image as a restoration process, the normal light image being obtained by imaging the observed object illuminated by normal light.

7. A medical image processing system, comprising: A light source is used to illuminate the object being observed with special light or normal light, wherein the special light is infrared light; An imaging unit that images the object of observation illuminated by the special light or the normal light; An extended depth-of-field optical system is arranged on the optical axis of the light incident on the imaging unit; and The signal processing unit performs signal processing on the image captured by the imaging unit, wherein... The signal processing unit includes: An extended depth-of-field signal processing unit performs extended depth-of-field signal processing as a recovery process on a special light image, the special light image being obtained by imaging the observed object illuminated by the special light; and The setting unit sets the recovery level of the extended depth-of-field signal processing of the extended depth-of-field signal processing unit, and The setting unit sets a parameter for the degree of restoration in the extended depth-of-field signal processing performed on the special light image, such that the degree of restoration in the extended depth-of-field signal processing performed on the special light image is lower than the degree of restoration in the extended depth-of-field signal processing performed on the normal light image as a restoration process, the normal light image being obtained by imaging the observed object illuminated by the normal light.

Images