Image processing apparatus, image processing method, navigation method, and endoscope system

Abstract

The image processing apparatus includes: a condition specification section that sets, for an image acquisition section, a first acquisition condition including a display acquisition condition and a second acquisition condition including the display acquisition condition and an analysis acquisition condition, the image acquisition section acquiring a first image based on the display acquisition condition for acquiring a display image and acquiring a second image based on the analysis acquisition condition for acquiring an image for image analysis; an image analysis section that performs image analysis on the image acquired by the image acquisition section; a support information generation section that generates support information based on the image analysis result of the image analysis section; and a control section that controls switching of the first acquisition condition and the second acquisition condition.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing apparatus, an image processing method, a navigation method, and an endoscope system for navigation when observing an image. BACKGROUND

[0002] In the past, a navigation technique for supporting various work using an image processing technique has been developed. For example, in the medical field, using an image processing technique, it is possible to perform insertion support for supporting insertion of an endoscope, diagnosis support for an estimation result of a disease, and the like. For example, a computer-aided diagnosis (CAD) has also been developed, which provides estimation results of a disease, and the like, based on determination of a fine structure that should be focused on at the time of provision / diagnosis of a quantitative judgment scale / image analysis. In an image processing apparatus that realizes such a CAD, insertion support, and the like, research has been conducted for providing appropriate support to a surgeon.

[0003] For example, in Japanese Patent Application Publication No. 2019-42156, a technique is disclosed in which two analysis results regarding a first medical image and a second medical image can be displayed in a manner that enables comparison of positions or ranges (sizes), and the like, and confirmation of the analysis results becomes easy.

[0004] However, with respect to a real-time medical image taken by an endoscope or the like, not only is image processing performed for image analysis, but also a monitor or the like performs image display, whereby in surgery, examination, and the like, it is possible to provide a surgeon with extremely useful image information of a lesion or the like. However, in Japanese Patent Application Publication No. 2019-42156, the taken image is not suitable for visual observation at the time of surgery, examination, and the like, and optimal support is not necessarily provided to a surgeon.

[0005] An object of the present application is to provide an image processing apparatus, an image processing method, a navigation method, and an endoscope system that can provide extremely effective support to a surgeon by optimizing a condition for taking an image. SUMMARY

[0006] Means for solving the problem

[0007] The image processing apparatus of one embodiment of the present application includes: a condition specification unit that sets, for an image acquisition unit, a first acquisition condition including a display acquisition condition and a second acquisition condition including the display acquisition condition and an analysis acquisition condition, the image acquisition unit mixedly acquiring a first image based on the display acquisition condition and a second image based on the analysis acquisition condition, in which the display acquisition condition is used to acquire a display image at a visual recognition frame rate, and the analysis acquisition condition is used to acquire an image for image analysis at a frame rate lower than the visual recognition frame rate; an image analysis unit that performs image analysis on an image acquired by the image acquisition unit; a support information generation unit that generates support information based on an image analysis result of the image analysis unit; and a control unit that controls switching between the first acquisition condition and the second acquisition condition in accordance with the image analysis result.

[0008] The image processing method of one embodiment of the present application includes the steps of: setting, for an image acquisition unit, a first acquisition condition including a display acquisition condition and a second acquisition condition including the display acquisition condition and an analysis acquisition condition, the image acquisition unit mixedly acquiring a first image based on the display acquisition condition and a second image based on the analysis acquisition condition, in which the display acquisition condition is used to acquire a display image at a visual recognition frame rate, and the analysis acquisition condition is used to acquire an image for image analysis at a frame rate lower than the visual recognition frame rate; performing image analysis on an image acquired by the image acquisition unit to obtain an image analysis result; generating support information based on the image analysis result; and controlling switching between the first acquisition condition and the second acquisition condition in accordance with the image analysis result.

[0009] In addition, in the navigation method of one embodiment of the present application, a first acquisition condition including a display acquisition condition is set for an image acquisition unit, the image acquisition unit mixedly acquires a first image based on the display acquisition condition and a second image based on an analysis acquisition condition, in which the display acquisition condition is used to acquire a display image at a visual recognition frame rate, and the analysis acquisition condition is used to acquire an image for image analysis at a frame rate lower than the visual recognition frame rate, a second acquisition condition including the display acquisition condition and the analysis acquisition condition is set for the image acquisition unit, image analysis is performed on an image acquired by the image acquisition unit to obtain an image analysis result, switching between the first acquisition condition and the second acquisition condition is controlled in accordance with the image analysis result, and a third acquisition condition is set for the image acquisition unit, the third acquisition condition including the display acquisition condition and an analysis acquisition condition different from the analysis acquisition condition included in the second acquisition condition.

[0010] Also, an endoscope system of one embodiment of the present application includes an endoscope including an illumination unit and an imaging unit, the endoscope mixedly acquiring a first image based on a display acquisition condition for acquiring a display image at a low frame rate for visual recognition and a second image based on an analysis acquisition condition for acquiring an image for image analysis at a frame rate lower than the frame rate for visual recognition, a video processor that causes the endoscope to acquire the first image and the second image based on at least one of the display acquisition condition and the analysis acquisition condition, and an image processing apparatus including an acquisition condition specification unit that specifies a first acquisition condition including the display acquisition condition and a second acquisition condition including the display acquisition condition and the analysis acquisition condition, an image analysis unit that performs image analysis on an image acquired by the video processor, a support information generation unit that generates support information based on a result of the image analysis by the image analysis unit, and a control unit that controls switching between the first acquisition condition and the second acquisition condition in accordance with the result of the image analysis. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a block diagram showing the structure of an endoscope system including an image processing apparatus of the first embodiment of the present application.

[0012] Figure 2 is a graph for explaining the demand for a display image and an analysis image.

[0013] Figure 3 is a diagram showing an example of a use mode of the endoscope system of Figure 1

[0014] Figure 4 is a graph for explaining the relationship between WLI light and NBI light irradiated from the endoscope of the first embodiment and blood vessels in the mucosa of an object.

[0015] Figure 5 is a graph for explaining the relationship between DRI light and blood vessels in the mucosa of an object.

[0016] Figure 6 is a diagram showing an example of an imaging image acquired by the video processor 3.

[0017] Figure 7 is a diagram showing an example of an image output to the monitor 5.

[0018] Figure 8 is a diagram showing an example of an image supplied to the image analysis unit 32.

[0019] Figure 9 ​is a graph for explaining an example of image processing by the image processing section 12 based on the display-use acquisition condition and the analysis-use acquisition condition.

[0020] Figure 10 is a flowchart for explaining the operation of the first embodiment.

[0021] Figure 11 is an explanatory diagram for explaining an image acquired in a specific use case.

[0022] Figure 12 is a graph for explaining an example of the acquisition condition I3 based on the determination by the determination section 34.

[0023] Figure 13 is an explanatory diagram for explaining the support display.

[0024] Figure 14 is a graph for explaining the priority order of the acquisition conditions for a plurality of requests.

[0025] Figure 15 is a flowchart showing the operation flow adopted by the second embodiment.

[0026] Figure 16 is a block diagram showing the third embodiment. DETAILED DESCRIPTION

[0027] Hereinafter, the embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0028] (First Embodiment)

[0029] Figure 1 is a block diagram showing the structure of an endoscope system including the image processing apparatus of the first embodiment of the present application.

[0030] For example, in an endoscope, when it is intended to acquire an image used for image analysis for navigation and an image displayed on a monitor for display by different imaging devices, it is unavoidable that the endoscope front end portion is upsized. For this reason, generally, the image used for image analysis for navigation is the same as the image for display. However, the image for display is acquired with an acquisition condition suitable for display, and information required for image analysis can be missing. Note that there is a possibility that the image for image analysis is poor in visual recognition, and it is not preferable to use the image for image analysis as the image for display. According to the above, it is difficult to support based on a high-precision analysis result while displaying an endoscope image easy to observe. Further, in the present specification, the high-precision analysis result means an analysis result capable of more effectively supporting a surgeon, and means not only a correct analysis result but also an analysis result of a kind required for support among various analysis results.

[0031] Thus, in the present embodiment, by being able to acquire a plurality of images different in acquisition conditions, i.e., a plurality of images including a visual recognition excellent image for image display and an analysis excellent image, it is possible to provide extremely effective support to the operator. Further, the analysis excellent image refers to an image that enables a high-precision analysis result.

[0032] Further, in the present embodiment, in order to acquire an image more excellent in analysis while maintaining the visual recognition excellent image display, it is also possible to adaptively change the image acquisition conditions. In addition, in the present embodiment, the image acquisition conditions are changed in accordance with the image display conditions, but the present embodiment is not limited thereto. Figure 1 The present embodiment is described taking an endoscope system as an example, but is not limited thereto, and can be applied to various devices for performing various operations with observation.

[0033] Figure 2 is a chart for explaining the demand for a display image and an analysis image.

[0034] The display image (display image) is an image for acquiring required information by recognizing the image displayed on the screen by human vision. On the other hand, the analysis image (analysis image) is an image that becomes an analysis object in a navigation device. If the quality of information processing by humans and computers is considered, the characteristics corresponding to the display image and the analysis image are different from each other.

[0035] As shown in Figure 2 , in order to make it easy for humans to recognize, the display image is preferably a visual recognition excellent image that contains only useful information as much as possible. For example, the display image is preferably an image with less noise, gamma processing that implements characteristics close to human eyes, and a frequency band that is emphasized for observation.

[0036] On the other hand, the analysis image is processed by a computer or the like, and thus the more information the image information for analysis contains, the more useful analysis results (high-precision analysis results) can be obtained. For example, regarding the quality, even if the analysis image is conspicuous image information other than the focus, the adverse effect on the analysis result is small. In addition, if noise reduction, gamma processing, image emphasis processing, or the like is performed on the image, the information required for analysis is sometimes missing, and thus for the analysis image, it is preferable not to perform such image processing.

[0037] In addition, for example, in the case of special light observation such as NBI (Narrow Band Imaging) that is effective for observation of blood vessels in the mucosa or the like, if the image recognition ability of humans is considered, as an image displayed on a monitor screen, it is preferable to display only one kind of special light observation image, or at least to limit the display to superimposing a special light observation image on a general light observation image.

[0038] On the other hand, even if a plurality of special light observation image signals are continuously input to the navigation device, no adverse effect is caused on the image analysis processing, and instead, the possibility of obtaining a useful analysis result from the plurality of image information is increased.

[0039] In addition, the frame rate of the display image is preferably 30 FPS or more in terms of human visual recognition, but even with a relatively low frame rate, for example, 1 FPS or less, useful information can be obtained for the analysis image.

[0040] (Structure)

[0041] Figure 3 is a diagram illustrating an example of a usage mode of the endoscope system of Figure 1 . Referring to Figure 3 , an example of a usage mode of the endoscope system will be described.

[0042] Figure 3 An example of performing a treatment in the abdominal cavity of a subject P using an endoscope system 1 is shown. The endoscope system 1 is an example of a laparoscopic surgery system. The endoscope system 1 mainly includes an endoscope 2 (laparoscope) that images the inside of a body cavity of a subject P and outputs an imaging signal, a video processor 3 that is connected to the endoscope 2 and controls the driving of the endoscope 2, and acquires an imaging signal related to the subject imaged in the endoscope 2 and performs a predetermined image processing on the imaging signal, a light source device 4 that is built in the video processor 3 and provides a predetermined illumination light for irradiating the subject, a monitor 5 that displays an observation image corresponding to the imaging signal, and a navigation device 30 that is connected to the video processor 3 and is an image processing device for performing diagnosis support and the like.

[0043] In Figure 3 , a case where the endoscope 2 and a treatment instrument 7 are inserted into the abdomen of the subject P via a trocar is shown. The endoscope 2 is connected to the video processor 3 via a general-purpose cord. The light source device 4 is built in the video processor 3, and is configured to illuminate the inside of the abdominal cavity by the light source device 4. The endoscope 2 is driven by the video processor 3, and images the inside of the abdominal cavity of the subject P. The imaging image acquired by the endoscope 2 is provided to the navigation device 30 after being subjected to signal processing by the video processor 3.

[0044] The navigation device 30 provides the input imaging image to the monitor 5 to be displayed, and generates support information by analyzing and processing the imaging image. The navigation device 30 outputs the generated support information to the monitor 5 as needed and causes it to be displayed, thereby supporting the operator.

[0045] In the present embodiment, the navigation device 30 gives an instruction to the video processor 3 to set the image acquisition conditions including at least one of the imaging conditions in imaging of the endoscope 2 and the image processing conditions in image processing of the video processor 3, thereby acquiring an image effective for support for image analysis while acquiring an image for image display that is excellent in visual recognition.

[0046] (endoscope)

[0047] In Figure 1 , as the endoscope 2, various endoscopes such as a digestive organ endoscope or a laparoscope can be employed. The endoscope 2 has an elongated insertion section that is inserted into a body lumen or the like of a subject, and an operation section that is provided at a base end side of the insertion section to be held by a surgeon to be operated. A general-purpose cord is provided to extend from a base end section of the operation section, and the endoscope 2 is detachably connected to the video processor 3 including the light source device 4 through the general-purpose cord.

[0048] A camera head 20 is provided at a distal end of the insertion section, for example. The camera head 20 includes an optical system 21, an imaging element 22, and an illumination section 23. The illumination section 23 is controlled by the light source device 4 to generate illumination light, and the generated illumination light is irradiated to an object. The illumination section 23 can also be a structure having a prescribed light source such as an LED (Light Emitting Diode) or the like not shown. In the present embodiment, the illumination section 23 can have a plurality of light sources including a light source that generates white light for general observation, a light source that generates narrow-band light for narrow-band observation, a light source that generates infrared light of a prescribed wavelength, and the like. The illumination section 23 has various irradiation modes, is controlled by the light source device 4, and can perform switching of the wavelength of the illumination light, control of the irradiation intensity, control of the temporal pattern of irradiation, and the like.

[0049] In addition, in Figure 1 , an example in which the illumination section 23 is provided in the camera head 20 is shown, but a structure can also be employed in which the light source device 4 generates illumination light, the illumination light is guided to the distal end of the endoscope 2 by a light guide not shown, and the object is irradiated.

[0050] The optical system 21 includes lenses, apertures, and the like not shown for zooming and focusing, and can include zoom (variable magnification) mechanisms, focusing mechanisms, and aperture mechanisms not shown that drive these lenses. The illumination light from the illumination section 23 is irradiated to the subject, and the return light from the subject is guided to the imaging surface of the imaging element 22 by the optical system 21.

[0051] The imaging element 22 is configured by a CCD, a CMOS sensor, or the like, and performs photoelectric conversion on the optical image of the object from the optical system 21 to acquire an imaging image (imaging signal) of the object. The camera head 20 outputs the acquired imaging image to the video processor 3.

[0052] The video processor 3 has a control section 11 that controls each section of the video processor 3, the imaging device 20, and the light source device 4. The control section 11 and each section in the control section 11 can be configured by a processor using a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or the like, can control each section by acting in accordance with a program stored in a memory (not shown), and can realize part or all of the functions by an electronic circuit of hardware.

[0053] (light source device)

[0054] The light source device 4 controls the illumination section 23 to generate white light and various special observation light. For example, the light source device 4 can also cause the illumination section 23 to generate white light, NBI (Narrow Band Imaging) light, DRI (Dual Red Imaging) light, and excitation light for AFI (Auto Fluorescence Imaging) (hereinafter referred to as AFI light). The white light is used as illumination light (hereinafter referred to as WLI light) for so-called WLI (white light imaging) observation (normal observation), the NBI light is used for narrow band light observation, the DRI light is used for long wavelength narrow band light observation, and the AFI light is used for fluorescence observation.

[0055] In addition, the illumination section 23 can be configured by a plurality of LEDs, laser diodes, xenon lamps, or the like to generate these illumination lights, or can generate these illumination lights using white light, NBI filters, DRI filters, AFI filters, or the like. The amount of light is increased or decreased by the illumination section 23, the exposure value at the time of imaging by the imaging device 20 can be changed, and exposure control that excludes the influence of saturation and low luminance noise can be performed. In addition, as the NBI light, blue light having a wavelength λ = 415 nm and green light having a wavelength λ = 540 nm can also be generated.

[0056] (video processor)

[0057] The control section 11 of the video processor 3 has an image processing section 12, an imaging parameter setting section 13, an image processing parameter setting section 14, and a display control section 15. The imaging parameter setting section 13 can control the light source device 4 to set the state of the illumination light generated by the illumination section 23. In addition, the imaging parameter setting section 13 can control the imaging device 20 to set the state of the optical system of the optical system 21 and the driving state of the imaging element 22.

[0058] That is, the imaging parameter setting section 13 can set the imaging conditions including the optical conditions at the time of imaging of the imaging device 20 and the driving conditions of the imaging element 23. For example, by the setting of the imaging parameter setting section 13, NBI light, DRI light, AFI light, and the like can be generated as illumination light, and the wavelength, intensity, and the like of the generated illumination light can be controlled. In addition, by the setting of the imaging parameter setting section 13, the imaging device 20 can perform imaging signal output in various modes, for example, frame rate, number of pixels, pixel addition operation, change of readout region, sensitivity switching, discrimination output of color signals, and the like can be controlled.

[0059] In addition, the imaging signal output from the imaging element 22 is sometimes referred to as RAW data, which is sometimes used as original data before image processing.

[0060] (Image processing section)

[0061] The image processing section 12 is supplied with the imaging image (moving image and still image) taken in from the imaging device 20, and performs prescribed signal processing such as color adjustment processing, matrix conversion processing, noise removal processing, synthesis of images, adaptive processing, and other various signal processing on the taken imaging image. The image processing parameter setting section 14 sets the processing parameters of the image processing in the image processing section 12.

[0062] By the image processing of the image processing section 12, the visual recognition of the imaging image can be improved. In addition, by the image processing of the image processing section 12, the analysis characteristics of the image analysis processing for the imaging image can also be improved. In addition, the image processing section 12 can also convert the so-called RAW data from the imaging element into data of a specific form.

[0063] The display control section 15 is supplied with the imaging image on which the signal processing has been performed by the image processing section 12. The display control section 15 converts the imaging image taken in by the imaging device 20 into an observation image that can be processed in the monitor 5 and outputs it.

[0064] In addition, an operation section 16 is provided in the video processor 3. The operation section 16 can be constituted by various buttons, dials, touch panels, for example, and receives user operations and outputs an operation signal based on the user operations to the control section 11. In addition, the operation section 16 can be constituted so as to correspond to a hands-free, accept gesture input, voice input, and the like to generate an operation signal. The control section 11 can control each section in accordance with the operation signal.

[0065] In the present embodiment, the setting by the imaging parameter setting section 13 and the image processing parameter setting section 14 is controlled by the navigation device 30.

[0066] (Navigation device)

[0067] The navigation device 30 has a control section 31, an image analysis section 32, a retrieval condition storage section 33, a determination section 34, a retrieval condition designation section 35, and a support information generation section 36. The control section 31 can be constituted by a processor using a CPU, an FPGA, or the like, can control each section in accordance with a program stored in a memory (not shown), and can realize part or all of the functions by an electronic circuit of hardware. Also, the entire navigation device 30 or each constituent section of the navigation device 30 can be constituted by a processor using a CPU, an FPGA, or the like, can control each section in accordance with a program stored in a memory (not shown), and can realize part or all of the functions by an electronic circuit of hardware.

[0068] The retrieval condition storage section 33 stores retrieval conditions for deciding the setting contents of the imaging parameter setting section 13 and the image processing parameter setting section 14 of the video processor 3. For example, in the retrieval condition storage section 33, information (hereinafter referred to as light source setting information) related to the kind of illumination light emitted by the illumination section 23 of the light source device 4, information (hereinafter referred to as optical system setting information) related to the driving of the optical system 21, and information (hereinafter referred to as imaging setting information) related to the driving of the imaging element 22 can be stored. Further, the retrieval condition storage section 33 can store information (hereinafter referred to as image processing setting information) for deciding the image processing contents of the image processing section 12.

[0069] Further, in the retrieval condition storage section 33, these light source setting information, optical system setting information, imaging setting information, and image processing setting information (hereinafter, these will also be referred to as retrieval condition setting information) can be stored as a group. For example, retrieval condition setting information in an initial state, retrieval condition setting information in a prescribed observation mode, retrieval condition setting information corresponding to a prescribed analysis condition, and the like can be stored in advance.

[0070] The retrieval condition designation section 35 is controlled by the control section 31 to designate the retrieval condition setting information read from the retrieval condition storage section 33 to the imaging parameter setting section 13 and the image processing parameter setting section 14. In accordance with the designation of the retrieval condition designation section 35, control related to the observation mode in the endoscope 2, the kind of illumination light, imaging, processing of image processing in the video processor 3, and the like are performed. Further, the retrieval condition designation section 35 can be constituted so as to also generate and output to the video processor 3, by the control of the control section 31, retrieval condition setting information that is not stored in the retrieval condition storage section 33. Further, the retrieval condition storage section 33 can be omitted, and the retrieval condition designation section 35 can generate retrieval condition setting information as needed.

[0071] For example, light source setting information is specified by the acquisition condition specifying section 35, whereby the light source device 4 specifies which of the illumination light using WLI light, NBI light, DRI light, AFI light, and the like is used.

[0072] (WLI light, NBI light, DRI light, AFI light)

[0073] Here, reference is made to Figure 4 , Figure 5 The WLI light, NBI light, DRI, AFI light employed in the present embodiment are described. Figure 4 is a diagram for explaining the relationship between the WLI light and NBI light irradiated from the endoscope of the first embodiment and the blood vessels in the mucosa of the subject, Figure 5 is a diagram for explaining the relationship between the DRI light and the blood vessels in the mucosa of the subject.

[0074] By irradiating the WLI light (white light) to the mucosa surface, the human (doctor) can reproduce the blood vessels and the like present in the mucosa on the monitor by the natural color. On the other hand, in the case of using the WLI light (white light), as for the capillary blood vessels and the mucosal fine pattern of the mucosa surface layer portion, it is not necessarily possible to clearly reproduce the same for the human's recognition.

[0075] In the present embodiment, it is also possible to employ NBI (Narrow Band Imaging) light based on two wavelengths (blue light: 390 to 445 nm (415 nm in the present embodiment) / green light: 530 to 550 nm (540 nm in the present embodiment)) of narrow banding that are easily absorbed by hemoglobin in blood to observe the mucosa.

[0076] By irradiating this NBI light, as shown in Figure 4 , the blue light (415 nm) in the NBI light is absorbed in the capillary blood vessels 64 in the mucosa surface layer portion 61, as a result of which the capillary blood vessels 64 are clearly delineated, and, similarly, the blood vessels 65 in the layer 62 in the slightly deeper portion than the surface layer portion are delineated by the green light (540 nm). Thus, the capillary blood vessels and the mucosal fine pattern in the mucosa surface layer portion 61 are emphasized.

[0077] Further, as described above, in the present embodiment, it is also possible to perform special light observation by setting the wavelength of the NBI light as narrow band light to different other wavelengths.

[0078] On the other hand, in the present embodiment, it is also possible to employ DRI (Dual Red Imaging) light based on light of two long wavelengths (600 nm / 630 nm) of narrow frequency banding, to irradiate the DRI light to the subject, whereby the submucosal layer is emphasized from the mucosa deep layer that is difficult to visually recognize in normal light observation (Figure 5 blood flow information of the blood vessel 66 of the layer 63 in the region R.

[0079] Further, in the present embodiment, it is also possible to perform irradiation of a subject with prescribed excitation light for fluorescence observation, and to emphasize a tumor lesion and normal mucosa in different color tones, so-called fluorescence observation AFI (Auto Fluorescence Imaging).

[0080] In addition, not only such light source control is possible, but also control of the optical system 21 and the imaging element 22 according to the acquisition condition setting information is possible, and for example, it is also possible to change the exposure time of the imaging element or the like according to the setting of the acquisition condition. By exposure control, it is also possible to eliminate the influence of saturation or low luminance noise.

[0081] (Example of acquisition method of multiple images)

[0082] In the present embodiment, the acquisition condition designating section 35 can also mix acquisition condition setting information (hereinafter, referred to as display use acquisition condition setting information) which generates a prescribed display use acquisition condition, which is a condition for acquiring an image for display with excellent visual recognition, and acquisition condition setting information (hereinafter, referred to as analysis use acquisition condition setting information) which generates a prescribed analysis use acquisition condition, which is a condition for acquiring an image for analysis with excellent analysis for image analysis processing. For example, it is also possible to output only the display use acquisition condition setting information in a prescribed first period, and to mix and output the display use acquisition condition setting information and the analysis use acquisition condition setting information in a prescribed second period.

[0083] When the display use acquisition condition setting information is supplied to the video processor 3, the video processor 3 controls at least one of the light source device 4 (the illumination section 23), the optical system 21, the imaging element 22, and the image processing section 12 based on the display use acquisition condition setting information, so that an image for display with excellent visual recognition can be output. In addition, when the display use acquisition condition setting information and the analysis use acquisition condition setting information are mixed and input to the video processor 3, the video processor 3 controls at least one of the light source device 4 (the illumination section 23), the optical system 21, the imaging element 22, and the image processing section 12 based on the display use acquisition condition setting information and the analysis use acquisition condition setting information, so that an image for display with excellent visual recognition and an image with excellent analysis can be output.

[0084] Thus, the display acquisition condition is a condition for imaging and lighting that is set so that the wavelength of the light source approaches natural light (sunlight), the image processing of the imaging result is focused on visual recognition, and the frame rate and the like are also focused on continuity, so that it feels natural when the physician searches for the affected part under natural light or observes the affected part (mainly the surface) by irradiating it. In addition, the analysis acquisition condition is set to a condition for imaging and lighting in which the wavelength of the light source reaches not only the surface of the affected part but also the inside of the affected part, the image processing of the imaging result is focused on the effective information amount for analysis, and the frame rate and the like are focused on analysis rather than continuity, so that it is easy to determine the characteristics of a specific pattern or image.

[0085] Figures 6 to 8 are explanatory diagrams showing an example of an imaging image acquired by the video processor 3, an image output to the monitor 5, or an image supplied to the image analysis section 32 in a case where the display acquisition condition setting information and the analysis acquisition condition setting information are mixed and input to the video processor 3.

[0086] Figure 6 indicates a series of frames obtained by imaging with the imaging element 22. In Figure 6 WLI <Raw> indicates a high frame rate (e.g., 30 FPS or more) imaging image obtained by imaging using high light amount WLI light as illumination light. In addition, Figure 6 NBI <Raw> of FIG. 8 indicates a low frame rate (e.g., about 1 FPS) imaging image obtained by imaging using NBI light as illumination light (narrow band light observation). In addition, low light amount WLI <Raw> indicates a low frame rate (e.g., 1 FPS) imaging image obtained by imaging using low light amount WLI light as illumination light.

[0087] The WLI <Raw> frames are used to generate a display image. The NBI <raw>Frame and low light WLI <raw>The frames are used to generate the analysis images. Note that the WLI <raw>The frame can be used to generate an analysis image. In addition, although not described in the Figure 6 , as an image analysis image, a DRI <Raw> frame obtained by imaging using DRI light as illumination light can be acquired at a low frame rate (around 1 FPS), or an imaging image based on AFI observation excitation light can be acquired. In this way, by changing the conditions for acquiring the results of imaging at the same position of the object, useful information acquisition can be performed with a simple structure without complex operations.

[0088] For example, it is expected that a display image with excellent visual recognition can be obtained from a high frame rate (for example, 30 FPS or more) imaging image obtained by imaging using high light amount WLI light as illumination light, and the setting conditions of the light source, the setting conditions of the optical system, the imaging setting conditions, and the like for obtaining such an image become display acquisition conditions. The setting conditions of the light source, the setting conditions of the optical system, the imaging setting conditions, and the like for obtaining such an image become display acquisition conditions.

[0089] In addition, for example, it is expected that an image obtained by special light observation from an NBI <Raw> frame or the like has excellent analysis for image analysis, and the setting conditions of the light source, the setting conditions of the optical system, the imaging setting conditions, and the like for obtaining such an image become analysis acquisition conditions. In addition, the conditions of the image processing for obtaining an image with excellent visual recognition are display acquisition conditions, and the conditions of the image processing for obtaining an image with excellent analysis are analysis acquisition conditions.

[0090] Figure 9 is a chart showing specific examples of display acquisition conditions and analysis acquisition conditions with respect to image processing, and is used to explain one example of conditions that can be achieved by image processing by the image processing section 12. As shown in Figure 9 , the video processor 3, for example, with respect to gamma processing in image processing performed on an imaging signal, performs gamma processing that matches the characteristics of the human eye in accordance with display acquisition conditions. In addition, the video processor 3 does not perform gamma processing that is not required for analysis processing in accordance with analysis acquisition conditions. Similarly, with respect to other image processing such as white balance, color correction, noise reduction, image emphasis, and the like, the video processor 3 distinguishes between image processing for obtaining a display image and image processing for obtaining an analysis image, as shown in Figure 9 .

[0091] The image processing section 12 of the video processor 3 acquires a display WLI image with excellent visual recognition from the imaging image of the WLI <Raw> by signal processing in accordance with display acquisition condition setting information. The navigation device 30 outputs the WLI image with excellent visual recognition from the imaging image of the video processor 3 as a display image to the monitor 5.

[0092] Figure 7 In this case, the control section 31 of the navigation device 30 extracts the WLI image from the image output from the video processor 3 and outputs to the monitor 5. In Figure 7 In the example of Fig. 6, in the series of frames, the WLI image is extracted by Figure 6 processing the WLI image in the series of frames, and the WLI image is output to the monitor 5. <raw>The WLI image obtained by the image processing of the endoscope 2 is supplied to the monitor 5. In addition, the imaging is performed in a manner such that the frame rate of the WLI image supplied to the monitor 5 is, for example, 30 FPS or more. The image is observed for visual recognition confirmation, and thus the image frames are not lost per unit time as much as possible, but for example, the image is not limited to this as long as the image does not change.

[0093] Thus, the imaging image obtained by the imaging device 20 of the endoscope 2 is displayed on the display screen of the monitor 5. The image displayed on the monitor 5 is a WLI image excellent in visual recognition, and the operator can confirm the image of the field of view of the imaging device 20 as an image easily observed on the display screen of the monitor 5.

[0094] For the WLI image excellent in visual recognition, information useful for image analysis for navigation can be lost by the signal processing in the image processing section 12. Thus, as shown in FIG. 1, the video processor 3 stops the more image processing for the analysis image in accordance with the analysis acquisition condition, and adds information useful for image analysis. Thus, by the analysis acquisition condition setting information, the analysis image useful for image analysis can be output. Figure 9

[0095] For example, for the capillary vessels and the mucosal fine pattern of the mucosal surface layer portion described above, it is difficult to discriminate by the WLI image, and it can be discriminated relatively easily by image analysis using an NBI image obtained by imaging using NBI light or the like. Thus, the control section 31, for example, supplies all the output images of the video processor 3 including the NBI image to the image analysis section 32 to perform image analysis. Figure 8 The image supplied to the image analysis section 32 is shown. In addition, the control section 31 can supply only the images other than the WLI image in the output images of the video processor 3 to the image analysis section 32.

[0096] The image analysis section 32 performs various image analyses in order to assist the operator. The image analysis section 32 performs image analysis on the imaging image input from the video processor 3, and obtains an image analysis result. The image analysis section 32, for example, obtains an image analysis result on the advancing direction of the insertion portion of the endoscope 2, or obtains an image analysis result on the discrimination result of the lesion portion. The image analysis result of the image analysis section 32 is supplied to the assistance information generation section 36.

[0097] ​The support information generation section 36 generates support information based on the image analysis result of the image analysis section 32. For example, the support information generation section 36 generates support information indicating the direction in which the insertion section should be inserted, in a case where it is determined from the image analysis result that the insertion section should be inserted in a direction. Also, for example, in a case where it is determined from the image analysis result that the differential diagnosis result of the lesion section is obtained, the support information generation section 36 generates support information for prompting the operator of the differential diagnosis result. The support information generation section 36 can also generate support display data such as an image (support image) for display on the monitor 5, text (support text), and the like as support information. In addition, the support information generation section 36 can also generate sound data for sound output from a speaker not illustrated as support information.

[0098] (Change of acquisition conditions)

[0099] Further, in the present embodiment, the navigation device 30 can change the image acquisition conditions based on the image analysis result including the characteristics of the image used for analysis, various information obtained from the image. The determination section 34 determines whether or not the acquisition conditions of the image should be changed, how the acquisition conditions of the image should be changed. For example, the determination section 34 instructs the acquisition condition specification section 35 to change the acquisition conditions required for desired image analysis in a case where sufficient analysis result cannot be obtained from the image analysis result or in a case where it is determined that more detailed image analysis is required.

[0100] For example, the determination section 34 can also decide the change to a specific acquisition condition based on a specific criterion. For example, the determination section 34 can also decide the acquisition conditions to be changed by comparing the value included in the image analysis result such as contrast information, histogram information, and the like obtained from the image used for analysis with a prescribed criterion value. In addition, the determination section 34 can also determine whether or not the image used in the analysis includes a specific image feature, pattern, and the like by pattern matching or the like, and decide the acquisition conditions to be set based on the result thereof.

[0101] In addition, the determination section 34 can also instruct the acquisition condition specification section 35 to change the acquisition conditions required for desired analysis result not only according to the image analysis result, but also according to the observation mode, the content of the surgery, and the like.

[0102] (Action)

[0103] Next, the operation of the embodiment thus configured will be described with reference to Figures 10 to 14 The operation of the embodiment thus configured will be described with reference to Figure 10 is a flowchart for explaining the operation of the first embodiment, Figure 11 is an explanatory diagram for explaining the image acquired in a specific use case.

[0104] Figure 11 The example of Figure 3 The same application scenario is shown, illustrating the insertion of endoscope 2 (rigid endoscope) into a body cavity to observe internal tissues and organs.

[0105] For example, immediately after power is switched on, the acquisition condition specification unit 35 of the navigation device 30 reads the display acquisition condition setting information from the initial settings of the acquisition condition storage unit 33 and provides it to the video processor 3. The display acquisition condition setting information can set the acquisition conditions for acquiring the display image, and the camera parameter setting unit 13 in the control unit 11 of the video processor 3 sets the parameters of the light source device 4, the optical system 21, and the camera element 22 according to the display acquisition condition setting information.

[0106] Therefore, in Figure 10 In step S1, a routine observation is performed. Additionally, Figure 10 The acquisition condition I1 is, for example, an acquisition condition corresponding to the display acquisition condition setting information in the initial setting, and is a predetermined condition. According to the acquisition condition I1, for example, the light source device 4 causes the illumination unit 23 to emit high-intensity WLI light, and the control unit 11 drives the imaging element 22 at a high frame rate (for example, 30 FPS or more), thereby outputting a WLI <Raw> image from the imaging device 20.

[0107] Furthermore, the image processing parameter setting unit 14 in the control unit 11 sets the image processing parameters of the image processing unit 12 based on the display acquisition condition setting information. Therefore, for example, as... Figure 9 As shown, the image processing unit 12 performs gamma processing, white balance processing, color correction matching the characteristics of the human eye, noise reduction processing, image enhancement processing, etc. on the image captured from the camera device 20, and generates a WLI image suitable for display.

[0108] The WLI image acquired by the image processing unit 12 is provided to the navigation device 30. The control unit 31 outputs the input WLI image as a display image to the monitor 5. In this way, a WLI image with excellent visual clarity is displayed on the display screen of the monitor 5. The surgeon can reliably observe the internal tissues, organs, etc. within the body cavity through the WLI image with good visual clarity displayed on the monitor 5.

[0109] exist Figure 10 In the example of FIG. 9, the control section 31 determines whether or not it is the timing at which the acquisition condition II should be changed to the acquisition condition I2 in step S2. It is not necessary to perform assistance by the navigation device 30 during the entire period from the start to the end of surgery or examination. If the processing amount of image analysis in the navigation device 30 is taken into consideration, it is considered preferable to perform assistance by the navigation device 30 only in a case where assistance is needed. Therefore, the control section 31 switches the timing of transition from the acquisition condition II based on the display acquisition condition setting information to the acquisition condition I2 including the analysis acquisition condition setting information, in a case where there is an instruction from the operator or in a case where it is determined that a predetermined medical scenario is reached. The acquisition condition I2 is a condition that is set in advance. In addition, the acquisition conditions II, I2 can be set to appropriate contents according to user settings.

[0110] The control section 31 causes the processing to proceed to step S3 when it is determined that the timing is reached in accordance with the operation of the operator, for example, and instructs the acquisition condition designating section 35 to transition to the acquisition condition I2. In addition, the control section 31 causes the processing to proceed to step S4 in a case where it is determined that the timing is not reached.

[0111] The acquisition condition designating section 35 reads the acquisition condition setting information including the display acquisition condition setting information and the analysis acquisition condition setting information and outputs it to the video processor 3 in step S3, thereby transitioning to the acquisition condition I2. That is, the acquisition condition I2 is a condition for acquiring not only a display image but also an analysis image by using the display acquisition condition setting information and the analysis acquisition condition setting information.

[0112] In this case, the light source device 4, the optical system 21, and the imaging element 22 are controlled by the imaging parameter setting section 13 and the image processing parameter setting section 14 to acquire a WLI <Raw> at a frame rate of 30 FPS or more, for example, and to acquire an image suitable for image analysis. For example, as shown in FIG. 10, the imaging device 20 repeatedly acquires a WLI <Raw> and an image suitable for image analysis. Figure 11 <raw>WLI <raw>, NBI <raw>WLI <raw>Low light level WLI <raw>WLI <raw>frames. In Figure 11 In the example, 4 frames of the series of 6 frames are WLI <Raw> frames acquired based on the display acquisition condition setting information, and 2 frames are NBI <Raw>, low-light WLI <Raw> frames acquired based on the analysis acquisition condition setting information.

[0113] The image processing parameter setting section 14 controls the image processing section 12 based on the display acquisition condition setting information and the analysis acquisition condition setting information. Thereby, the image processing section 12 performs signal processing based on the display acquisition condition setting information with respect to the WLI <Raw> frames, and acquires WLI images. In addition, the image processing section 12 does not perform, for example, display signal processing based on the analysis acquisition condition setting information with respect to the NBI <Raw>, low-light WLI <Raw> frames. In addition, the image processing section 12 converts the NBI <Raw> frames and the low-light WLI <Raw> frames into NBI images and low-light WLI images, respectively. The image processing section 12 outputs these images to the navigation device 30.

[0114] As shown in Fig. 6, the control section 31 of the navigation device 30 outputs the WLI images as display images to the monitor 5, and outputs the NBI images and the low-light WLI images to the image analysis section 32. In addition, the WLI images are also supplied to the image analysis section 32. The image analysis section 32 performs image analysis using the WLI images, the NBI images, and the low-light WLI images, and obtains a prescribed analysis result. For example, in the case of performing diagnosis support, the image analysis section 32 obtains a desired analysis result such as the presence or absence of a candidate for a lesion portion, and the identification of a lesion portion. Figure 11 The image analyzed in the image analysis section 32 contains an image such as an NBI image suitable for analysis, which is obtained by special light observation, and in addition, image processing without loss of accompanying information is not performed, and thus has an information amount sufficient for image analysis, and a high-precision analysis result can be obtained in the image analysis section 32. The information amount is an amount of information possessed by each of pixels from which something is derived from an image, or an amount of information that significantly represents a change in arrangement of pixels and the like, and is an amount of information required to identify a feature of an object possessed by each of the images analyzed, such as contrast, spatial frequency, gradation characteristics, color change, recognition of differences in wavelength, and the like.

[0115]

[0116] ​In the present embodiment, the processing of step S4 and the determination of step S5 are performed after step S2 or S3, but in the case where the determination of step S5 is "No", the processing is transferred to step S7. In step S7, it is determined whether or not support display is required. The control section 31 determines that support display is required and causes the support information generation section 36 to generate support information, for example, in the case where a lesion candidate is found based on the image analysis result of the image analysis section 32. The support information generation section 36 generates support information based on the analysis result of the image analysis section 32.

[0117] The support information generation section 36 can also generate, for example, display data for displaying a mark (support display) indicating the position of a lesion candidate on a display image displayed on the display screen of the monitor 5, as support information in the case where a lesion candidate is found. The control section 31 supplies the display data generated by the support information generation section 36 to the monitor 5. In this way, a mark indicating the position of a lesion candidate is displayed on the display image (observation image of the endoscope 2) displayed on the monitor 5 (step S8).

[0118] In this way, in the present embodiment, by displaying a WLI image excellent in visual recognition on the monitor 5, it is easy to confirm a lesion portion or the like, and by performing image analysis for support using an NBI image or the like suitable for image analysis, it is possible to obtain a highly accurate analysis result, and it is possible to perform support extremely effective for a surgeon. In addition, the analysis image is acquired only in the case where support is required, it is possible to perform display at high quality without unnecessarily reducing the frame rate of the display image, and it is possible to prevent the processing amount of image analysis from unnecessarily increasing. In addition, the display image and the analysis image are acquired from the imaging signal of the imaging device 20, it is not necessary to arrange a plurality of imaging devices at the distal end portion of the endoscope insertion portion, it does not lead to enlargement of the distal end portion, and in addition, it is not necessary to use high-performance hardware due to a significant increase in the information processing amount of the processing.

[0119] (Adaptively changing acquisition conditions)

[0120] Further, in the present embodiment, by setting the acquisition condition I3 which changes according to the situation, it is possible to perform analysis at higher accuracy. The determination section 34 determines, in step S4, based on the analysis image and the image analysis result of the image analysis section 32, whether or not the acquisition condition should be changed and the acquisition condition in the case of change in order to acquire an analysis result at higher accuracy. The determination section 34 determines whether or not an analysis result at higher accuracy can be obtained (step S5), and in the case where it can be obtained, causes the acquisition condition designation section 35 to set the acquisition condition I3 for this (step S6). In addition, the determination section 34 transfers the processing to step S7 in the case where it is determined that an analysis result at higher accuracy cannot be obtained.

[0121] In step S6, the acquisition condition specifying section 35 reads out the display-use acquisition condition setting information and the analysis-use acquisition condition setting information from the acquisition condition storage section 33 in accordance with the determination result of the determination section 34, and outputs them to the imaging parameter setting section 13 and the image processing parameter setting section 14 as the acquisition condition I3. That is, the acquisition condition I3 which changes adaptively in accordance with the output of the video processor 3 is fed back to the video processor 3. Further, the acquisition condition specifying section 35 can generate the display-use acquisition condition setting information and the analysis-use acquisition condition setting information and output them in accordance with the determination result of the determination section 34, instead of the information stored in the acquisition condition storage section 33.

[0122] Figure 12 is a chart for explaining an example of the acquisition condition I3 based on the determination of the determination section 34. Figure 12 The condition column of indicates information obtained from the analysis result of the image analysis section 32, and the feedback content indicates the acquisition condition I3 specified by the acquisition condition specifying section 35 based on the determination result of the determination section 34.

[0123] Even in the case where the display-use image acquired based on the acquisition condition I1 is output, the image analysis section 32 can perform image analysis using the display-use image (WLI image). The determination section 34 determines the analysis result of the WLI image by the image analysis section 32 in the case where the process is shifted from step S2 to step S4. For example, blood vessel information relating to the mucous membrane is obtained from the analysis result of the WLI image by the image analysis section 32. In the case where the determination section 34 determines that the blood vessels of the mucous membrane surface layer portion appear to be more, the acquisition condition for acquiring an analysis-use image such as an NBI image using illumination light of a long wavelength is set as the acquisition condition I3.

[0124] The display-use image using illumination light of a short wavelength (short-wavelength image) makes it easy to confirm the fine blood vessels of the tissue surface layer. Therefore, in the case where the fine blood vessels appear to be more, it is determined from the blood vessel information of the mucous membrane surface layer portion that there is a possibility of a certain malignant tumor being latent, and the acquisition condition I3 for acquiring an NBI image or the like as an analysis-use image is set in order to more clearly grasp the fine blood vessel structure of the mucous membrane surface layer portion.

[0125] Further, for example, the determination section 34, in the case where the analysis-use image (WLI image, NBI image, or the like) is acquired based on the acquisition condition I2, sets the acquisition information I3 for acquiring a DRI image in order to obtain blood vessel information of a deeper portion of the mucous membrane (for example, blood vessel information from a deeper layer of the mucous membrane to the submucosal layer) when the information indicating the fine blood vessels of the mucous membrane surface layer portion is less based on the analysis result of the analysis-use image, the DRI image being obtained by observation based on DRI special light of a short wavelength.

[0126] Further, for example, the determination section 34 increases or decreases the frame rate of the display image in accordance with the size of the movement of the image of the subject affected part in the image analyzed by the image analysis section 32, and sets the acquisition information I3 for increasing or decreasing the kind of the image of the analysis image.

[0127] Further, for example, the determination section 34 sets the acquisition information I3 for changing the brightness of the analysis image in accordance with the brightness information of the periphery of the subject affected part in the image analyzed by the image analysis section 32. For example, in a case where the image of the periphery of the subject affected part is dark, the acquisition information I3 for making the brightness of the analysis image brighter is set, and in a case where the image of the periphery of the subject affected part is bright, the acquisition information I3 for making the brightness of the analysis image darker is set. Further, such control can be performed by appropriately correcting the light amount of the light source or the exposure time of the imaging element, or the like. In this way, the support display in step S8 is performed using the image acquired on the basis of the acquisition condition I3.

[0128] Further, in the flowchart of Figure 10 , an example in which the change of the acquisition condition I3 is performed only once is shown, but the determination section 34 can repeatedly change the setting content of the acquisition condition I3 as needed.

[0129] Further, in the explanation of Figure 10 , an example in which only the display acquisition condition setting information for acquiring the display image in a prescribed first period is output, and the display acquisition condition setting information and the analysis acquisition condition setting information are mixed and present in a prescribed second period corresponding to the operation of the operator or the like, and the display image and the analysis image are acquired, is explained. For example, the halfway of the movement of the insertion part of the endoscope 2 by the operator to the observation target part can be set as the first period, and the timing at which the detection of the lesion candidate is started after the distal end part of the endoscope 2 reaches the observation target part can be set as the second period.

[0130] Furthermore, an example is shown where acquisition condition I1 is generated initially using only the display acquisition condition setting information for acquiring the image to be displayed. However, both the display acquisition condition and the analysis acquisition condition can be set continuously after power is turned on. For example, acquisition condition I1 can be set to acquire one, for example, NBI <Raw> frame for every predetermined number of WLI <Raw> frames, using the WLI image based on WLI <Raw> as the display image, and using the WLI image and the NBI image based on NBI <Raw> as the analysis image. In this case, high-quality images can be displayed using WLI images with a higher frame rate, and the analysis required for support can be performed while significantly reducing the processing load on the navigation device 30. Moreover, by setting acquisition condition I2, which increases the acquisition ratio of the analysis image based on the analysis results or according to the surgeon's operation, high-precision analysis corresponding to the support requested by the surgeon can be performed.

[0131] In other words, such image acquisition control does not require special attention from the surgeon and is performed in the same manner as background processing. Therefore, for example, the surgeon does not need to focus on determining whether a special observation based on NBI light or the like is required, enabling accurate navigation and providing effective support to the surgeon in an instant.

[0132] Figure 13 This is an explanatory diagram used to illustrate the support display. Figure 13 This indicates that the endoscope 2 is inserted into the body cavity P to observe the internal tissues and organs. The arrow indicates the illumination light emitted from the front end of the rigid endoscope and its reflected light. The reflected light is incident on the camera device 20 of the endoscope 2.

[0133] (Image for display based on condition I1)

[0134] The display image (Im1) obtained according to acquisition condition I1 is obtained by observation under white light, which is close to the result of observation under natural light that people are accustomed to. That is, in this example, acquisition condition I1 is a condition used to obtain a display image that emphasizes visual recognizability. However, in the image captured based on this acquisition condition I1, the reflective component from the surface of the object prevails, and the information inside the tissue is relatively reduced. Therefore, even if there is an anomaly in the part surrounded by the dotted line, it is sometimes difficult to detect the anomaly.

[0135] (Image for analysis based on condition I2)

[0136] The image for analysis based on acquisition condition I2 is an image (Im2) acquired according to imaging conditions including observation light conditions that allow observation of the tissue interior, as well as image processing conditions. Therefore, it is possible to detect abnormalities within the tissue interior that are not visible on the surface of the tissue. Figure 13 In the image, the shaded area represents the lesion detected. For example... Figure 4 and Figure 5 As explained in

[0137] (Image for analysis based on acquisition condition I3)

[0138] The image for analysis based on acquisition condition I3 is an image (Im3) obtained from an acquisition condition changed from acquisition condition I2, and is used to obtain a more highly accurate analysis result. In this case, as shown by the hatching of Figure 13 , the shape of the lesion portion is clearer than the image Im2. As a result, the analysis result using the image (Im3) is more highly accurate than the analysis result using the image (Im2).

[0139] (Support display)

[0140] The support information generating section 36 generates support information based on the more highly accurate analysis result. In the example of Figure 13 , the support information is display data indicating the shape of the lesion portion. The control section 31 superimposes display based on the support information on the display image (Im1) and displays it. Further, the support information generating section 36 can also generate display data for displaying a text such as "lesion portion found" in the vicinity of the position of the dotted line portion as support information. In this way, the observer can confirm display indicating the presence of the lesion portion detected in the image analysis section 32 on the display image natural to the human eye, and can also take measures such as re-examining the portion by other methods.

[0141] Further, the support display method by the support information generating section 36 can be variously improved and customized. For example, in Figure 13 , an example of support display based on the image for analysis based on acquisition condition I3 is explained, but support display can also be performed based on the image for analysis based on acquisition condition I2. In addition, the support information generating section 36 can directly display the image for analysis, or can display a composite image based on the analysis result as support display.

[0142] (Priority order of acquisition condition determination)

[0143] In the case where the acquisition condition is changed according to the situation, the determination section 34 sometimes needs to consider multiple requests (acquisition conditions). Figure 14 is a chart for explaining the priority order for such multiple requests.

[0144] For example, suppose that the WLI or NBI image used in the analysis by the image analysis unit 32 has few detected microvessels, a dark surrounding image, and a large amount of motion in the image. In this case, if possible, the determination unit 34 does not reduce the frame rate of the displayed image, and generates acquisition condition I3 in order to obtain a bright image using a long-wavelength DRI image or the like for analysis.

[0145] However, sometimes it is impossible to satisfy all requests. Therefore, the determination unit 34 assigns a priority order to each request (condition) and determines the acquisition condition I3. For example, as priority order 1, the determination unit 34 sets the condition of not reducing the frame rate of the displayed image. In addition, as priority order 2, the determination unit 34 sets the condition of acquiring an image for analysis, such as a DRI image using a long wavelength. In addition, as priority order 3, the determination unit 34 sets the condition of acquiring a bright image.

[0146] The determination unit 34 considers this priority order and instructs the acquisition condition specification unit 35 to generate acquisition condition I3. For example, the acquisition condition specification unit 35 generates display acquisition condition setting information for maintaining the frame rate of the WLI<Raw> frame used as the display image at 30 FPS or higher. Additionally, for example, the acquisition condition specification unit 35 generates analysis acquisition condition setting information for acquiring a DRI image at 2 FPS based on a DRI<Raw> frame used to generate the DRI image as the analysis image. Furthermore, for example, the acquisition condition specification unit 35 considers the limitation of the maximum frame rate that can be captured and does not respond to the request for priority order 3.

[0147] In this way, because the acquisition conditions generated by prioritizing requests to the video processor 3 are fed back, the video processor 3 can efficiently acquire images useful for both display and analysis. Furthermore, it can reliably acquire images corresponding to the acquisition conditions in various endoscopes and video processors with different performance and functions.

[0148] Thus, in this embodiment, images with excellent visual recognizability for display and images with excellent analytical capabilities can be obtained, enabling highly effective support for various operations while maintaining excellent visual recognizability. Furthermore, the image acquisition conditions can be adaptively varied, allowing for appropriate support based on the situation.

[0149] In addition, Figure 1 The example shown depicts a separate configuration of the video processor 3 and the navigation device 30, but it is also clearly possible for the navigation device 30 to be integrated into the video processor 3. Furthermore, as an endoscope system, it is not limited to laparoscopic surgery systems, but can also be applied to endoscope systems using conventional flexible endoscopes.

[0150] Further, the analysis by the image analysis section 32, the determination by the determination section 34, the generation of the acquisition condition setting information by the acquisition condition designation section 35, and the like in the navigation device 30 can also be implemented by an AI (Artificial Intelligence) device.

[0151] (Second Embodiment)

[0152] Figure 15 is a flowchart showing the flow of actions adopted in the second embodiment. The hardware structure in this embodiment is the same as that in the first embodiment, and the explanation thereof is omitted. Figure 1

[0153] In the first embodiment, an example in which the acquisition condition I3 is adaptively set in a case where a higher-precision analysis result is obtained than the acquisition conditions I1 and I2 was explained. In this embodiment, which of these analysis results is higher-precision is determined in a case where the acquisition condition I1 is changed to the acquisition condition I2. Further, that the analysis result is higher-precision means that, as explained above, a more suitable analysis result is obtained for the support, for example, including a case where the amount of information obtained from the image is increased, and the like. In this embodiment, in a case where the determination result is that a higher-precision analysis result can be obtained by the condition change, further change of the same kind as the change content of the acquisition condition is performed, and in a case where this is not true, change of a different kind from the change content of the acquisition condition is performed, whereby optimal setting of the acquisition condition can be performed.

[0154] Further change of the same kind as the change content of the acquisition condition means, for example, change in which the wavelength of the NBI light is changed in a case where the acquisition condition I1 for obtaining a normal light observation image is changed to the acquisition condition I2 for obtaining an NBI image. Further, change of a different kind from the change content of the acquisition condition means, for example, change in which the acquisition condition for obtaining a DRI image is changed instead of the NBI image in a case where the acquisition condition I1 for obtaining a normal light observation image is changed to the acquisition condition I2 for obtaining an NBI image.

[0155] For example, the acquisition condition I3 in a case where a higher-precision analysis result is obtained and the acquisition condition I3 in a case where the precision of the analysis result is reduced can also be registered in advance in the acquisition condition storage section 33 with respect to the combination of the acquisition conditions I1 and I2. In this case, the determination section 34 can also instruct the acquisition condition designation section 35 to read out which of the storage contents of the acquisition condition storage section 33 according to the determination result of whether the analysis result is higher-precision or reduced-precision.

[0156] In step S11 of the flowchart of Fig. 11, image pickup is performed based on the predetermined acquisition condition I1. For example, as shown in Fig. 12, the image analysis section 32 performs analysis of the image obtained by the image pickup, and the determination section 34 determines whether the analysis result is higher-precision or reduced-precision. Figure 15 Figure 13 ​​As shown, the examination in the body cavity of the subject is started, and under the control of the control section 11 of the video processor 3, the image is acquired by the endoscope 2, and the captured image is supplied to the monitor 5 via the navigation device 30. As the acquisition condition I1, for example, the WLI <Raw> frame is acquired using the display-use acquisition condition setting information. The image processing section 12 outputs the WLI image based on the WLI <Raw> frame to the navigation device 30, and the control section 31 supplies the WLI image to the monitor 5 and displays it on the screen. In this way, the WLI image excellent in visual recognition is displayed on the display screen of the monitor 5.

[0157] In addition, the video processor 3 temporarily records the WLI image acquired based on the acquisition condition I1 as the captured image Im1 to a recording device not shown (step S12). In addition, the image analysis section 32 of the navigation device 30 obtains an analysis result by image analysis of the WLI image acquired based on the acquisition condition I1.

[0158] The control section 31 determines whether there is a change instruction of the acquisition condition in step S13. As with the first embodiment, for example, the change instruction of the acquisition condition can be generated according to an instruction of the operator, and in addition, the determination section 34 can also generate the change instruction of the acquisition condition according to the analysis result of the image analysis section 32.

[0159] When the change instruction of the acquisition condition is generated, the control section 31 causes the acquisition condition specifying section 35 to generate a predetermined acquisition condition I2. The acquisition condition specifying section 35 can also read out the information of the acquisition condition I2 from the acquisition condition storage section 33. Now, the acquisition condition I2 is a condition for acquiring the WLI image of a prescribed frame rate or more and, for example, the NBI image or the like. Thus, for example, as shown in FIG. 1 of JP 2008- 271 1 1 A, the acquisition image including the WLI <Raw> frame and the NBI <Raw> frame is acquired by the endoscope 2 (step S14). The image processing section 12 generates the WLI image and the NBI image based on the captured image from the imaging device 20 and outputs them to the navigation device 30. Figure 6

[0160] The image analysis section 32 obtains an analysis result by image analysis of the WLI image and the NBI image acquired based on the acquisition condition I2. The support information generating section 36 generates support information based on the analysis result. In addition, the video processor 3 temporarily records the WLI image and the NBI image acquired based on the acquisition condition I2 as the captured image Im2 to a recording device not shown (step S15).

[0161] The determination section 34 determines whether the images based on the acquisition conditions I1, I2 are acquired for the same observation site in step S16. For example, the determination section 34 can determine whether it is the image based on the same observation site according to the analysis result of the image analysis section 32.

[0162] ​The determination section 34 determines whether the amount of information (hereinafter, the part of the amount of information is meant to indicate the amount of information of the features of the subject matter included in the image for some kind of support, assistance) is increased in the case where it is determined that the images based on the acquisition conditions I1, I2 are images for the same observation site in the next step S17. That is, the determination section 34 compares how much the amount of information of the image Im1 based on the acquisition condition I1 obtained by irradiating the WLI light to a certain region of the subject and the amount of information of the image Im2 based on the acquisition condition I2 obtained by irradiating the WLI light and the NBI light to the same region. The determination section 34 determines the image in which the amount of information (the amount of information required to obtain effective support) is relatively large between the amount of information of the image Im1 and the amount of information of the image Im2 (the amount of information required to obtain effective support).

[0163] The determination section 34 determines that a more effective image can be obtained by the same kind of acquisition condition in the case where the amount of information of the image based on the acquisition condition I2 is increased than the image based on the acquisition condition I1, and instructs the setting of the same kind of acquisition condition I3 to the acquisition condition designating section 35 in step S18. In addition, in the drawing, the part of the image condition written as the same kind of change content can not be further changed in the image acquisition, processing, and the like in the case where the image having sufficient amount of information is obtained.

[0164] The acquisition condition designating section 35 changes to the information for obtaining the image based on the NBI light whose wavelength band is different from the wavelength band designated by the acquisition condition I2, for example, as the acquisition condition I3 which is the same kind as the acquisition condition I2. In this case, for example, the acquisition image including the WLI <Raw> frame of the prescribed frame rate or more and the NBI <Raw> frame based on the NBI light of the different wavelength from the last time is obtained by the endoscope 2. The image processing section 12 generates the WLI image and the NBI image based on the captured image from the imaging device 20 and outputs them to the navigation device 30.

[0165] The image analysis section 32 obtains the analysis result by performing the image analysis on the WLI image and the NBI image obtained based on the acquisition condition I2. The support information generating section 36 generates the support information based on the analysis result. In addition, the video processor 3 temporarily records the WLI image and the NBI image obtained based on the acquisition condition I3 as the captured image Im3 to the recording device not illustrated (step S19).

[0166] On the other hand, in a case where the information amount is not increased, the determination section 34 determines whether the information amount is decreased in step S17. That is, the determination section 34 determines whether the information amount of the image Im2 based on the acquisition condition I2 obtained by irradiating the WLI light and the NBI light to the same region is decreased compared to the information amount of the image Im1 based on the acquisition condition I1 obtained by irradiating the WLI light to the certain region of the subject.

[0167] In a case where the information amount of the image based on the acquisition condition I2 is decreased compared to the image based on the acquisition condition I1, the determination section 34 determines that the effective image can be acquired by the acquisition condition different in kind from the acquisition condition I2, and instructs the acquisition condition designating section 35 to set the acquisition condition I3 different in kind from the acquisition condition I2 in step S21.

[0168] The acquisition condition designating section 35, for example, changes to information for acquiring the image based on the DRI light as the acquisition condition I3 different in kind from the acquisition condition I2 instead of the NBI light designated by the acquisition condition I2. In addition, the acquisition condition designating section 35 can also change to a condition for acquiring the image using the DRI light, the AFI light as the acquisition condition I3 different in kind from the acquisition condition I2, which includes the NBI light of the other wavelength band different from the wavelength band of the NBI light designated by the acquisition condition I2. Further, the acquisition condition designating section 35 can also change along with the change of the frame rate of the imaging element 22, the change of various image processing of the image processing section 12.

[0169] The image analysis section 32 obtains the analysis result by performing the image analysis on each image acquired based on the acquisition condition I3 different in kind from the acquisition condition I2. The support information generating section 36 generates the support information based on the analysis result. In addition, the video processor 3 temporarily records each image acquired based on the acquisition condition I3 different in kind from the acquisition condition I2 as the image Im4 to the recording device not illustrated (step S22).

[0170] The control section 31 transitions to the next step S23 in a case where the determination is "No" in steps S16, S20 or in a case where the process of step S22 is ended, and displays the display based on the support information generated by the support information generating section 36 on the image Im1 displayed on the monitor 5 in a case where the images acquired based on the acquisition conditions I1 to I3 are the images for the same observation site.

[0171] In addition, in a case where the determination is "Yes" in step S17, the control section 31 instructs the acquisition condition designating section 35 to change the acquisition condition I2 to the acquisition condition I3 different in kind from the acquisition condition I2 in step S18. Figure 16 The example shown illustrates that the condition I3 obtained in steps S18 and S21 is set only once in any given step. However, steps S16 to S22 can be executed repeatedly until the information content neither increases nor decreases. However, such repetition can be very time-consuming, making it difficult to quickly determine the condition, and therefore may end under certain circumstances. It can also be the better of the two conditions. Thus, by executing the image processing method, high-precision support information can be provided using images obtained under favorable conditions. This image processing method includes: an imaging step, capturing images under multiple different imaging conditions to obtain imaging results; a comparison step, comparing multiple imaging results under the different imaging conditions; and an imaging condition changing step, changing a third imaging condition based on the difference in information content obtained in the comparison results.

[0172] Thus, in this embodiment, the same effect as in the first embodiment can be achieved.

[0173] (Third Implementation)

[0174] Figure 16 This is a block diagram illustrating the third embodiment.

[0175] The endoscopic system of the third embodiment can be applied to various endoscopic systems, including those using examination endoscopes such as colonoscopes, and those using surgical endoscopes such as laparoscopes. Figure 16 The illustration depicts an endoscope system 1 of a laparoscopic surgical system.

[0176] like Figure 16 As shown, the system mainly includes: an endoscope 2 (laparoscopy) that captures images of the body cavity of the subject P and outputs image signals; a video processor 3 that is connected to the endoscope 2 and controls its operation, acquires image signals related to the subject captured by the endoscope 2, and performs prescribed image processing on the image signals; a light source device 4, which is built into the video processor 3 and provides prescribed illumination light for illuminating the subject; a monitor 5 that displays the observation images corresponding to the image signals; and a navigation device 30 connected to the video processor 3. In systems using examination endoscopes, although the type of endoscope 2 differs, the other components are similar. Figure 16 The example shown is the same.

[0177] Furthermore, the structures of each structural element in the endoscope system 1 of the third embodiment, namely the endoscope 2, the video processor 3, the light source device 4, the monitor (display) 5, and the navigation device 30, are the same as those in the first embodiment, so detailed descriptions are omitted here.

[0178] The endoscope system 1 of the third embodiment outputs, for example, an image in which a lesion site is marked with high precision by the navigation device 30 to the monitor (display) 5 in the case of a system using an examination endoscope.

[0179] Specifically, in the case of the navigation device 30 in an examination endoscope system using a large intestine endoscope, as shown in FIG. 8, based on the non-missing image information (display image information + analysis image information) provided from the video processor 3 as described above, an image in which a region considered to be a lesion site is marked with high precision is output to the monitor (display) 5, for example, as navigation information to the operator. Figure 16

[0180] On the other hand, in the case of a system using a surgical endoscope, the navigation device 30 outputs an image in which information useful for surgery is prompted to the monitor (display) 5.

[0181] Specifically, in the case of the navigation device 30 in a surgical endoscope system using a laparoscope, as shown in FIG. 9, based on the non-missing image information (display image information + analysis image information) provided from the video processor 3 as described above, information such as the position of a tumor, a resection region, the position of a main blood vessel, and the like is output to the monitor (display) 5, for example, as navigation information to the operator. Figure 16

[0182] As the endoscope system of the third embodiment, the present application, in an endoscope system 1 using various endoscopes, prepares, as image information provided from the video processor 3 to the navigation device 30 as described above, in addition to display image information, analysis image information for the navigation device 30, and performs recognition processing in the navigation device 30 using non-missing image information, and thus in all endoscope systems 1, it is possible to provide useful navigation information (support information) to the operator.

[0183] In addition, the endoscope system of the third embodiment is exemplified as an examination endoscope system and a surgical endoscope system as described above, but is not limited thereto, and the endoscope system of the third embodiment can also be applied to an endoscope system using other kinds of endoscopes.

[0184] ​​In addition, in the technology described here, the controls and functions mainly described in the flowcharts can be set by a program, and the above-described controls and functions can be realized by a computer reading and executing the program. The program, as a computer program product, can record or store the whole or a part thereof in a removable medium such as a floppy disk, a CD-ROM, a nonvolatile memory, or the like, a storage medium such as a hard disk, a volatile memory, or the like, and can be circulated or provided at the time of product shipment or via a removable medium or a communication line. A user can easily realize the image processing apparatus of the present embodiment by downloading the program via a communication network and installing it in a computer, or installing it in a computer from a recording medium.

[0185] The present application is not directly limited to the above-described embodiments, and can be embodied by modifying the constituent elements within a range not departing from the gist thereof in the implementation stage. In addition, various applications can be formed by appropriately combining the plurality of constituent elements disclosed by the above-described embodiments. For example, several of the constituent elements shown in the embodiments can be deleted. Furthermore, the constituent elements in different embodiments can be appropriately combined. Here, the description is made with the example of medical use, but it is needless to say that the application can be applied to devices for civilian use, industrial use, and the like. For example, the navigation device can be applied to devices for detecting abnormalities, devices for rewriting support information into information for calling attention, devices for assisting in determining the quality of articles or work in a production line in a factory using an in-process camera, devices for attention guidance when monitoring using a wearable camera or a robot camera, devices for obstacle determination using a vehicle-mounted camera, and the like, in addition to the medical field. Even a civilian camera can have various uses for guidance. In a microscope, observation based on switching of a light source and image processing is known, and the application of the present application is effective.< / raw> < / raw> < / raw> < / raw> < / raw> < / raw> ​< / raw> < / raw> < / raw> < / raw>

Claims

1. An image processing apparatus, characterized in that, have: The acquisition condition specifying unit sets a first acquisition condition including display acquisition conditions and a second acquisition condition including the display acquisition conditions and the analysis acquisition conditions for the image acquisition unit. The image acquisition unit is capable of acquiring a first image based on the display acquisition conditions and a second image based on the analysis acquisition conditions in a mixed manner. The display acquisition conditions are used to acquire the display image at a visual recognition frame rate, and the analysis acquisition conditions are used to acquire the image for image analysis at a frame rate lower than the visual recognition frame rate. An image analysis unit performs image analysis on the first image acquired by the image acquisition unit; The support information generation unit generates support information based on the image analysis results of the image analysis unit. as well as The control unit controls the switching between the first acquisition condition and the second acquisition condition based on information about blood vessels related to the mucosa based on the image analysis results.

2. The image processing apparatus according to claim 1, characterized in that, The image acquisition unit sets at least one of the following parameters based on at least one of the display acquisition conditions and the analysis acquisition conditions: a first parameter for controlling at least one of the optical system and the imaging element of the imaging device, a second parameter for controlling the illumination light of the imaging object of the imaging device, and a third parameter for signal processing of the first image and the second image.

3. The image processing apparatus according to claim 1, characterized in that, The display acquisition conditions include information that the display image can be acquired at a rate of at least a specified frame rate.

4. The image processing apparatus according to claim 1, characterized in that, The conditions for obtaining the analysis include information that allows the illumination light to be limited to a specified wavelength band.

5. The image processing apparatus according to claim 1, characterized in that, The display acquisition conditions include information for causing the image processing unit, which performs signal processing on the image acquired by the image acquisition unit, to perform signal processing for display. The conditions for obtaining the analysis include information for preventing the image processing unit from performing signal processing for display.

6. The image processing apparatus according to claim 1, characterized in that, The acquisition condition designation unit can set a third acquisition condition, which includes the display acquisition condition and an analysis acquisition condition that is different from the analysis acquisition condition included in the second acquisition condition. The image processing apparatus further includes a determination unit that, based on the image analysis results of the image analysis unit, instructs the acquisition condition specification unit to set the third acquisition condition.

7. The image processing apparatus according to claim 6, characterized in that, The first and second acquisition conditions are the prescribed conditions. The third condition for obtaining the condition is an adaptively changing condition.

8. The image processing apparatus according to claim 6, characterized in that, The determination unit determines the third acquisition condition based on a comparison between the values ​​contained in the image analysis results of the image analysis unit and a predetermined benchmark value.

9. The image processing apparatus according to claim 6, characterized in that, The determination unit determines the third acquisition condition by comparing the image analysis results of the image analysis unit on the image obtained based on the first acquisition condition with the image analysis results of the image analysis unit on the image obtained based on the second acquisition condition.

10. An image processing method comprising the following steps: For the image acquisition unit, a first acquisition condition including display acquisition conditions and a second acquisition condition including the display acquisition conditions and the analysis acquisition conditions are set. The image acquisition unit is capable of acquiring a first image based on the display acquisition conditions and a second image based on the analysis acquisition conditions in a mixed manner. The display acquisition conditions are used to acquire the display image at a visual recognition frame rate, and the analysis acquisition conditions are used to acquire the image for image analysis at a frame rate lower than the visual recognition frame rate. Image analysis results are obtained by performing image analysis on the first image acquired by the image acquisition unit; Support information is generated based on the image analysis results; as well as The switching between the first acquisition condition and the second acquisition condition is controlled based on information about mucosal blood vessels obtained from the image analysis results.

11. A navigation method, characterized in that, A first acquisition condition, including a display acquisition condition, is set for the image acquisition unit. This image acquisition unit is capable of acquiring a first image based on the display acquisition condition and a second image based on an analysis acquisition condition in a mixed manner. The display acquisition condition is used to acquire the display image at a visual recognition frame rate, and the analysis acquisition condition is used to acquire the image for image analysis at a frame rate lower than the visual recognition frame rate. A second acquisition condition is set for the image acquisition unit, which includes the acquisition conditions for display and the acquisition conditions for analysis. Image analysis results are obtained by performing image analysis on the first image acquired by the image acquisition unit. Based on the information about mucosal blood vessels obtained from the image analysis results, the switching between the first acquisition condition and the second acquisition condition is controlled, and a third acquisition condition is set for the image acquisition unit. This third acquisition condition includes the display acquisition condition and an analysis acquisition condition that is different from the analysis acquisition condition included in the second acquisition condition.

12. An endoscope system, characterized in that, have: An endoscope having an illumination section and a camera section, the endoscope being capable of acquiring a first image based on display acquisition conditions and a second image based on analysis acquisition conditions, wherein the display acquisition conditions are used to acquire the display image at a visual recognition frame rate, and the analysis acquisition conditions are used to acquire the image for image analysis at a frame rate lower than the visual recognition frame rate; A video processor, which, based on at least one of the display acquisition conditions and the analysis acquisition conditions, causes the endoscope to acquire the first image and the second image; and Image processing device The image processing device has: The condition specification unit sets a first acquisition condition that includes the display acquisition condition and a second acquisition condition that includes the display acquisition condition and the analysis acquisition condition; An image analysis unit performs image analysis on the first image acquired by the video processor; A support information generation unit generates support information based on the image analysis results of the image analysis unit. as well as The control unit controls the switching between the first acquisition condition and the second acquisition condition based on information about blood vessels related to the mucosa obtained from the image analysis results.

Images