Medical image processing device

Abstract

This invention provides a medical image processing apparatus that does not obstruct a user's observation of medical images, prevents neglect of areas of interest, and maintains the necessary attention during observation. The medical image processing apparatus includes: an image acquisition unit that acquires a medical image containing a subject; a display unit that displays the medical image in a first display area; and a display control unit that controls whether to display notification information for the user in the display unit or not to display notification information in the display unit. The display control unit also controls whether to display notification information in a second display area different from the first display area or to change the displayed notification information to not be displayed. Specifically, the display control unit controls the display of notification information in the first display area when a first condition is met, and controls the change of the displayed notification information in the first display area to not be displayed and the display of notification information in the second display area when a second condition different from the first condition is met.

Description

[0001] This application is a divisional application of the invention patent application filed on April 12, 2018, with national application number 201880033364.6 (international application number PCT / JP2018 / 015361) and entitled "Medical Image Processing Device, Endoscopic System, Diagnostic Support Device and Medical Service Support Device". Technical Field

[0002] The present invention relates to a medical image processing apparatus for providing notification information to a user using medical images. Background Technology

[0003] In the current medical field, medical image processing devices, such as endoscopic systems equipped with light sources, endoscopes, and processors, are becoming increasingly common. Furthermore, in recent years, research has focused on analyzing medical images to identify areas of interest for diagnosis and notifying the user. In addition, notifications are sent regarding error messages related to the subject or medical image, the observation status of the subject, the presence and type of lesions, etc.

[0004] As described above, when displaying notification information to inform users on a monitor, it is desirable to minimize obstruction of the user's observation of the medical image. In contrast, for example, as shown in Patent Document 1, referencing techniques that prevent the operation cursor in the main screen from overlapping with sub-screens, consideration is given to preventing the notification information from overlapping with the area of ​​interest in the medical image. Furthermore, as shown in Patent Document 2, referencing techniques such as clocks that disable the display of necessary information after a certain period, consideration is given to disabling the notification information after a certain period. And, as shown in Patent Document 3, referencing techniques that switch from a normal image to an error message when an anomaly is detected, consideration is given to stopping the display of the medical image and displaying only the notification information when an error message occurs.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2011-096084

[0008] Patent Document 2: Japanese Patent Application Publication No. 2007-243275

[0009] Patent Document 3: Japanese Patent Publication No. 7-027481 Summary of the Invention

[0010] The technical problem to be solved by the invention

[0011] When providing notification information to users, as mentioned above, in addition to minimizing obstruction of the user's observation of medical images, it is necessary to prevent the neglect of the area of ​​interest or maintain the attention required for observation. Therefore, as in Patent Document 2, when the notification information is set to not be displayed, the area of ​​interest may be ignored. Furthermore, as in Patent Document 3, even when the display of the medical image is stopped, the area of ​​interest may still be ignored. In addition, regarding Patent Document 1, the area where the operation cursor is located is not necessarily limited to the area of ​​interest, so the subscreen displaying the notification information may overlap with the area of ​​interest.

[0012] The purpose of this invention is to provide a medical image processing device, endoscope system, diagnostic support device, and medical service support device that does not hinder the user's observation of medical images, prevents the neglect of areas of interest, and maintains the attention required for observation.

[0013] means for solving technical problems

[0014] The medical image processing apparatus of the present invention includes: an image acquisition unit that acquires a medical image including a subject;

[0015] A display unit displays the medical image in a first display area; and a display control unit controls whether to display notification information for notifying the user in the display unit, or whether to not display the notification information in the display unit. The display control unit also controls whether to display the notification information in a second display area different from the first display area, or to change the displayed notification information to not be displayed. When a first condition is met, the display control unit controls whether to display the notification information in the first display area. When a second condition different from the first condition is met, the display control unit controls whether to change the notification information displayed in the first display area to not be displayed, and to display the notification information in the second display area.

[0016] Preferably, the medical image processing device includes a region of interest detection unit that detects a region of interest in the medical image. The first condition is that the position of the region of interest is outside the center of the first display area, and the second condition is that the position of the region of interest is inside the center of the first display area.

[0017] Preferably, the medical image processing device includes a region of interest detection unit, which detects a region of interest from the medical image. The first condition is that the change in the region of interest over time exceeds a threshold value for the change amount, and the second condition is that the change in the region of interest over time is below the threshold value for the change amount.

[0018] Preferably, the medical image processing apparatus includes a region of interest detection unit, which detects a region of interest from the medical image. The first condition is that the brightness value of the region of interest is below a brightness value threshold, and the second condition is that the brightness value of the region of interest exceeds the brightness value threshold.

[0019] Preferably, the first condition is that the first display time of the notification information displayed in the first display area is less than the display limit time, and the second condition is that the first display time exceeds the display limit time.

[0020] Preferably, the medical image processing device has a region of interest detection unit that detects a region of interest from the medical image. The first condition is that a combination of at least two of the following information satisfies the combination condition: the area of ​​the region of interest, the position of the region of interest, the amount of change of the region of interest over time, the brightness value of the region of interest, and the first display time of the notification information displayed in the first display area. Preferably, the second condition is that the combination information does not satisfy the combination condition.

[0021] Preferably, the first condition is that the area-brightness combination information, which combines the area of ​​the region of interest with the brightness value of the region of interest, satisfies the area-brightness combination condition, and the second condition is that the area-brightness combination information does not satisfy the area-brightness combination condition.

[0022] Preferably, the display control unit controls the display of the notification information overlaid with the medical image in the first display area.

[0023] Preferably, the medical image includes a region of interest detection unit that detects regions of interest from the medical image, and the notification information is a notification message used to notify information related to the region of interest.

[0024] Preferably, the notification information is the presence or absence of lesions, the type of lesions, or the detection status of the treatment equipment.

[0025] Preferably, the notification information is saved overlapping with the medical image.

[0026] Preferably, the medical image processing apparatus has a feature quantity image generation unit, which generates a feature quantity image by performing calculations based on at least one of a general light image obtained by irradiating light in a white frequency band or light in multiple wavelength frequency bands as light in the white frequency band, and a special light image obtained by irradiating light in a specific wavelength frequency band, wherein the medical image is the feature quantity image.

[0027] Preferably, the change in the region of interest over time is the amount of movement of the region of interest within a specific time period.

[0028] Preferably, the medical image processing apparatus includes an image analysis unit for detecting error information from the medical image, wherein the error information is a blur or dirt on the subject in the medical image, and the notification information includes information for notifying the error information.

[0029] Preferably, the change in the region of interest over time is the amount of movement of the region of interest within a specific time period. The medical image processing device also includes an image analysis unit for detecting error information from the medical image. The error information is a blur or dirt on the subject in the medical image. The notification information includes information for notifying the error information.

[0030] Invention Effects

[0031] According to the present invention, it does not hinder the user's observation of medical images, prevents the neglect of areas of interest, and maintains the attention required for observation. Attached Figure Description

[0032] Figure 1 This is an external view of the endoscope system.

[0033] Figure 2 This is a block diagram of an endoscope system.

[0034] Figure 3 This is a block diagram showing the image processing unit of the first embodiment.

[0035] Figure 4 This is an illustration of a bounding box, one of the notification messages.

[0036] Figure 5 It is an image diagram representing the first display area A1 and the second display area A2 in the monitor.

[0037] Figure 6 (A) is an image showing the bounding box as a notification message when the area of ​​interest is small. Figure 6 (B) is an image of a notification icon displayed when the area of ​​interest is large.

[0038] Figure 7 (A) is an image that displays a bounding box as a notification when the area of ​​interest moves outside the center. Figure 7 (B) is an image that displays a notification icon when the area of ​​interest enters the center area.

[0039] Figure 8(A) is an image showing a notification icon when the area of ​​interest moves slowly. Figure 8 (B) is an image showing the bounding box as a notification message when the area of ​​interest moves quickly.

[0040] Figure 9 It is a flowchart representing a series of processes related to the display control of notification information.

[0041] Figure 10 This is a block diagram showing the image processing unit of the second embodiment.

[0042] Figure 11 It is an image that displays "blurred" text information as a notification message.

[0043] Figure 12 This is a block diagram showing the image processing unit of the third embodiment.

[0044] Figure 13 It is an image that displays the text message "white light mode" as a notification.

[0045] Figure 14 This is a block diagram showing the image processing unit of the fourth embodiment.

[0046] Figure 15 This is an explanatory diagram showing the identification evaluation value and the drag bar.

[0047] Figure 16 It is an image that displays the identification evaluation value and the drag bar as a notification message. Detailed Implementation

[0048] [First Implementation]

[0049] like Figure 1 As shown, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 16, a display 18, and a control console 19. The endoscope 12 illuminates the object being observed (i.e., the subject) with light and takes a photograph of the subject illuminated by the light. The light source device 14 generates illumination light for illuminating the subject. The processor device 16 performs system control and image processing of the endoscope system 10. The display 18 is a display unit that displays images output from the processor device 16. The control console 19 is an input device for inputting settings to the processor device 16, etc. (e.g., a keyboard).

[0050] The endoscope 12 has an insertion portion 12a that is inserted into the subject, an operating portion 12b provided at the base of the insertion portion 12a, a bending portion 12c provided at the front end of the insertion portion 12a, and a front end portion 12d. The bending portion 12c is bent by operating the bend knob 12e of the operating portion 12b. By bending the bending portion 12c, the front end portion 12d is oriented in the desired direction. A spray nozzle (not shown) is provided at the front end portion 12d for spraying air or water, etc., toward the subject.

[0051] Furthermore, in addition to the angle knob 12e, the operation section 12b also includes a zoom operation section 13. By operating the zoom operation section 13, the subject can be magnified or reduced for photographing. Additionally, a clamp channel (not shown) for inserting penetrating instruments or the like is provided from the insertion section 12a to the front end section 12d. The penetrating instrument is inserted into the clamp channel through the clamp inlet 12f.

[0052] like Figure 2 As shown, the light source device 14 includes a light source unit 20 and a light source control unit 22. The light source unit 20 emits illumination light for illuminating the subject. The light source unit 20 includes one or more light sources. The light source control unit 22 controls the driving of the light source unit 20. The light source control unit 22 independently controls the timing of the on / off of the light sources constituting the light source unit 20, as well as the amount of light emitted when on. As a result, the light source unit 20 can emit various types of illumination light with different amounts or timings of light emission.

[0053] Illumination light emitted by the light source 20 is incident on the light guide 41. The light guide 41 is built into the endoscope 12 and the universal plug cord, and propagates the illumination light to the front end 12d of the endoscope 12. The universal plug cord is a plug cord that connects the endoscope 12 to the light source device 14 and the processor device 16. In addition, multimode optical fiber can be used as the light guide 41. As an example, a core diameter of 105 μm, a cladding diameter of 125 μm, and a diameter including the protective layer that forms the outer skin can be used. Thin optical fiber cable.

[0054] An illumination optical system 30a and an imaging optical system 30b are provided at the front end 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 45 through which illumination light is emitted toward the subject. The imaging optical system 30b has an objective lens 46, a zoom lens 47, and an image sensor 48. The image sensor 48 captures the subject via the objective lens 46 and the zoom lens 47 using reflected light from the illumination light returning from the subject (in addition to reflected light, it also includes scattered light, fluorescence emitted by the subject, or fluorescence caused by a drug injected into the subject). The zoom lens 47 is moved by operating the zoom operation unit 13, and magnifies or reduces the subject captured by the image sensor 48.

[0055] Image sensor 48 is, for example, a color sensor with primary color filters, and includes three types of pixels: B pixels (blue pixels) with a blue filter, G pixels (green pixels) with a green filter, and R pixels (red pixels) with a red filter. The blue filter primarily transmits violet to blue light. The green filter primarily transmits green light. The red filter primarily transmits red light. As described above, if a subject is photographed using the primary color image sensor 48, up to three images can be simultaneously obtained: a B image (blue image) obtained from the B pixels, a G image (green image) obtained from the G pixels, and an R image (red image) obtained from the R pixels.

[0056] Furthermore, as the image sensor 48, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor can be used. In this embodiment, the image sensor 48 is a primary color sensor, but a complementary color sensor can also be used. A complementary color sensor, for example, has cyan pixels with a cyan filter, magenta pixels with a magenta filter, yellow pixels with a yellow filter, and green pixels with a green filter. When using a complementary color sensor, if a complementary-to-primary color conversion is performed, the images obtained from the pixels of each of the above colors can be converted into B images, G images, and R images. Furthermore, instead of a color sensor, a monochrome sensor without color filters can be used as the image sensor 48. In this case, by sequentially photographing the subject using illumination light of each color such as BGR, images of the above colors can be obtained.

[0057] The processor unit 16 includes a control unit 52, an image acquisition unit 54, an image processing unit 61, a display control unit 66, a recording control unit 68, and a storage memory 69. The control unit 52 performs centralized control of the endoscope system 10, such as the synchronization control of illumination timing and image capture timing. Furthermore, when various settings are input using the console 19 or the like, the control unit 52 inputs these settings to various parts of the endoscope system 10, such as the light source control unit 22, the image sensor 48, or the image processing unit 61.

[0058] The image acquisition unit 54 acquires an image of the subject from the image sensor 48. The image acquired by this image acquisition unit 54 is obtained through a medical device such as the endoscope 12, and is therefore called a medical image. The image acquisition unit 54 includes a DSP (Digital Signal Processor) 56, a noise reduction unit 58, and a conversion unit 59, and uses them to perform various processing on the acquired medical image as needed. As needed, the DSP 56 performs various processing on the acquired medical image, such as defect correction processing, offset processing, gain correction processing, linear matrix processing, gamma conversion processing, demosaic processing, and YC conversion processing.

[0059] The defect correction process corrects the pixel values ​​of pixels corresponding to defective pixels on the image sensor 48. The offset process reduces the dark current component and sets an accurate zero level from the image after defect correction. The gain correction process adjusts the signal levels of each image by multiplying the offset image by the gain. The linear matrix processing improves the color reproducibility of the offset image, and the gamma conversion process adjusts the brightness or chroma of the image after linear matrix processing.

[0060] Additionally, when the image sensor 48 is a color sensor, demosaic processing is performed. Demosaic processing (also known as isotropic processing or synchronous processing) is a process of interpolating the pixel values ​​of missing pixels and is applied to the gamma-converted image. Missing pixels are pixels that lack pixel values ​​due to the arrangement of color filters (because pixels of other colors are arranged in the image sensor 48). For example, a B image is an image obtained by capturing the subject in B pixels; therefore, pixels at positions corresponding to G or R pixels have no pixel values. Demosaic processing interpolates the B image to generate pixel values ​​for the pixels located at the G and R pixel positions in the image sensor 48. YC conversion processing converts the demosaiced image into a luminance channel Y and a chrominance channel Cb and chrominance channel Cr.

[0061] The noise reduction unit 58 performs noise reduction processing on the luminance channel Y, chrominance channel Cb, and chrominance channel Cr, for example, using a moving average method or a median filtering method. The conversion unit 59 converts the noise-reduced luminance channel Y, chrominance channel Cb, and chrominance channel Cr back into BGR images of each color.

[0062] The image processing unit 61 performs various image processing operations on the medical images acquired by the image acquisition unit 54. In this embodiment, the image processing unit 61 generates notification information for notifying the user. The image processing unit 61 then transmits the notification information and the medical images to the display control unit 66 or the recording control unit 68. Details regarding the image processing unit 61 will be described later.

[0063] The display control unit 66 converts the notification and medical information transmitted from the image processing unit 61 into a format suitable for display on the display 18 and outputs it to the display 18. Thus, at least the medical image and notification information are displayed on the display 18. In this embodiment, the display control unit 66 controls whether the notification information is displayed on the display 18 or whether the notification information is set to not be displayed from the display 18. Details regarding the display control unit 66 will be described later.

[0064] The recording control unit 68 converts the notification information and medical images transmitted from the image processing unit 61 into a format suitable for storage in the storage memory 69 and records them in the storage memory 69. In the recording control unit 68, when recording control is performed according to the control in the display control unit 66, recording control is sometimes performed independently of the control in the display control unit 66. For example, when recording control is performed according to the control in the display control unit 66, the notification information and medical images at that time are saved in the storage memory 69 by the recording control unit 68 at a time when the notification information is displayed by the display control unit 66. On the other hand, when recording control is performed independently of the control in the display control unit 66, for example, when a region of interest is detected in the image processing unit 61, the notification information and medical images at that time are saved in the storage memory 69 by the recording control unit 68.

[0065] Furthermore, when saving notification information and medical images to the storage memory 69, it can be configured to save the notification information and medical images overlapping in the storage memory 69, or it can be configured to save the notification information and medical images as at least two different data in the storage memory 69. Moreover, when the notification information and medical images are combined into one data set and saved to the storage memory 69, it is preferable to save it in a data format such as PNG, Exif, GIF, or MP4.

[0066] like Figure 3As shown, the image processing unit 61 includes a region of interest detection unit 70 and a notification information generation unit 72. The region of interest detection unit 70 detects regions of interest from the medical image as objects of examination or diagnosis. In the region of interest detection unit 70, for example, in addition to performing a Convolutional Neural Network on the medical image, the region of interest is also detected based on feature quantities obtained from the color information, pixel value gradient, etc. of the medical image. In addition, the pixel value gradient, etc., represents changes through the shape of the subject (most of the undulations or local depressions or bulges of the mucous membrane, etc.), color (the color of whitening caused by inflammation, bleeding, redness or atrophy, etc.), tissue characteristics (the thickness, depth, density of blood vessels, or combinations thereof, etc.), or structural characteristics (pit patterns, etc.).

[0067] Furthermore, the areas of interest detected by the area of ​​interest detection unit 70 include, for example, lesions such as cancer, benign tumors, inflamed areas (including areas with changes such as bleeding or atrophy in addition to so-called inflammation), areas with burn marks caused by heating, areas marked by staining with dyes and fluorescent agents, or areas where a biopsy was performed. In other words, areas containing lesions, areas where lesions are likely to exist, areas where certain procedures such as biopsies have been performed, areas treated with clamps or forceps, or dark areas (areas where light cannot reach due to folds or deep lumens) that require detailed observation regardless of the likelihood of a lesion can be considered areas of interest. In the endoscope system 10, the area of ​​interest detection unit 70 detects areas containing at least one of the following: lesions, benign tumors, inflamed areas, marked areas, or biopsy areas.

[0068] The notification information generation unit 72 generates notification information related to the area of ​​interest based on the area of ​​interest detected by the area of ​​interest detection unit 70. Specifically, such as... Figure 4 As shown, as a notification message, a rectangular bounding box 74 is generated, enclosing the region of interest (ROI) with dotted lines. Additionally, as a notification message, arrows representing the ROI or icons indicating detection within the ROI can also be generated (see reference). Figure 6 (e.g., icon 76). Furthermore, as notification information, it can also be set to color information related to a color that can distinguish the area of ​​interest from other areas. Additionally, the notification information generation unit 72 preferably generates multiple notification messages suitable for display control, so that different notification messages can be displayed according to the display control in the display control unit 66.

[0069] like Figure 5As shown, the display control unit 66 controls the display of a medical image in the first display area A1 of the display 18. Furthermore, the display control unit 66 controls the display of notification information on the display 18 and controls the removal of currently displayed notification information from the display 18. Specifically, the display control unit 66 controls the display of notification information in the first display area A1 and the removal of currently displayed notification information from the first display area. Additionally, the display control unit 66 controls the display of notification information in a second display area A2, which is different from the first display area on the display 18, and the removal of currently displayed notification information from the display. Moreover, when displaying notification information in the first display area A1, it is preferable that the notification information overlaps with the medical image.

[0070] Furthermore, the display control unit 66 controls the display of notification information in both the first display area A1 and the second display area A2, and controls the setting of the notification information displayed in at least one of the first display area A1 or the second display area A2 to be non-displayed. Additionally, the display control unit 66 controls the display to overlap the notification information with the medical image displayed in the first display area A1 when the notification information is displayed in the first display area A1.

[0071] Here, the first display area A1 is an area that at least includes the image of the subject located in front of the endoscope 12. The second display area A2 is an area that at least includes the subject located generally to the side or behind the endoscope 12. In this embodiment, the first display area A1 is a barrel-shaped area, and the second display area A2 is the area of ​​the entire screen of the display 18 except for the barrel-shaped first display area A1.

[0072] The display control unit 66 controls the display of notification information in a manner that does not obstruct observation of the area of ​​interest. Specifically, the display control unit 66 displays the notification information in the first display area A1 when a first condition is met, and sets the notification information displayed in the first display area to not be displayed when a second condition different from the first condition is met, and displays the notification information in the second display area A2.

[0073] For example, the first condition could be that the area of ​​the region of interest is below a certain area threshold, and the second condition could be that the area of ​​the region of interest exceeds the area threshold. In this case, such as... Figure 6 As shown in (A), when the area of ​​the region of interest (ROI) falls below the area threshold (when the first condition is met), the display control unit 66 controls the display to overlay the bounding box 74 surrounding the ROI onto the first display area A1 as a notification. Thus, by overlaying the bounding box 74, the user can reliably identify the region of interest even when the ROI is small.

[0074] Then, if the medical image displayed in the first display area A1 changes due to the operation of the endoscope 12, the area of ​​this region of interest also changes. In this case, if the region of interest on the medical image becomes larger, the bounding box 74 will correspondingly become larger, and sometimes such a larger bounding box 74 may obstruct the observation of the medical image. Therefore, as Figure 6 As shown in (B), when the area of ​​the region of interest (ROI) exceeds the area threshold (when the second condition is met), the display control unit 66 sets the bounding box 74 to be invisible from the first display area A1. Furthermore, as a notification, the display control unit 66 displays the icon 76 indicating the detection of the ROI in the second display area A2. Therefore, even if the bounding box 74 is invisible from the first display area A1, the ROI of interest can be prevented from being ignored.

[0075] Furthermore, the first condition can be that the location of the area of ​​interest is outside the center range of the first display area A1, and the second condition can be that the location of the area of ​​interest is inside the center range of the first display area A1. Here, the center range CR of the first display area A1 refers to the range from the center CP of the first display area to a predetermined distance L1 (see reference). Figure 7 (A)). For example Figure 7 As shown in (A), when the location of the region of interest (ROI) is outside the center range (CR) (when the first condition is met), it is difficult for the user to identify the region of interest (ROI). Therefore, as a notification, the bounding box 74 surrounding the region of interest (ROI) is overlaid and displayed.

[0076] Then, if the medical image displayed in the first display area A1 changes due to the operation of the endoscope 12, the position of this area of ​​interest also changes. In this case, if the area of ​​interest moves from outside the central CR to within the central CR, the user can identify the area of ​​interest even without the bounding box 74. Therefore, as Figure 7 As shown in (B), when the region of interest (ROI) enters the center range CR (when the second condition is met), the display control unit 66 sets the bounding box 74 to be invisible from the first display area A1. Furthermore, as a notification, the display control unit 66 displays the icon 76 indicating that the ROI has been detected in the second display area A2. Therefore, even if the bounding box 74 is invisible from the first display area A1, the region of interest can be prevented from being ignored.

[0077] Furthermore, the first condition can be that the change in the area of ​​interest over time exceeds a threshold value for the change, and the second condition can be that the change in the area of ​​interest over time is below the threshold value for the change. Here, the change in the area of ​​interest over time refers, for example, the amount of movement of the area of ​​interest within a specific time period. Figure 8As shown in (A), when the change in the region of interest (ROI) over time is below a threshold (when the second condition is met), the display control unit 66 displays an icon 76 indicating the detected ROI in the second display area A2 as a notification. Thus, when the movement of the ROI is slow, the user can grasp the ROI; therefore, it is preferable to display only icon 76 to draw attention.

[0078] Moreover, when the area of ​​interest moves faster, such as Figure 8 As shown in (B), the display control unit 66 displays the bounding box 74 surrounding the region of interest ROI in the first display area A1 as a notification when the change in the region of interest ROI over time exceeds a threshold (when the first condition is met). Therefore, even if the region of interest moves faster, the user can reliably determine the location of the region of interest and perform diagnostics.

[0079] Furthermore, the first condition can be that the brightness value of the area of ​​interest is below a brightness value threshold, and the second condition can be that the brightness value of the area of ​​interest exceeds the brightness value threshold. Here, the brightness value of the area of ​​interest refers, for example, to a representative value such as the average pixel value of the pixels contained in the area of ​​interest or the most frequent value of the pixel value obtained from the pixel values ​​of the pixels contained in the area of ​​interest. When observing an area of ​​interest displayed on a medical image, when the brightness value of the area of ​​interest is low, it becomes difficult for the user to identify the area of ​​interest. Therefore, when the brightness value of the area of ​​interest is below the brightness value threshold (when the first condition is met), the display control unit 66 displays a notification message indicating that the bounding box surrounding the area of ​​interest (and...) Figure 6 The bounding box 74 (which is the same as the bounding box 74) is displayed in the first display area A1. Thus, even when the area of ​​interest is dark, it becomes possible to identify the area of ​​interest and perform reliable diagnosis.

[0080] On the other hand, when the brightness value of the area of ​​interest is high, the user can identify the area of ​​interest even without information such as a bounding box indicating its location. In this case, when the brightness value of the area of ​​interest exceeds a brightness threshold (when the second condition is met), the display control unit 66 sets the bounding box displayed in the first display area A1 to be hidden. Furthermore, the display control unit 66 sends a notification indicating that an icon (and...) has been detected in the area of ​​interest. Figure 6 (The same icon 76) is displayed in the second display area A2. Thus, even if the bounding box 74 of the first display area A1 becomes invisible, the area of ​​interest can be prevented from being ignored.

[0081] Furthermore, when a notification message is displayed in the first display area A1 based on the detection of the area of ​​interest, the first condition can be set to the case where the first display time of the notification message in the first display area A1 is within a preset display limit time, and the second condition can be set to the case where the first display time exceeds the display limit time. Here, the first display time is timed based on the start of displaying the notification message in the first display area A1.

[0082] When the first display time is within the display limit time (when the first condition is met), the display control unit 66, as a notification message, displays the bounding box (and) surrounding the area of ​​interest. Figure 6 The bounding box 74 (same as the bounding box 74) is displayed in the first display area A1. Furthermore, when the first display time exceeds the display limit time (when the second condition is met), the display control unit 66 sets the bounding box to be hidden from the first display area A1. Additionally, the display control unit 66 sends a notification message indicating that the icon (same as the bounding box 74) in the detection area of ​​interest is displayed in the first display area A1. Figure 6 The same icon 76 is displayed in the second display area A2. Therefore, as long as the bounding box is displayed for a certain period of time, the user can identify the area of ​​interest, and after a certain period, by setting the bounding box to be invisible, it no longer obstructs the observation of the medical image. Furthermore, even if the bounding box 74 in the first display area A1 becomes invisible, displaying the icon in the second display area A2 prevents the area of ​​interest from being overlooked.

[0083] Furthermore, the first condition can be that the area for displaying notification information is designated as the first display area A1 by user input, and the second condition can be that the area for displaying notification information is designated as the second display area A2 by user input. User input can be performed not only through the console 19 but also through the display area designation unit (not shown) provided in the operation unit 12b of the endoscope 12. When the first display area A1 is designated (when the first condition is met), the display control unit 66 displays the notification information in the first display area A1 as a condition for detecting the area of ​​interest. When the second display area A2 is designated (when the second condition is met), the display control unit 66 displays the notification information in the second display area A2 as a condition for detecting the area of ​​interest. In this case, if the notification information is displayed in the first display area A1 before the second display area A2 is designated, the notification information can also be set to not be displayed from the first display area A1.

[0084] Furthermore, the first condition can be defined as a combination of at least two of the following: the area of ​​the region of interest, the position of the region of interest, the amount of change of the region of interest over time, the brightness value of the region of interest, and the first display time, satisfying the combination condition. The second condition can be defined as a combination of the aforementioned information not satisfying the combination condition. For example, the first condition can be defined as an area-brightness combination of the area of ​​the region of interest and the brightness value of the region of interest satisfying the area-brightness combination condition, and the second condition can be defined as an area-brightness combination of the area of ​​the region of interest and the brightness value of the region of interest not satisfying the area-brightness combination condition.

[0085] Here, when the area-brightness combination information satisfies the area-brightness combination condition (when the first condition is met), it is defined as the area of ​​the region of interest being below the area threshold and the brightness value of the region of interest being below the brightness value threshold. In this case, the display control unit 66, as a notification message, displays the bounding box surrounding the region of interest (and...) Figure 6 (The bounding box 74 is the same as the bounding box) is displayed in the first display area A1.

[0086] On the other hand, if the area-brightness combination information does not meet the area-brightness combination conditions (when the second condition is met), the following scenarios are defined: the area of ​​the region of interest exceeds the area threshold and the brightness value of the region of interest is below the brightness value threshold; the area of ​​the region of interest is below the area threshold and the brightness value of the region of interest exceeds the brightness value threshold; and the area of ​​the region of interest exceeds the area threshold and the brightness value of the region of interest exceeds the brightness value threshold. In this case, the display control unit 66 sets the bounding box to non-display from the first display area A1. Furthermore, as a notification, the display control unit 66 will display an icon (and icon) indicating the detection of the region of interest. Figure 6 (The same icon as 76) is displayed in the second display area A2.

[0087] The display control unit 66 can also be configured such that, when the first and second conditions are used to control the display of notification information, a third condition different from the first and second conditions is used. When the third condition is met, the display control unit 66 controls the notification information displayed in the second display area A2 to be hidden. By hiding the notification information in either the first display area A1 or the second display area A2, the user can reliably identify the area of ​​interest, thus preventing the notification information from obstructing the observation of the medical image. Furthermore, as a third condition, for example, there is a case where the second display time of the notification information displayed in the second display area A2 exceeds the display limit time. Here, the second display time is timed from the moment the notification information begins to be displayed in the second display area A2.

[0088] Furthermore, the display control unit 66 can also be configured such that, when a specific condition is met, it controls the display of notification information in either the first display area A1, the second display area A2, or both the first display area A1 and the second display area A2, or controls the setting of the notification information to not be displayed. Additionally, the display control unit 66 can also be configured such that, when the specific condition is not met, it controls the display of notification information in a display area different from when the specific condition is met, or controls the setting of the notification information to not be displayed. Moreover, "when the specific condition is met" can be set to "when the first condition is met," and "when the specific condition is not met" can be set to "when the second condition is met."

[0089] For example, when a specific condition is met, the display control unit 66 displays a notification message in the first display area A1, and sets the notification message to not be displayed in the second display area A2, based on the condition that the area of ​​interest is detected. Conversely, when the specific condition is not met, the display control unit 66 displays a notification message in the second display area A2, and sets the notification message to not be displayed in the first display area A1, based on the condition that the area of ​​interest is detected. Alternatively, the display control unit 66 may be configured to display a notification message in both the first display area A1 and the second display area A2, based on the condition that the specific condition is not met. Furthermore, as described above, user input is performed not only through the console 19 but also through the display area designation unit provided in the operation unit 12b of the endoscope 12.

[0090] Next, according to Figure 9 The flowchart shown illustrates the display control based on notification information from the display control unit 66. The endoscope 12 illuminates the object of observation, i.e., the subject, and takes a picture of the subject. A medical image is obtained by taking a picture of the subject. The region of interest detection unit 70 detects a region of interest from the medical image. If a region of interest is detected, notification information is generated by the notification information generation unit 72 to notify information related to the region of interest. For example, a bounding box surrounding the region of interest is generated as the notification information. The medical image and the notification information are transmitted to the display control unit 66.

[0091] The display control unit 66 controls the display of medical images in the first display area A1 of the display 18. Furthermore, the display control unit 66 controls the display of notification information on the display 18 and controls the removal of notification information from the display 18. The display control unit 66 controls the display of notification information in a manner that does not obstruct observation of the area of ​​interest. When the area of ​​interest is small, or when the first condition is met, the display control unit 66 displays the notification information overlaid with the medical image in the first display area A1. For example, the area of ​​interest is surrounded by a bounding box. Furthermore, the entire area of ​​interest is painted with a specific color, or a semi-transparent color is overlaid on the area of ​​interest. At this time, the overlapping color can also be changed according to the content of the notification information.

[0092] In contrast, the display control unit 66 is configured such that when the area of ​​interest is large or the second condition is met, a notification message is displayed in the second display area A2. In this case, when the notification message is displayed in the first display area A1, the notification message in the first display area A1 is set to not be displayed. For example, if a bounding box is displayed in the first display area A1, the bounding box is set to not be displayed in the first display area A1, while an icon indicating that the area of ​​interest is being detected is displayed in the second display area A2. Thus, even if the bounding box is set to not be displayed, the area of ​​interest can be prevented from being ignored. Furthermore, the above-described display control of the notification message based on the display control unit 66 is repeated until the observation of the area of ​​interest ends. Commands related to ending the observation of the area of ​​interest are executed via the operation console 19, etc.

[0093] [Second Implementation]

[0094] In the second embodiment, as notification information, error information related to the subject image or medical image is displayed on the display 18. Therefore, as... Figure 10 As shown, in the image processing unit 61 of the second embodiment, in addition to the region of interest detection unit 70, an error information detection unit for detecting error information from medical images is also provided as an error information detection unit. Alternatively, an error information detection sensor can be used as the error information detection unit to detect error information. The error information detection sensor can also be provided not only at the front end 12d of the endoscope 12, but also at the insertion site of the treatment device that penetrates the forceps channel. Furthermore, in the second embodiment, since error information can be detected, region of interest detection may not be necessary; therefore, the region of interest detection unit 70 may not be provided in the image processing unit 61.

[0095] In the image analysis unit 80 for error information detection, blurring in the medical image is detected as error information. This blurring is caused by movement of at least one of the subject or the tip of the endoscope 12. One method for detecting blur is to calculate the high-frequency components of the medical image, and consider blurring as occurring when the high-frequency components fall below a certain value. Furthermore, the image analysis unit 80 also detects the degree of dirt on the subject as error information. Dirt on the subject can include residue, residual liquid, etc. One method for detecting the degree of dirt on the subject is to calculate color information (e.g., the ratio of B image to G image and the ratio of G image to R image) from the medical image, and calculate the degree of dirt on the subject by comparing the calculated color information with the dirt color information corresponding to the dirt on the subject. In this case, the closer the color information is to the dirt color information, the greater the degree of dirt on the subject.

[0096] Furthermore, in the notification information generation unit 72 of the second embodiment, notification information for notifying error information is generated. The notification information may include text indicating that an error has occurred and the type of error. When the error information is blurriness appearing in a medical image, it is preferable to use text information indicating "blurriness has occurred." And when the error information is dirt on the subject, it is preferable to use text information indicating the occurrence and degree of dirt, such as "dirt has occurred (moderate)."

[0097] The display control unit 66 controls the display of notification information in a manner that does not obstruct the observation of medical images. Specifically, such as... Figure 11 As shown, when specific conditions used in error information detection are met, the display control unit 66 displays notification information in the second display area A2 (in... Figure 11 (The text information is "fuzzy generated"). Here, the specific conditions are preferably set to the conditions related to error information detection in the first and second conditions shown in the first embodiment.

[0098] If a specific condition is no longer met due to subsequent changes in circumstances, the notification information can be set to not be displayed in the second display area A2. However, even if the specific condition is not met, the notification information can continue to be displayed in the second display area A2 if error detection continues for a certain period of time. Furthermore, it can also be displayed in the first display area A1, in addition to the second display area A2.

[0099] [Third Implementation]

[0100] In the third embodiment, the observation status of the subject is displayed on the display 18 as notification information. Various observation statuses of subjects using the endoscope system 10 can be exemplified. For example, in the endoscope system 10, various types of illumination light with different wavelengths can be irradiated onto the subject. Therefore, the observation status of the subject can be changed according to multiple illumination modes with different types of illumination light.

[0101] Furthermore, the endoscope system 10 can perform various image processing operations on medical images obtained by photographing the subject. Therefore, the observation state of the subject can be changed according to the different image processing modes applied to the medical images. Moreover, the endoscope system 10 can magnify the subject through the operation of the zoom operation unit 13. Therefore, the observation state of the subject can be changed by either non-magnified observation (without magnification) or magnified observation (with magnification).

[0102] like Figure 12 As shown, in the image processing unit 61 of the third embodiment, in addition to the region of interest detection unit 70, an observation state acquisition unit 81 is also provided to acquire the observation state of the subject. The observation state acquisition unit 81 communicates not only with the processor device 16, but also with the endoscope 12 or the light source device 14. Therefore, the observation state acquisition unit 81 acquires the illumination mode or image processing mode currently used in the image processing unit 61 as the observation state of the subject. Furthermore, the observation state acquisition unit 81 acquires non-magnified observation or magnified observation as the observation state of the subject based on the operation status of the zoom operation unit 13.

[0103] Furthermore, in the notification information generation unit 72 of the third embodiment, notification information is generated to notify the subject of its observation state. The notification information may include, for example, text information indicating the subject's observation state. When the subject's observation state is represented by an illumination mode, it is preferable to set the text information to the currently used illumination mode (e.g., if white light is used as the illumination light, it is set to "white light mode"). Also, when the subject's observation state is represented by an image processing mode, it is preferable to set the text information to the currently used image processing mode (e.g., when performing image processing on a white image obtained by photographing a subject illuminated with white light, it is set to "white image mode"). Furthermore, when the subject's observation state is non-magnified observation, it is preferable to set the text information to "non-magnified observation," and when the subject's observation state is magnified observation, it is preferable to set the text information to "magnified observation."

[0104] The display control unit 66 controls the display of notification information in a manner that does not obstruct the observation of medical images. Specifically, such as... Figure 13 As shown, when certain conditions are met, the display control unit 66 displays notification information in the second display area A2 (in... Figure 13 (The text information is "white light mode"). Here, the specific conditions are preferably set to the conditions related to the observation state of the subject in the first and second conditions shown in the first embodiment.

[0105] When a specific condition is no longer met due to subsequent changes in circumstances, the notification information can be set to not be displayed from the second display area A2. Alternatively, even when the specific condition is not met, if there are other user-specified actions via the console 19, the notification information can continue to be displayed in the second display area A2. Furthermore, it can be displayed in the first display area A1 in addition to the second display area A2. It can also be configured to switch the type of notification information when the observation state of the subject changes. For example, when changing from a white light mode to a special light mode using a specific frequency band, the notification information can be switched from "white light mode" to "special light mode". The notification information can also be set to be displayed continuously for a certain period of time and then set to not be displayed.

[0106] [Fourth Implementation]

[0107] In the fourth embodiment, as notification information, the presence or absence of a lesion, the type of lesion, or the detection status of the treatment device are displayed on the display 18. Regarding the presence or absence of a lesion and the type of lesion, such as Figure 14 As shown, identification is performed by the identification processing unit 82 based on the feature quantities contained in the region of interest. In the identification processing unit 82, the presence or absence of lesions and the type of lesions are calculated using identification processing such as a Convolutional Neural Network in the region of interest. Furthermore, the detection of treatment devices is performed by the treatment device detection unit 86 based on the feature quantities contained in the region of interest. The treatment device detection unit 84 also uses treatment device detection processing such as a Convolutional Neural Network to detect whether a treatment device is present in the region of interest.

[0108] Furthermore, in the notification information generation unit 72 of the fourth embodiment, notification information is generated to notify the subject of its observation status. When the notification information concerns the presence or absence of a lesion or the type of lesion, such as... Figure 15 As shown, as notification information, it is preferable to generate an identification evaluation value 88 indicating the degree of deterioration or similarity of the lesion, and a drag bar 90 that moves in conjunction with the identification evaluation value 88. The identification evaluation value 88 indicates that the higher the evaluation value, the higher the probability of the lesion. Regarding the drag bar 90, the higher the identification evaluation value 88, the larger the length Ls becomes. Furthermore, when the notification information indicates the presence or absence of a treatment device, it is preferable to generate text information indicating the presence or absence of a treatment device (e.g., "Treatment device detection detected," etc.).

[0109] The display control unit 66 controls the display of notification information in a manner that does not obstruct the observation of medical images. Specifically, such as... Figure 16 As shown, when certain conditions are met, the display control unit 66 displays notification information in the second display area A2 (in... Figure 16 The evaluation value 88 and the drag bar 90 are shown in the middle. When a specific condition is no longer met due to subsequent changes in the situation, the notification information can be set to not be displayed from the second display area A2. However, even if the specific condition is not met, the notification information can continue to be displayed in the second display area A2 if there are other user-specified actions via the console 19, etc. Furthermore, the information can be displayed in the first display area A1 instead of the second display area A2, or in addition to replacing the second display area A2.

[0110] Alternatively, control can be implemented by combining two or more of the following as notification information for notifying the user: "information related to the area of ​​interest" shown in the first embodiment, "error information related to the subject or medical image" shown in the second embodiment, "observation status of the subject" shown in the third embodiment, and "presence or absence of lesions, type of lesions, and detection status of treatment instruments" shown in the fourth embodiment, and setting them to be displayed or not displayed in the first display area A1 or the second display area.

[0111] Furthermore, while the invention is applied to the endoscope system 10 in embodiments 1 to 4, it is not limited to the endoscope system 10; the invention can also be applied to medical image processing apparatuses that process medical images. Moreover, the invention can also be applied to diagnostic support apparatuses that provide diagnostic support to users using medical images. Furthermore, the invention can also be applied to medical service support apparatuses that support medical services such as diagnostic reports using medical images.

[0112] Furthermore, the medical images used in embodiments 1 to 4 are preferably ordinary light images obtained by irradiating light in the white frequency band or by irradiating light in multiple wavelength bands as the aforementioned white frequency band.

[0113] Furthermore, the medical images used in embodiments 1 to 4 are preferably special light images obtained by irradiating light with a specific wavelength band. The specific wavelength band is preferably a band narrower than the white band. The specific wavelength band is preferably included in the blue or green band of the visible region. The peak value of the specific wavelength band is preferably in the range of 390 nm to 450 nm or 530 nm to 550 nm.

[0114] The specific wavelength band is preferably included in the red band of the visible region. The peak value of the specific wavelength band is preferably in the range of 585nm to 615nm or 610nm to 730nm.

[0115] The peak value of a specific wavelength band is preferably located within the frequency band where the absorption coefficients of oxyhemoglobin and deoxyhemoglobin differ. The peak value of a specific wavelength band is preferably in the range of 400±10nm, 440±10nm, 470±10nm, or 600nm~750nm.

[0116] Furthermore, the medical images used in embodiments 1 to 4 are preferably in vivo images of a living organism. The in vivo images preferably contain information about the fluorescence emitted by a fluorescent substance within the living organism. The fluorescence is preferably obtained by irradiating the living organism with excitation light having a peak value of 390 to 470 nm.

[0117] Furthermore, the medical images used in embodiments 1 to 4 are preferably in vivo images of living organisms. The specific wavelength band is preferably an infrared wavelength band. The peak value of the specific wavelength band is preferably in the range of 790 nm to 820 nm or 905 nm to 970 nm.

[0118] Furthermore, the image acquisition unit 54 for acquiring medical images used in embodiments 1 to 4 preferably includes a special light image acquisition unit. This special light image acquisition unit acquires a special light image with a signal having a specific wavelength band from a normal light image obtained by irradiating light in the white frequency band or light in multiple wavelength bands as light in the white frequency band. The medical image is preferably a special light image. The signal having the specific wavelength band is preferably obtained by calculation based on the RGB or CMY color information contained in the normal light image.

[0119] Furthermore, the medical images used in embodiments 1 to 4 are preferably characteristic images. The characteristic image generation unit that generates the characteristic image preferably generates the characteristic image by calculation based on at least one of a general light image obtained by irradiating light in the white frequency band or light in multiple wavelength frequency bands as the aforementioned white frequency band, and a special light image obtained by irradiating light in a specific wavelength frequency band.

[0120] Explanation of symbols

[0121] 10-Endoscopic system (medical image processing device), 12-Endoscope, 12a-Insertion section, 12b-Operating section, 12c-Bend section, 12d-Tip section, 12e-Angle knob, 12f-Forceps inlet, 13-Zoom operating section, 14-Light source device, 16-Processor device, 18-Display, 19-Control console, 20-Light source section, 22-Light source control section, 30a-Illumination optical system, 30b-Camera optical system, 41-Light guide, 45-Illumination lens, 46-Objective lens, 47-Zoom lens, 48-Image sensor, 52-Control section, 54-Image acquisition section, 56-DSP (Digital Signal Processor) Processor (Digital Signal Processor), 58-Noise Reduction Unit, 59-Conversion Unit, 61-Image Processing Unit, 66-Display Control Unit, 68-Recording Control Unit, 69-Storage Memory, 70-Region Detection Unit, 72-Notification Information Generation Unit, 74-Bounding Box, 76-Icons, 80-Image Analysis Unit for Error Detection, 81-Observation Status Acquisition Unit, 82-Identification Processing Unit, 86-Disposal Equipment Detection Unit, 88-Identification Evaluation Value, 90-Drag Bar.

Claims

1. A medical image processing apparatus having: an image acquisition section that acquires a medical image containing an object; a display section that displays the medical image in a first display region; a display control section that performs control to display notification information for notifying a user in the display section or control to not display the notification information in the display section; and a region-of-interest detection section that detects a region-of-interest from the medical image, the display control section performs control to display the notification information in a second display region different from the first display region or control to change the notification information being displayed to not be displayed, the display control section performs control to display the notification information in the first display region when a first condition is satisfied, the display control section performs control to change the notification information being displayed in the first display region to not be displayed and to display the notification information in the second display region when a second condition different from the first condition is satisfied, the first condition is a case where a position of the region-of-interest departs from a center range of the first display region, and the second condition is a case where the position of the region-of-interest enters the center range of the first display region.

2. The medical image processing apparatus according to claim 1, wherein the first condition is a case where combined information combining at least two of an area of the region-of-interest, a position of the region-of-interest, an amount of change of the region-of-interest over time, a luminance value of the region-of-interest, and a first display time of displaying the notification information in the first display region satisfies a combined condition, and the second condition is a case where the combined information does not satisfy the combined condition.

3. The medical image processing apparatus according to claim 2, wherein the first condition is a case where area-luminance combined information combining the area of the region-of-interest and the luminance value of the region-of-interest satisfies an area-luminance combined condition, and the second condition is a case where the area-luminance combined information does not satisfy the area-luminance combined condition.

4. The medical image processing apparatus according to claim 1, wherein the display control section performs control to overlap and display the notification information and the medical image in the first display region.

5. The medical image processing apparatus according to claim 1, wherein the notification information is notification information for notifying information related to the region-of-interest.

6. The medical image processing apparatus according to claim 1, wherein the notification information is presence or absence of a lesion, a kind of a lesion, or a detection state of a treatment tool.

7. The medical image processing apparatus according to claim 1, wherein the notification information is saved overlapping the medical image.

8. The medical image processing apparatus according to claim 1, wherein ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The medical image processing apparatus has a feature amount image generation section that generates a feature amount image by calculation based on at least one of a normal light image obtained by irradiating light of a white color band or light of a plurality of wavelength bands as the light of the white color band and a special light image obtained by irradiating light of a specific wavelength band, The medical image is the feature amount image.

9. The medical image processing apparatus according to claim 2, wherein The amount of change in the region of interest over time is an amount of movement of the region of interest moving within a certain time.

10. The medical image processing apparatus according to claim 1, wherein The medical image processing apparatus has an error information detection image analysis section that detects error information from the medical image, The error information is blur appearing on the medical image or dirt of the subject, The notification information includes information for notifying the error information.

11. The medical image processing apparatus according to claim 2, wherein The amount of change in the region of interest over time is an amount of movement of the region of interest moving within a certain time, The medical image processing apparatus further has an error information detection image analysis section that detects error information from the medical image, The error information is blur appearing on the medical image or dirt of the subject, The notification information includes information for notifying the error information.

Images