Medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program

Abstract

An object of the present application is to provide a medical image processing apparatus, an endoscope system, a medical image processing method, and a medical image processing program capable of reducing the possibility of missing a region of interest and suppressing unnecessary voice output. A medical image processing apparatus according to an aspect of the present application includes a processor configured to execute: an image acquisition process configured to acquire a time-series of medical images; a region of interest detection process configured to detect a region of interest from the acquired medical images; a display control process configured to cause a display device to display the medical images; a first notification process configured to cause the display device to superimposedly display the medical images and information related to the detected region of interest when the region of interest is detected by the region of interest detection process; and a second notification process configured to output a voice from a voice output device when the region of interest is detected by the region of interest detection process, the second notification process being executed later than the first notification process.

Description

Technical Field

[0001] This invention relates to medical image processing apparatus, endoscope system, medical image processing method and medical image processing program, and in particular to technology for notifying the detection results of areas of interest. Background Technology

[0002] As an aid for doctors and other users in observing or diagnosing medical images, techniques for notifying medical image processing devices of detection results for regions of interest are known. For example, Patent Document 1 describes a technique that detects regions of interest by selecting one of a plurality of regions of interest detection units and notifies (displays) the results. Furthermore, Patent Document 2 describes a technique for notifying detection or identification results using voice.

[0003] Previous technical documents

[0004] Patent documents

[0005] Patent Document 1: Publication No. W2017 / 081976

[0006] Patent Document 2: Japanese Patent Application Publication No. 2020-69300 Summary of the Invention

[0007] The technical problem to be solved by the invention

[0008] In applications where AI (Artificial Intelligence) is used to detect areas of interest, primarily lesions, from medical images such as endoscopic or ultrasound images and notify the user, immediate notification is required to prevent lesions from being missed. However, AI-driven automated detection can produce false positives, leading to erroneous notifications and user frustration. This is particularly problematic when voice-based notifications are frequently erroneously output, tending to be more cumbersome for users compared to display-based notifications. Furthermore, prior art, such as patent documents 1 and 2 mentioned above, struggles to reduce the likelihood of missed areas of interest and suppress unnecessary voice output.

[0009] The present invention was made in view of the following circumstances, and its object is to provide a medical image processing apparatus, endoscope system, medical image processing method and medical image processing program that can reduce the possibility of missing areas of interest and suppress unnecessary voice output.

[0010] means for solving technical problems

[0011] To achieve the above objectives, the medical image processing apparatus according to the first aspect of the present invention includes a processor, wherein the processor performs: image acquisition processing to acquire time-series medical images; region of interest detection processing to detect regions of interest from the acquired medical images; display control processing to cause a display device to display the medical images; a first notification processing to cause the display device to overlay the medical images and information related to the detected region of interest when a region of interest is detected by the region of interest detection processing; and a second notification processing to output voice from a voice output device when a region of interest is detected by the region of interest detection processing, and to execute the second notification processing after the first notification processing.

[0012] In the medical image processing apparatus described in the first party, the processor performs a first notification process (display to a display device) when it detects a region of interest. This achieves the effect of preventing the region of interest from being missed. Furthermore, from the viewpoint of preventing missed views, it is preferable that the processor performs the first notification process immediately (with the shortest possible delay) when it detects a region of interest, but allow for inherent delays in the processing within the apparatus and delays caused by the intermittent nature of medical image acquisition.

[0013] In the detection of regions of interest, false positives tend to occur mostly momentarily and rarely persist. Therefore, suppressing the voice output when a region of interest is momentarily detected can reduce the annoyance users experience due to voice output caused by false positives. In the first embodiment, from the viewpoint of this, the processor executes the second notification process for outputting voice after the first notification process. That is, even if a region of interest is detected during the period after the first notification process (screen display), the processor does not perform voice output, thus suppressing voice output based on momentary false positives during this period.

[0014] The processor (medical image processing device) can be configured, either by user specifications (delay time or standby time), or not by user specifications. Users can specify the delay time by considering a balance between suppressing false positives in voice output and enhancing the effectiveness of the voice notification.

[0015] According to the medical image processing device involved in the first method, the possibility of missing the area of ​​interest can be reduced and unnecessary voice output (voice output based on transient false positives) can be suppressed through such first and second notification processing.

[0016] Furthermore, in the first method and the following methods, "acquisition of time-series medical images" includes sequentially acquiring multiple medical images captured at a defined frame rate. The acquisition may be real-time or not.

[0017] The medical image processing device involved in the first approach can be implemented as a processor part of a medical image processing system, but is not limited to this approach. Furthermore, "medical image" refers to an image obtained as a result of photographing or measuring a living organism such as a human body for purposes such as diagnosis, treatment, or measurement; examples include endoscopic images, ultrasound images, CT images (Computed Tomography), and MRI images (Magnetic Resonance Imaging). Medical images are also called medical images. Additionally, in the first approach and the following approaches, the "Region of Interest" can be a lesion area or potential lesion area, an organ or blood vessel, an area where treatment has been performed, or an area where treatment instruments are captured in the medical image. The "Region of Interest" is sometimes also referred to as the "Region of Concern."

[0018] The medical image processing device involved in the second method, in the first method, when a region of interest is detected in the region of interest detection process and then detected again after a first period, the processor performs second notification processing; when no region of interest is detected after the first period, the processor does not perform second notification processing. When a region of interest is detected and then detected again after the first period, it can be considered as continuous detection, with a low possibility of transient false positives, thus enabling second notification processing (voice output). Furthermore, in the second method, the processor can set the value of the "first period" according to the purpose or object of observation, or the user's specification.

[0019] The third approach to medical image processing apparatus, in the first or second approach, involves a processor that overlaps information with the location of the region of interest in the medical image during the first notification process. The third approach specifically specifies the method of the first notification process. The processor can display information within the region of interest in the medical image, or it can display information around the periphery of the region of interest. Alternatively, the processor can display information outside the medical image display area on the display screen of the display device.

[0020] In the fourth method, the medical image processing device, in any of the first to third methods, performs a detection count calculation process, calculating the number of consecutive detections for the region of interest detected by the region of interest detection process. When the number of consecutive detections exceeds a predetermined number, a second notification process is executed. When the number of consecutive detections exceeds the predetermined number, it can be considered that "it is continuous detection, and the possibility of instantaneous false positives is low," and therefore the second notification process can be executed as in the fourth method. The processor can set the "predetermined number" according to user specifications, or it can set it without user specifications.

[0021] In the fifth method, the medical image processing apparatus, in addition to the fourth method, further includes the following steps performed by the processor: feature retention processing, which retains the feature quantities of the detected region of interest; and similarity determination processing, which compares the feature quantities of the first region of interest detected in a medical image taken at a first moment with the feature quantities of the second region of interest detected in a second medical image taken at a moment earlier than the first moment of retention, to determine the similarity between the first and second regions of interest; and in the detection count calculation processing, it calculates the number of consecutive detections of the first region of interest based on the determination result in the similarity determination processing. As described in the fifth method, by considering the similarity of the regions of interest in calculating the number of consecutive detections, the second notification processing can be performed more appropriately. Furthermore, "retention" can be a temporary record (storage) or a non-temporary record. Additionally, regarding the first and second regions of interest, "feature quantities" can be, for example, type, location, shape, size, or color, but are not limited to these examples.

[0022] In the sixth method, the medical image processing device, in the fifth method, when the processor determines that the first region of interest and the second region of interest are the same in the sameness determination process, calculates the number of consecutive detections of the second region of interest as the number of consecutive detections of the first region of interest in the detection count process after adding the number of consecutive detections recorded for the second region of interest. The sixth method specifies the specific method for calculating the number of consecutive detections.

[0023] In the seventh method, the medical image processing apparatus, in the fifth or sixth method, compares the feature values ​​held by the feature value holding process with respect to a time prior to a period earlier than the first time value with the feature values ​​of the first region of interest during the sameness determination process, thereby determining sameness. In the seventh method, the "time prior to a period earlier than the first time value" corresponds to the "second time value" in the fifth method. This "determined period" can be set to account for increased computational costs or decreased accuracy in the sameness determination.

[0024] In the medical image processing apparatus involved in the eighth method, in any of the first to seventh methods, after the processor causes the voice output device to output voice during the second notification process, it does not output voice for a defined period. The eighth method takes into account that users may find frequent or prolonged voice output tedious, and therefore sets a pause period for voice output. The processor can set the "defined period" according to user specifications, or it can set it independently of user specifications.

[0025] In any of the first to eighth methods of the medical image processing apparatus involved in the ninth method, the processor changes the mode of the first notification processing based on the voice output state in the second notification processing during the first notification processing. In the ninth method, for example, when performing the second notification processing (i.e., when the region of interest is continuously detected and the possibility of transient false positives is low), the processor can increase the recognition power of information overlaid by the first notification processing. By changing the mode of the first notification processing in this way, it can notify that continuous detection is occurring. Furthermore, the processor can change the mode of the first notification processing simultaneously with the second notification processing (voice output), or it can change it before or after a specific time.

[0026] In any of the first to ninth methods, the medical image processing apparatus of the tenth method involves the processor overlaying at least one of characters, graphics, and symbols as information (information related to the detected region of interest) during the first notification process. The tenth method specifies the particular manner in which the information overlaid with the medical image is displayed. The processor may also overlay information corresponding to the feature values ​​of the region of interest.

[0027] To achieve the above objectives, the endoscopic system according to the eleventh aspect of the present invention comprises: a medical image processing device as described in any of the first to tenth aspects; an endoscopic observer inserted into the subject and having a camera unit for capturing medical images; a display device; and a voice output device. The endoscopic system according to the eleventh aspect, by including the medical image processing device described in any of the first to tenth aspects, can reduce the possibility of missing areas of interest and suppress unnecessary voice output. Furthermore, the endoscopic system according to the eleventh aspect may also include a light source device. This light source device can use ordinary light (white light), special light (narrowband light, etc.), and combinations thereof as observation light. Preferably, the light source device irradiates observation light with different wavelength bands depending on the organ or site, the purpose of observation, or the type of object being observed.

[0028] To achieve the above objectives, the medical image processing method according to the twelfth aspect of the present invention is executed by a medical image processing apparatus equipped with a processor, wherein the processor executes: an image acquisition step, acquiring a time-series medical image; a region of interest detection step, detecting a region of interest from the acquired medical image; a display control step, causing a display device to display the medical image; a first notification step, when a region of interest is detected in the region of interest detection step, causing the display device to overlay the medical image and information related to the detected region of interest; and a second notification step, when a region of interest is detected in the region of interest detection step, outputting voice from a voice output device, performing the second notification step after the first notification step. According to the twelfth aspect, the possibility of missing a region of interest can be reduced and unnecessary voice output can be suppressed in the same way as in the first aspect. The medical image processing method according to the twelfth aspect may also include the same structure as the second to tenth aspects.

[0029] To achieve the above objectives, the medical image processing program according to the thirteenth aspect of the present invention causes a medical image processing apparatus equipped with a processor to execute a medical image processing method, wherein the medical image processing method includes: an image acquisition step for acquiring a time-series medical image; a region of interest detection step for detecting a region of interest from the acquired medical image; a display control step for causing a display device to display the medical image; a first notification step for causing the display device to overlay the medical image and information related to the detected region of interest when a region of interest is detected in the region of interest detection step; and a second notification step for outputting voice from a voice output device when a region of interest is detected in the region of interest detection step, wherein the second notification step is performed after the first notification step. According to the thirteenth aspect, the possibility of missing a region of interest and the suppression of unnecessary voice output can be reduced in the same way as in the first and twelfth aspects. In addition, the medical image processing program according to the thirteenth aspect may also be a program that performs the same processing as in the second to tenth aspects. Furthermore, a non-transitory recording medium containing computer-readable code of the program of these aspects can also be cited as an aspect of the present invention.

[0030] Invention Effects

[0031] As described above, the medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program according to the present invention can reduce the possibility of missing areas of interest and suppress unnecessary voice output. Attached Figure Description

[0032] Figure 1 This is an external view of the endoscope system according to the first embodiment.

[0033] Figure 2 This is a diagram showing the main structural components of an endoscope system.

[0034] Figure 3 It is a block diagram representing the functional structure of a processor.

[0035] Figure 4 This is a diagram showing the structure of the detection section in the area of ​​interest.

[0036] Figure 5 This is a diagram illustrating an example of the layered structure of a detector.

[0037] Figure 6 This is a diagram illustrating the case of filter-based convolution processing.

[0038] Figure 7 This is a flowchart showing the sequence of the medical image processing methods involved in the first embodiment.

[0039] Figure 8 This is an example image showing the settings screen for processing conditions.

[0040] Figure 9 This is a diagram illustrating an example of overlapping display (first notification processing).

[0041] Figure 10 This is a flowchart showing the details of a voice-based notification (second notification processing).

[0042] Figure 11 This is a diagram representing a specific example (1) of voice output.

[0043] Figure 12 This is a diagram representing a specific example (2) of voice output.

[0044] Figure 13 This is a diagram representing a specific example (3) of voice output.

[0045] Figure 14 This is a diagram representing a specific example (4) of voice output.

[0046] Figure 15 This is a diagram representing a specific example (5) of voice output.

[0047] Figure 16 This is a diagram representing a specific example (6) of voice output. Detailed Implementation

[0048] Hereinafter, with reference to the accompanying drawings, the embodiments of the medical image processing apparatus, endoscope system, medical image processing method and medical image processing program involved in the present invention will be described in detail.

[0049] <First Implementation Method>

[0050] <Structure of an Endoscopic System>

[0051] Figure 1 This is an external view of the endoscope system 10 (medical image processing device, endoscope system) according to the first embodiment. Figure 2 This is a block diagram showing the main structural components of the endoscope system 10. For example... Figure 1 , 2 As shown, the endoscope system 10 comprises an endoscope observer 100 (endoscope observer, camera device), a medical image processing unit 200 (medical image processing device, processor), a light source device 300 (light source device), and a monitor 400 (display device). An external device (determination device) that uses electromagnetic waves, ultrasound, or magnetism to determine the state of the endoscope observer 100 may also be connected to the endoscope system 10.

[0052] <Structure of the Endoscopic Observation Device>

[0053] The endoscope observation device 100 includes a handheld operating unit 102 and an insertion unit 104 connected to the handheld operating unit 102. The operator (user) holds the handheld operating unit 102 and operates it to insert the insertion unit 104 into the body of the subject for observation. Additionally, the handheld operating unit 102 is equipped with an air / water supply button 141, a suction button 142, function buttons 143 assigned various functions, and a photography button 144 for receiving photographic instructions (still image, moving image). The insertion unit 104, starting from the handheld operating unit 102 side, consists of a flexible part 112, a curved part 114, and a rigid front end 116. Specifically, the curved part 114 is connected to the base of the rigid front end 116, and the flexible part 112 is connected to the base of the curved part 114. The handheld operating unit 102 is connected to the base of the insertion unit 104. The user can bend the bending part 114 and change the direction of the front rigid part 116 by operating the hand-held operating unit 102. The front rigid part 116 is equipped with a photographic optical system 130, an illumination unit 123, a clamping mouth 126, etc. (see reference). Figure 1 , 2 ).

[0054] During observation and handling, the operation unit 208 (refer to...) Figure 2The operation allows for the illumination of white light (ordinary light) and / or narrowband light (special light: such as one or more of red, green, blue, and purple narrowband light) from the illumination lenses 123A and 123B of the illumination unit 123. Additionally, by operating the air / water supply button 141, cleaning water can be released from a water supply nozzle (not shown) to clean the imaging lens 132 (imaging lens, imaging unit) and illumination lenses 123A and 123B of the imaging optical system 130. A (not shown) conduit communicates with the forceps port 126 opened at the front rigid part 116, through which a (not shown) treatment instrument for tumor removal, etc., is inserted. By appropriately moving the instrument forward and backward, the necessary treatment of the subject can be performed.

[0055] like Figure 1 , 2 As shown, a photographic lens 132 (image-taking unit) is disposed on the front end face 116A of the front end rigid part 116. A CMOS (Complementary Metal-Oxide Semiconductor) type image sensor 134 (image sensor, image-taking unit), a drive circuit 136, and an AFE 138 (AFE: Analog Front End, image-taking unit) are disposed inside the photographic lens 132, and image signals are output from these elements. The image sensor 134 is a color image sensor, comprising multiple pixels composed of multiple light-receiving elements arranged in a matrix in a specific pattern (Bayer arrangement, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.) (two-dimensional arrangement). Each pixel of the image sensor 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The photographic optical system 130 can generate a color image from pixel signals of red, green, and blue, or it can generate an image from pixel signals of any one or two of the three colors of red, green, and blue. Furthermore, in the first embodiment, the image sensor 134 is described as a CMOS type image sensor, but the image sensor 134 may also be a CCD (Charge Coupled Device) type. Additionally, each pixel of the image sensor 134 may also include a violet color filter corresponding to the violet light source 310V, and / or an infrared filter corresponding to an infrared light source.

[0056] The optical image of the subject is imaged on the light-receiving surface (image-receiving surface) of the imaging element 134 through the photographic lens 132 and converted into an electrical signal. This signal is then output to the medical image processing unit 200 via a signal cable (not shown) and converted into an image signal. As a result, the endoscopic image is displayed on a monitor 400 connected to the medical image processing unit 200.

[0057] Additionally, on the front end face 116A of the front rigid portion 116, adjacent to the photographic lens 132, illumination lenses 123A and 123B of the illumination portion 123 are provided. Within the illumination lenses 123A and 123B, the emission end of a light guide 170 (described later) is disposed. This light guide 170 is inserted into the insertion portion 104, the hand-operated portion 102, and the universal cable 106. The incident end of the light guide 170 is disposed within the light guide connector 108.

[0058] The handheld operation unit 102 may also include an observer information recording unit (not shown) that records individual information (individual information, observer information) of the endoscope observer 100. Individual information may include, for example, the type of endoscope observer 100 (whether it is direct or lateral viewing), model, individual identification number, and characteristics of the optical system (viewing angle, deformation, etc.). The processor 210 (observer information acquisition unit, individual information acquisition unit) can acquire this individual information and use it for medical image processing. Furthermore, the observer information recording unit may also be provided on the optical connector 108.

[0059] In the endoscope system 10, the subject is sequentially photographed at a determined frame rate using the endoscope observer 100 with the above-described structure (via the camera unit and image acquisition unit 220 (see reference)). Figure 3 (Controlled by the endoscope), it can sequentially acquire time-series medical images. The user observes the patient while inserting or removing the endoscope 100 (insertion part 104) into the biological body of the subject.

[0060] <Structure of the Light Source Device>

[0061] like Figure 2 As shown, the light source device 300 comprises a light source 310 for illumination, an aperture 330, a condenser lens 340, and a light source control unit 350, etc., so that the observation light is incident along the light guide 170. The light source 310 is equipped with a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V, which respectively illuminate narrow bands of red, green, blue, and violet light, and can illuminate narrow bands of red, green, blue, and violet light. The illuminance of the observation light generated by the light source 310 is controlled by the light source control unit 350, which can change (increase or decrease) the illuminance of the observation light and stop the illumination as needed.

[0062] The light source 310 can emit narrowband light of red, green, blue, and violet in any combination. For example, it can emit narrowband light of red, green, blue, and violet simultaneously and use white light (ordinary light) as observation light, or it can emit any one or two types of light to illuminate narrowband light (special light). The light source 310 may also be equipped with an infrared light source that illuminates infrared light (an example of narrowband light). Alternatively, it can use a light source that illuminates white light and filters that allow white light and each narrowband light to pass through to illuminate either white light or narrowband light as observation light.

[0063] <wavelength band of the light source>

[0064] Light source 310 can be a light source that generates light in a white frequency band, or a light source that generates light in multiple wavelength bands as white frequency band light, or a light source that generates light in a specific wavelength band narrower than the white wavelength band. The specific wavelength band can be the blue or green band within the visible range, or the red band within the visible range. When the specific wavelength band is the blue or green band within the visible range, it can also include wavelength bands between 390nm and 450nm, or between 530nm and 550nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band between 390nm and 450nm or between 530nm and 550nm. Alternatively, when the specific wavelength band is the red band within the visible range, it can also include wavelength bands between 585nm and 615nm, or between 610nm and 730nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band between 585nm and 615nm or between 610nm and 730nm.

[0065] The aforementioned specific wavelength bands can also include wavelength bands with different absorption coefficients in oxyhemoglobin and deoxyhemoglobin, and the light in the specific wavelength band has a peak wavelength in the wavelength band with different absorption coefficients in oxyhemoglobin and deoxyhemoglobin. In this case, the specific wavelength band can also include wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0066] In addition, the wavelength band of the light generated by the light source 310 may also include wavelength bands of 790nm to 820nm or 905nm to 970nm, and the light generated by the light source 310 has a peak wavelength in the wavelength bands of 790nm to 820nm or 905nm to 970nm.

[0067] Alternatively, the light source 310 may also be equipped with an excitation light source with a peak illumination of 390 nm to 470 nm. In this case, medical images (medical images, in vivo images) containing information about the fluorescence emitted by fluorescent substances within the subject (organism) can be acquired. When acquiring fluorescence images, fluorescent dyes (fluorescein, acridine orange, etc.) can also be used.

[0068] The type of light source 310 (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), wavelength, and presence or absence of filters are preferably configured according to the type, part, organ of the subject, and purpose of observation. Furthermore, during observation, it is preferable to combine and / or switch the wavelengths of the observation light according to the type, part, organ of the subject, and purpose of observation. When switching wavelengths, for example, the wavelength of the illuminated light can be switched by rotating a disc-shaped filter (rotating color filter) positioned in front of the light source and equipped with filters that transmit or block light of specific wavelengths.

[0069] Furthermore, the imaging element used in implementing this invention is not limited to a color imaging element like imaging element 134, which has a color filter for each pixel; a monochrome imaging element can also be used. When using a monochrome imaging element, the wavelengths of the observation light can be switched sequentially, and images can be taken in face order (color order). For example, the wavelengths of the emitted observation light can be switched sequentially between (violet, blue, green, red), or broadband light (white light) can be irradiated and the wavelengths of the emitted observation light can be switched using a rotating color filter (red, green, blue, violet, etc.). Alternatively, one or more narrowband lights (green, blue, violet, etc.) can be irradiated and the wavelengths of the emitted observation light can be switched using a rotating color filter (green, blue, violet, etc.). The narrowband light can also be infrared light with two or more different wavelengths.

[0070] By using the optical guide connector 108 (reference) Figure 1 , 2 The light source device 300 is connected to the observation light source device 300. The observation light emanating from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and then illuminates the observation area from the illumination lenses 123A and 123B.

[0071] <Structure of the Medical Image Processing Unit>

[0072] based on Figure 2The structure of the medical image processing unit 200 will be described. The medical image processing unit 200 receives the image signal output from the endoscope observer 100 via the image input controller 202, performs the necessary image processing by the processor 210 (image acquisition unit 220: processor, computer, medical image processing device), and outputs the signal via the video output unit 206. Thus, the observed image (medical image, medical image) is displayed on the monitor 400 (display device). The communication control unit 205 performs communication control with a hospital information system (HIS) or a hospital LAN (Local Area Network) (not shown), and / or external systems or networks. The recording unit 207 (recording device) records information such as images of the examined body (endoscopic images, medical images, medical images), site information, and test results. The voice processing unit 209, under the control of the processor 210, can output messages (voice) related to test results or notification processing (second notification processing) from the speaker 209A (voice output device).

[0073] In addition, ROM211 (ROM: Read Only Memory) is a non-volatile storage element (non-temporary recording medium) that stores computer-readable code that enables the processor 210 to execute various image processing methods. RAM212 (RAM: Random Access Memory) is a storage element used for temporary storage during various processing operations, and can also be used as a buffer during image acquisition.

[0074] Furthermore, the user can issue execution instructions for medical image processing or specify the necessary conditions via the operation unit 208, and the display control unit 232 (see reference) Figure 3 This enables the monitor 400 to display the screen showing the actions taken when these instructions were given (e.g., see reference). Figure 8 (or the detection results of the area of ​​interest, etc.)

[0075] <Processor Function>

[0076] Figure 3 This is a block diagram showing the functional structure of processor 210. Processor 210 includes an image acquisition unit 220, a region of interest detection unit 222, a detection count calculation unit 226, a feature calculation unit 228, a similarity determination unit 230, a display control unit 232, a first notification unit 234, a second notification unit 236, a recording control unit 238, and a communication control unit 240. Additionally, as... Figure 4As shown, the region of interest detection unit 222 includes a detector 223 and a switching control unit 224. The detector 223 can be composed of multiple detectors with different observation sites or organs, detection algorithms, etc. Figure 4 The illustrated configuration includes a pharyngeal detector 223A, an esophageal detector 223B, a gastric detector 223C, and a duodenal detector 223D. The switching control unit 224 can switch the detectors whose detection results are displayed on the monitor 400 (display device) based on the analysis results of the endoscopic images (location or organ, line of sight, etc.), or it can switch based on the camera information (information indicating the position and / or orientation of the camera device) obtained from the aforementioned external device (determination device). Furthermore, the processor 210 can activate multiple detectors and display the detection results of some of them, or it can activate only the detectors displaying the detection results.

[0077] The processor 210 can utilize the aforementioned functions to calculate feature quantities of medical images, process for emphasizing or reducing specific frequency band components, emphasize specific objects (areas of interest, blood vessels at desired depth, etc.) or make them less prominent. The processor 210 may also include a special light image acquisition unit, which acquires a special light image with information of a specific wavelength band based on a normal light image obtained from light illuminating a white frequency band or light of multiple wavelength bands that are white frequency bands. In this case, the signal of the specific wavelength band can be obtained by calculation based on the color information of RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) contained in the normal light image. Furthermore, the processor 210 may also include a feature quantity image generation unit to acquire and display a feature quantity image as a medical image (medical image). This feature quantity image generation unit generates the feature quantity image by calculation based on at least one of a normal light image obtained from light illuminating a white frequency band or light of multiple wavelength bands that are white frequency bands and a special light image obtained from light illuminating a specific wavelength band.

[0078] In addition, the image acquisition unit 220 (processor) may acquire, as the medical image, an endoscopic image (medical image) captured with observation light in a wavelength band corresponding to the site indicated by the site information, and the display control unit 232 causes the monitor 400 (display device) to display the recognition result of the medical image captured with the observation light in the wavelength band. For example, in the case of the stomach, an image captured with white light (ordinary light) can be used for detection (recognition), and in the case of the esophagus, an image captured with special light (blue narrow-band light) such as BLI (Blue Laser Imaging: registered trademark) can be used for detection (recognition). The image acquisition unit 220 may also acquire, depending on the site, an image captured with special light such as LCI (Linked Color Imaging: registered trademark) that has been subjected to image processing (in the case of LCI, the chroma difference or hue difference of colors close to the mucosal color is expanded).

[0079] The details of the medical image processing using the above functions will be described later.

[0080] <Detector Using Trained Model>

[0081] The aforementioned detector may be configured using a trained model constructed by machine learning (a model trained using an image set formed of images obtained by photographing a living organism) such as CNN (Convolutional Neural Network) and SVM (SupportVector Machine). Hereinafter, the layer structure when the detector 223 (the detector 223A for pharynx to the detector 223D for duodenum) is constructed by CNN will be described.

[0082] <Example of CNN Layer Structure>

[0083] Figure 5 is a diagram showing an example of the layer structure of the detector 223. In Figure 5 the example shown in part (a), the detector 223 includes an input layer 250, an intermediate layer 252, and an output layer 254. The input layer 250 receives the endoscopic image (medical image) acquired by the image acquisition unit 220 and outputs feature quantities. The intermediate layer 252 includes a convolution layer 256 and a pooling layer 258, receives the feature quantities output by the input layer 250 and calculates other feature quantities. These layers have a structure formed by connecting a plurality of "nodes" with "edges", and store a plurality of weight parameters. The values of the weight parameters change as the learning progresses. As Figure 5In the example shown in part (b), detector 223 may also include fully connected layers 260. The layer structure of detector 223 is not limited to the case where convolutional layers 256 and pooling layers 258 are repeated layer by layer, but may also include multiple layers of any kind (e.g., convolutional layers 256) consecutively. In addition, multiple fully connected layers 260 may also be included consecutively.

[0084] <Processing in the intermediate layer>

[0085] Intermediate layer 252 calculates feature values ​​through convolution and pooling operations. The convolution operation performed by convolutional layer 256 is a process of obtaining feature maps by using the convolution operation of filters, responsible for feature extraction such as edge detection from the image. By using the convolution operation of the filter, a channel (one) of "feature map" is generated for each filter. The size of the "feature map" is scaled down through convolution, becoming smaller as convolution occurs in each layer. The pooling operation performed by pooling layer 258 is a process of shrinking (or enlarging) the feature map output by the convolution operation to form a new feature map, providing robustness so that the extracted features are not affected by parallel shifts, etc. Intermediate layer 252 may consist of one or more layers performing these processes.

[0086] Figure 6 This diagram illustrates the convolutional processing performed by the filter. In the initial (first) convolutional layer of intermediate layer 252, a convolution operation is performed between an image set consisting of multiple medical images (a learning image set during learning, and a recognition image set during detection and other recognition processes) and filter F1. The image set consists of N images (N channels) with image dimensions H and W. When a normal light image is input, the images constituting the image set are images from the three channels R (red), G (green), and B (blue). Regarding filter F1, which performs the convolution operation with this image set, since the image set has N channels (N images), for example, in the case of a filter with size 5 (5×5), the filter size becomes 5×5×N. By using the convolution operation of filter F1, a "feature map" for one channel (one image) is generated for each filter F1. Regarding filter F2 used in the second convolutional layer, for example, in the case of a filter with size 3 (3×3), the filter size becomes 3×3×M.

[0087] Similar to the first convolutional layer, filters F2 to F3 are used in the second to nth convolutional layers. n The convolution operation. The reason why the size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer is because it is scaled down by the convolutional or pooling layers up to the previous one.

[0088] In the convolutional layers near the input side of the intermediate layer 252, low-order feature extraction (edge ​​extraction, etc.) is performed, while higher-order feature extraction (features related to the shape and structure of objects, etc.) is performed closer to the output side. Furthermore, in cases where segmentation is performed for measurement purposes, the feature maps are scaled up by the latter half of the convolutional layers, resulting in a "feature map" of the same size as the input image set in the final convolutional layer. On the other hand, when performing object detection, outputting positional information is sufficient, so scaling up is not necessary.

[0089] In addition to the convolutional layer 256 and the pooling layer 258, the intermediate layer 252 may also include a layer for batch normalization. Batch normalization is a process that normalizes the distribution of data in small batches during learning, and plays a role in accelerating learning, reducing dependence on initial values, and suppressing overlearning.

[0090] Processing in the output layer

[0091] Output layer 254 is a layer that detects the location of regions of interest in the input medical image (ordinary light image, special light image) based on the feature data output from intermediate layer 252 and outputs the results. When segmenting, output layer 254 uses the "feature map" obtained from intermediate layer 252 to determine the location of the captured region of interest in the image at the pixel level. That is, it can detect whether each pixel of the endoscopic image belongs to a region of interest and output its detection result. On the other hand, when performing object detection, pixel-level judgment is not required; output layer 254 outputs the location information of the object.

[0092] Output layer 254 can also be a layer that performs lesion-related identification (classification) and outputs the identification results. For example, output layer 254 divides endoscopic images into three categories: "tumorous," "non-tumorous," and "other." As the identification result, it can be set as three scores corresponding to "tumorous," "non-tumorous," and "other" (the sum of the three scores is 100%) for output. If the classification can be clearly defined based on the three scores, the classification result can also be output. In addition, when outputting identification results, intermediate layer 252 or output layer 254 may include fully connected layers as the last layer or multiple layers (see reference). Figure 5 (b) can also exclude fully connected layers.

[0093] Output layer 254 can also be a layer that outputs the measurement results of the region of interest. In the case of measurement using a CNN, the region of interest, which will become the object, can be segmented as described above, and then measured by the processor 210 or the like based on the results. Alternatively, the measurement values ​​of the region of interest, which will become the object, can be directly output from the detector 223. In the case of directly outputting the measurement values, since the measurement values ​​are learned from the image itself, it becomes a regression problem of the measurement values.

[0094] Preferably, when using the CNN with the above structure, during the learning process, the result output by the output layer 254 is compared with the correct solution for the recognition of the image set to calculate the loss (error), and the weight parameters in the intermediate layer 252 are updated from the output layer to the input layer (error backpropagation) to reduce the loss.

[0095] <Recognition based on methods other than CNN>

[0096] Detector 223 can also perform detection using methods other than CNNs. For example, regions of interest can be detected based on the feature values ​​of pixels in the acquired medical image. In this case, detector 223 segments the target image into multiple rectangular regions, designates each segmented rectangular region as a local region, calculates the feature values ​​(e.g., hue) of pixels within each local region of the target image, and identifies local regions with specific hues as regions of interest. Similarly, detector 223 can also perform feature-based classification and measurement.

[0097] <Examples of detector structure modifications>

[0098] Each detector constituting detector 223 (pharyngeal detector 223A to duodenal detector 223D) can also be composed of multiple detectors (e.g., ordinary light detectors and special light detectors) corresponding to observation light of different wavelength bands. In this case, the ordinary light detector and the special light detector are preferably learned models constructed by machine learning using ordinary light images and special light images, respectively.

[0099] exist Figure 5 , Figure 6 The original text mainly described the structure of the detector, but in this invention, a classifier and a measuring device may also be provided instead of the detector. Furthermore, the detector, classifier, or measuring device may be categorized as either for general optical applications or for special optical applications.

[0100] <Implementation of the functions of various processors>

[0101] The functions of the processor 210 described above can be implemented using various processors and recording media. Among these various processors are, for example, general-purpose processors that execute software (programs) to implement various functions, namely CPUs (Central Processing Units). Additionally, among the various processors described above, there are processors specifically designed for image processing, such as GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), and other processors whose circuit structures can be modified after manufacturing, i.e., programmable logic devices (PLDs). When performing image processing as described in this invention, using a GPU structure is effective. Furthermore, processors with circuit structures specifically designed for performing specific processing, such as ASICs (Application Specific Integrated Circuits), i.e., dedicated circuits, are also included among the "various processors" described above.

[0102] The functions of each part can be implemented by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Alternatively, a single processor can implement multiple functions. As examples of a single processor performing multiple functions, firstly, there are forms such as computers, where a single processor is constructed using a combination of one or more CPUs and software, and that processor performs multiple functions. Secondly, there are forms such as System-on-Chips (SIC), where a single IC (Integrated Circuit) chip is used to implement the overall system functions. In this way, one or more of the above-mentioned processors are used as hardware structures to construct various functions. Furthermore, more specifically, the hardware structure of these various processors is a circuit composed of circuit elements such as semiconductor components. These circuits can also be circuits that implement the above functions using logical operations such as AND, product, NOT, XOR, and combinations thereof.

[0103] When the aforementioned processor or circuit executes the software (program), code that can be read by the computer executing the software (e.g., various processors, circuits, and / or combinations thereof constituting processor 210) is pre-stored in a non-transitory recording medium such as ROM 211 (Read Only Memory), and the computer refers to this software. The software pre-stored in the non-transitory recording medium includes a medical image processing program for executing the medical image processing method according to the present invention and data used during execution (data used in setting display mode and notification mode, weight parameters used in detector 223, etc.). The code may also be recorded in a non-transitory recording medium such as various optical magnetic recording devices or semiconductor memories instead of ROM 211. When processing using the software, RAM 212 (Random Access Memory) may be used as a temporary storage area, for example. Alternatively, data stored in EEPROM (Electrically Erasable and Programmable Read Only Memory) not shown may also be referenced. The recording unit 207 may also be used as the "non-transitory recording medium".

[0104] <Information recorded in the Records Department>

[0105] The recording unit 207 records endoscopic images (medical images), processed endoscopic images (medical images) after image processing, camera information (information indicating the camera position and / or camera direction of the endoscopic image), detection results, and processing conditions (conditions for conducting detection or notification). Other information may also be recorded simultaneously. The recording control unit 238 records this information in conjunction with each other.

[0106] <Medical Image Processing>

[0107] The medical image processing (execution of the medical image processing method and medical image processing program involved in the present invention) in the endoscope system 10 with the above-described structure will be described. Figure 7 This is a flowchart illustrating the sequence of the medical image processing method according to the first embodiment. The following description explains the case where the detector 223 detects the region of interest; the same processing can be performed during classification or measurement. Furthermore, the order of the following descriptions can be rearranged as needed.

[0108] <Initial Settings>

[0109] The processor 210 sets the conditions required for the execution of the medical image processing method / program based on user operations via the operation unit 208 and / or preset processing conditions (e.g., default processing conditions) (step S100: initial setting process). For example, it sets the conditions for specifying the detector to activate, switching or selecting the detector, and the method for displaying or notifying the detection results (display or non-display settings, displayed characters, graphics, symbols or their colors, voice output conditions, etc.). The processor 210 can activate all of the multiple detectors constituting the detector 223 (in which case, the detection results can also be displayed for a portion of the detectors), or it can activate a portion of the detectors while stopping the detectors (detection processing) that are not displaying detection results on the monitor 400 (display device). The user can, for example, access the system via... Figure 8 The processing conditions for the displayed image are set. Figure 8 In this example, the user can set processing conditions by turning the radio buttons on / off on the operation unit 208 or by entering values ​​into the numeric input field. Furthermore, the processor 210 can set processing conditions not only at the start of processing but also during the execution of subsequent steps.

[0110] <Acquisition of endoscopic images>

[0111] The image acquisition unit 220 (processor, image acquisition unit) acquires endoscopic images (medical images) taken inside the biological body of the subject (step S110: image acquisition processing, image acquisition process). The image acquisition unit 220 can acquire time-series endoscopic images (medical images) in real time by sequentially capturing images of the interior of the biological body of the subject at a predetermined frame rate through the imaging unit (imaging lens 132, imaging element 134, AFE 138, etc.) of the endoscope observer 100. In addition, the image acquisition unit 220 can also acquire endoscopic images that have already been captured and recorded in real time. For example, it can acquire endoscopic images or processed endoscopic images recorded in the recording unit 207, or it can acquire images from external devices or systems via the communication control unit 205. The display control unit 232 (processor, display control unit) causes the monitor 400 (display device) to display the acquired endoscopic images (step S120: display control processing, display control process).

[0112] <Detection of Areas of Interest>

[0113] The detector 223 (processor) uses the detector 223 to detect regions of interest from the endoscopic image (medical image) (step S130: region of interest detection processing, region of interest detection procedure). The detector 223 can perform multiple detection processes by using multiple detectors among the detectors constituting the detector 223. In the detection of regions of interest, the detector 223 can determine the position of the region of interest mapped in the image by the aforementioned "feature map" at the pixel level (i.e., detect whether each pixel of the endoscopic image belongs to the region of interest) and output its detection result. Furthermore, as examples of regions of interest (regions of interest) detected by the endoscope system 10, examples include polyps, cancer, colonic diverticulum, inflammation, treatment scars (EMR scars (Endoscopic Mucosal Resection), ESD scars (Endoscopic Submucosal Dissection), clamping sites, etc.), bleeding points, perforations, vascular aberrations, or various treatment instruments. In the case of ultrasound devices such as ultrasound endoscopes, organs or blood vessels can also be detected as regions of interest.

[0114] Furthermore, the processor 210 can detect or notify regions of interest for all frames of the acquired endoscopic images, or it can do so intermittently (at defined frame intervals).

[0115] Furthermore, the switching control unit 224 (processor) preferably switches the detector that displays the detection results on the monitor 400 (display device) based on the organ, part of the observed object, or camera information (switching process, switching procedure). When the detector of the switching target is in a stopped state, the switching control unit 224 starts the detection process based on that detector. In addition, the switching control unit 224 can also stop the operation (detection process) of detectors that do not display detection results on the monitor 400 (display device). In this way, by switching detectors (identifiers), appropriate diagnostic assistance functions (detection results detected by the detectors) can be provided to the user.

[0116] Notification when a region of interest is detected

[0117] When detector 223 detects a region of interest (YES in step S140), the first notification unit 234 (processor) causes monitor 400 (display device) to overlay information related to the detected region of interest and an endoscopic image (medical image) (step S150: first notification processing, first notification procedure). Additionally, the second notification unit 236 (processor) outputs voice from speaker 209A (voice output device) after the overlay display (first notification processing) (step S160: second notification processing, second notification procedure). Details and specific methods of notification will be described later. Processor 210 repeats steps S110 to S160 until the acquisition of the endoscopic image is completed or the user determines that "processing has ended" (YES in step S170).

[0118] <Specific methods for overlapping display>

[0119] Figure 9 This is a diagram illustrating an example of overlay display (first notification processing). In this diagram, an endoscope image 502 is displayed on the screen 500 of a monitor 400. When a region of interest 504 is detected from the endoscope image 502, the first notification unit 234 (processor) causes at least one of characters, graphics, or symbols to be overlaid on the endoscope image 502 as "information related to the region of interest." For example, such as... Figure 9 As shown in part (a), the first notification unit 234 can overlay an icon 506 (a flag-shaped graphic or symbol) outside the area of ​​the endoscopic image 502. Additionally, the first notification unit 234 can also... Figure 9 As shown in part (b), the image is overlaid at the location of the region of interest 504 (within the area of ​​the endoscopic image 502) (in this part, it is bounding box 508 (graphic)), or it can be overlaid at a location separate from the region of interest 504 as shown in part (c) of the figure (in this part, it is arrow 510 (graphic, symbol)).

[0120] The first notification section 234 can be displayed overlappingly at a location independent of the position of the region of interest 504, or it can be displayed overlappingly at a position corresponding to the position of the region of interest 504. For example, in Figure 9 In the example shown in part (d), the first notification unit 234 colors the area 512 in the lower right part of the image 500 corresponding to the capture of the region of interest 504 in the lower right part of the endoscopic image 502. Preferably, when the first notification unit 234 is overlaid at a position corresponding to the region of interest 504, if the position of the region of interest in the endoscopic image changes, the position of the overlaid information is moved accordingly. Furthermore, the first notification unit 234 can also combine changes in color and brightness when performing overlaid display.

[0121] <Details of voice-based notifications>

[0122] Even if certain objects, such as regions of interest, are continuously detected, the possibility of false positives is high if the same object is not continuously detected, so it can be considered that no speech output should be performed. Therefore, in the first method, the similarity of regions of interest is determined as follows, and speech output is performed based on the determination result.

[0123] Figure 10 This is a flowchart showing the details of the voice-based notification (second notification processing, second notification step) in step S160. When the first notification processing is performed in step S150, the feature quantity calculation unit 228 (processor) calculates and retains the feature quantities of the region of interest (step S200: feature quantity calculation processing / feature quantity calculation step, feature quantity retention processing / feature quantity retention step). "Feature quantity" refers to, for example, type, location, size, shape, color, etc. The feature quantity calculation unit 228 can calculate the feature quantities based on the analysis of the endoscopic image and the output of the detector 223. In addition, the feature quantity calculation unit 228 can retain the calculated feature quantities in a temporary recording medium such as RAM 212, or it can retain (record) them in a non-temporary recording medium such as the recording unit 207. Here, "temporary retention" includes, for example, clearing them sequentially after processing or clearing them when the power is turned off.

[0124] The similarity determination unit 230 (processor) compares the calculated feature values ​​with the held feature values ​​to determine the similarity of the regions of interest (step S210: similarity determination processing, similarity determination step). The similarity determination can be based on the number of consecutive detections of the region of interest. Specifically, the similarity determination unit 230 determines the similarity between the first and second regions of interest by comparing the feature values ​​of the region of interest (first region of interest) detected from an endoscopic image (medical image) captured at a first time with the feature values ​​of the region of interest (second region of interest) detected from an endoscopic image (second medical image) captured at a second time (a time earlier than the first time) (feature values ​​held by the feature value calculation unit 228). The second time can be set to a period determined earlier than the first time, and the user can determine the similarity via, for example... Figure 8 The screen showing the processing conditions allows you to specify the value for the "defined period".

[0125] The second notification unit 236 calculates the number of consecutive detections of the first region of interest based on the determination result in the sameness determination process. Specifically, when the first region of interest and the second region of interest are the same (YES in step S220), the second notification unit 236 calculates the number of consecutive detections of the second region of interest as the number of consecutive detections of the first region of interest (step S230: detection count calculation process, detection count calculation step). On the other hand, when the first region of interest and the second region of interest are different (NO in step S220), the second notification unit 236 calculates the number of consecutive detections of the first region of interest as the new region of interest (step S240: detection count calculation process, detection count calculation step). The second notification unit 236 can store the calculated number of consecutive detections in a temporary recording medium such as RAM 212, or it can store (record) it in a non-temporary recording medium such as recording unit 207, similar to the case of the feature quantity described above. Here, "temporary storage" includes, for example, clearing it sequentially after processing or clearing it when the power is turned off.

[0126] Furthermore, the similarity determination unit 230 can also determine the similarity of the region of interest based on the tracking performed by the detector 223, the overlap of the positions of the regions of interest, or the optical flow calculated from the endoscopic image. "Optical flow" refers to inferring and vectorizing the movement of the part captured in the image or the entire subject based on clues such as corresponding points between images.

[0127] Furthermore, the similarity determination unit 230 and the second notification unit 236 (processor) can perform similarity determination and calculate the number of consecutive detections for all frames of the endoscopic image, or they can perform these actions intermittently. For example, if the frame rate of the endoscopic image is 30 fps (frames per second), the similarity determination can be performed at 30 fps, or at a lower frame rate (e.g., 10 fps). The similarity determination can be matched with the frame rate of the detector 223.

[0128] If the number of consecutive detections calculated in this way exceeds the threshold (YES in step S250), the second notification unit 236 outputs voice from the speaker 209A (voice output device) (step S260: second notification processing, second notification process).

[0129] <Specific example of voice output (1): The case where the number of consecutive detections exceeds the threshold>

[0130] Figure 11This is a diagram illustrating a specific example (1) of voice output. In this example, no region of interest is detected at time t0, but a region of interest 504 is detected at time t1, and overlapping display of bounding boxes 508 begins (first notification processing). Furthermore, the same region of interest 504 is also detected at times t2, t3, and t4, and overlapping display occurs. In this situation, if the threshold for consecutive detections is set to three, then at time t4, the number of consecutive detections reaches four, exceeding the threshold (i.e., after detecting the region of interest 504 at time t1, a first period (=t2-t1) has elapsed before the same region of interest 504 is detected again). Therefore, the second notification unit 236 outputs voice from the speaker 209A. Furthermore, in Figure 11 In this context, a speaker icon 520 is used to indicate the output of voice (the icon itself may not be displayed on screen 500; the same applies in the following examples). Thus, the second notification unit 236 (processor) performs voice output after the overlay display.

[0131] <Specific example of speech output (2): Case where the number of consecutive detections does not exceed the threshold>

[0132] Figure 12 This is a diagram representing a specific example (2) of speech output. In Figure 12 In the example shown, from time t1 to time t3, the same region of interest 504 is detected and overlapped with a bounding box 508 (first notification processing), but at time t4, the region of interest 504 is not detected (i.e., after the first period following the detection of the region of interest 504 at time t1, the same region of interest 504 is not detected), therefore the number of consecutive detections (three times) does not exceed the threshold (three times) (NO in step S250), and the second notification unit 236 does not output voice. Figure 12 In the text, the crossed-out icon 522 indicates that no voice output is allowed.

[0133] As mentioned above, false positives (detector 223 identifying areas that are not of interest as areas of interest) tend to occur more frequently in an instantaneous manner and less frequently in a sustained manner. Therefore, as Figure 12 As the example shows, if the area of ​​interest is detected instantaneously (times t1 to t3), as long as no voice output is made, the possibility of users being annoyed by voice output due to false positives can be reduced. On the other hand, since the area of ​​interest is emphasized in the screen display (first notification processing), it is associated with arousing the user's attention, and the effect of preventing lesions from being missed can be expected.

[0134] <Specific example of speech output (3): Determining the similarity of regions of interest>

[0135] Figure 13This is a diagram illustrating a specific example (3) of the speech output when judging the similarity of regions of interest. Figure 13 In the example, regions of interest (504 and 507) are detected between times t1 and t4, and bounding boxes 508 and 509 are displayed overlaid (first notification processing). However, since region 504 is detected three times (times t1 to t3) and region 507 is detected twice (times t3 and t4), the second notification unit 236 does not output voice at time t4. This suppresses unnecessary voice output caused by transient false positives.

[0136] <Specific examples of voice output (4): Examples related to the judgment of similarity>

[0137] In the detection of regions of interest using AI such as detector 223, false negatives, similar to false positives (where the AI ​​incorrectly identifies a region of interest as absent despite its presence in the endoscopic image), are also a difficult problem to avoid. For example, as... Figure 14 In the example shown, a situation might occur where "even if the region of interest (regions of interest 504A and 504B) is detected in the endoscopic image at times t1, t2, and t4, the region of interest (region of interest 504A) might not be detected at time t3 due to a judgment error by detector 223." In this case, a problem might arise where "the number of consecutive detections cannot be properly calculated, and sound should be output but is not." Therefore, the detection count calculation unit 226, the similarity determination unit 230, and the second notification unit 236 (processor) include the similarity determination objects for regions of interest not only in the previous frame but also in frames further back.

[0138] exist Figure 14 In the example, assuming that the similarity determination unit 230 compares the detection results (presence or absence of the region of interest, feature quantity) of the region of interest 504B detected at time t4 and the region of interest 504A detected at time t2 and can determine that they are "same", in this case, the detection count calculation unit 226 increases the number of consecutive detections of the region of interest 504A. Thus, the number of consecutive detections at time t4 becomes four, exceeding the threshold (three), therefore the second notification unit 236 outputs voice from the speaker 209A at time t4. Figure 14 In the diagram, time t4 is represented by icon 520. This similarity determination avoids the problem of inappropriately calculating the number of consecutive detections due to false negatives. Furthermore, regarding past frames that become the comparison object, even excessive historical calculations can lead to increased computational costs or decreased accuracy in similarity determination; therefore, it is preferable to limit the calculation to the current frame (within the current frame). Figure 14In the example, the time near time t4 (the first time) (until a certain period before time t2 (the second time)).

[0139] In addition, Figure 14 In the example, the detection count calculation unit 226 and the similarity determination unit 230 may not count the consecutive detection count to "four times," but instead control it to "maintain the consecutive detection count at three times, but continue to determine that the same region of interest is continuously detected." When the consecutive detection count is maintained in this way without increasing, if the same region of interest 504A is detected in the next frame at time t4, the consecutive detection count reaches four times, and a notification based on voice output is performed.

[0140] <Specific Examples of Voice Output (Part 5): Examples of Restricted Voice Output>

[0141] In the endoscope system 10, the notification based on voice output (second notification processing) can also be controlled to prevent voice output after a certain number of consecutive detections. For example, in Figure 15 In the example, at time t4, after four consecutive detections, voice is output. However, during a defined period after time t5 (three frames up to time t7) when five consecutive detections have been performed, the second notification unit 236 (processor) does not output voice. This avoids the problem of frequent voice outputs causing user annoyance. Furthermore, in Figure 15 In the example, the non-output of speech could be deactivated at time t8 after a defined period, but if the same object (region of interest) is detected again after deactivation, speech output could be deactivated again (in this case, speech output would not be performed after time t8). This avoids the problem of frequent speech output during the observation of the same object, which would annoy the user.

[0142] <Specific example of voice output (6): An example of linking the overlay display method with the voice output status>

[0143] In the endoscope system 10, the display method (overlapping display: first notification processing) can also be changed according to the voice output status in the second notification processing. For example, in Figure 16 In the example, speech is output after time t4, when the number of consecutive detections reaches four and exceeds the threshold. However, the first notification unit 234 (processor) makes the border of the bounding box 511 overlapping in the region of interest 504 thicker than the bounding box 508 up to time t1 to t3. As a change in the way the screen is displayed corresponding to the speech output state, the color, size, or shape of such overlapping graphics can be changed, or combinations such as... Figure 9The example shows the overlapping display of other graphics, etc. By changing the way the screen is displayed in this way, the endoscope system 10 can more reliably convey the situation of the object being examined to the user intuitively. Furthermore, the first notification unit 234 can simultaneously change the way the screen is displayed and change the voice output status (voice output start / stop, etc.). Figure 9 In the example, the time is t4, but it can also be performed at approximately the same time before or after.

[0144] As described above, according to the first embodiment, the possibility of missing the area of ​​interest can be reduced and unnecessary voice output can be suppressed.

[0145] <Applications to other medical images>

[0146] In the first embodiment described above, the case of recognizing an endoscopic image (image from an optical endoscope) as a type of medical image (medical image) was explained. However, the medical image processing apparatus, medical image processing method, and medical image processing program involved in this invention can also be applied to cases using medical images other than endoscopic images, such as images obtained by an ultrasonic endoscope device (ultrasonic endoscope system), an ultrasonic image diagnostic device, etc.

[0147] (Postscript)

[0148] In addition to the above-described embodiments and variations, the structures described below are also included within the scope of this invention.

[0149] (Postscript 1)

[0150] A medical image processing device, wherein,

[0151] The medical image analysis and processing unit detects regions of interest (ROIs) based on the pixel features of the medical image.

[0152] The Medical Image Analysis Result Acquisition Department acquires the analysis results from the Medical Image Analysis and Processing Department.

[0153] (Postscript 2)

[0154] A medical image processing device, wherein,

[0155] The medical image analysis and processing department, based on the pixel features of medical images, should focus on detecting the presence or absence of objects.

[0156] The Medical Image Analysis Result Acquisition Department acquires the analysis results from the Medical Image Analysis and Processing Department.

[0157] (Note 3)

[0158] A medical image processing device, wherein,

[0159] The medical image analysis result acquisition unit acquires the results from a recording device that records the analysis results of medical images.

[0160] The analysis result is either or both of the regions of interest and the presence or absence of objects of interest contained in the medical image.

[0161] (Note 4)

[0162] A medical image processing device, wherein,

[0163] Medical images are ordinary light images obtained by illuminating light in the white frequency band, or by illuminating light in multiple wavelength bands as white frequency band light.

[0164] (Note 5)

[0165] A medical image processing device, wherein,

[0166] Medical images are images obtained by irradiating light with a specific wavelength band.

[0167] A specific wavelength band is a band narrower than the white wavelength band.

[0168] (Note 6)

[0169] A medical image processing device, wherein,

[0170] The specific wavelength band is the blue or green band within the visible range.

[0171] (Note 7)

[0172] A medical image processing device, wherein,

[0173] A specific wavelength band includes wavelengths above 390nm and below 450nm or above 530nm and below 550nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band above 390nm and below 450nm or above 530nm and below 550nm.

[0174] (Note 8)

[0175] A medical image processing device, wherein,

[0176] The specific wavelength band is the red band within the visible range.

[0177] (Note 9)

[0178] A medical image processing device, wherein,

[0179] A specific wavelength band includes wavelengths above 585nm and below 615nm or above 610nm and below 730nm, and the light in the specific wavelength band has a peak wavelength within the wavelength band above 585nm and below 615nm or above 610nm and below 730nm.

[0180] (Postscript 10)

[0181] A medical image processing device, wherein,

[0182] The specific wavelength band includes wavelength bands with different absorption coefficients in oxyhemoglobin and deoxyhemoglobin, and the light in the specific wavelength band has a peak wavelength in the wavelength bands with different absorption coefficients in oxyhemoglobin and deoxyhemoglobin.

[0183] (Postscript 11)

[0184] A medical image processing device, wherein,

[0185] A specific wavelength band includes wavelengths of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm, and the light in the specific wavelength band has a peak wavelength within the wavelength bands of 400±10nm, 440±10nm, 470±10nm, or 600nm to 750nm.

[0186] (Postscript 12)

[0187] A medical image processing device, wherein,

[0188] Medical imaging refers to images taken inside a living organism.

[0189] Images within a living organism contain information about the fluorescence emitted by fluorescent substances within that organism.

[0190] (Postscript 13)

[0191] A medical image processing device, wherein,

[0192] Fluorescence can be obtained by irradiating a living organism with an excitation light having a peak wavelength between 390 nm and 470 nm.

[0193] (Postscript 14)

[0194] A medical image processing device, wherein,

[0195] Medical imaging refers to images taken inside a living organism.

[0196] The specific wavelength band is the wavelength band of infrared light.

[0197] (Postscript 15)

[0198] A medical image processing device, wherein,

[0199] A specific wavelength band includes wavelengths above 790nm and below 820nm or above 905nm and below 970nm, and the light in the specific wavelength band has a peak wavelength within the wavelength bands above 790nm and below 820nm or above 905nm and below 970nm.

[0200] (Postscript 16)

[0201] A medical image processing device, wherein,

[0202] The medical image acquisition unit includes a special light image acquisition unit, which acquires a special light image with information of a specific wavelength band based on a normal light image obtained by illuminating light in the white frequency band, or light in multiple wavelength bands as white frequency band light illuminating light.

[0203] Medical images are special light images.

[0204] (Postscript 17)

[0205] A medical image processing device, wherein,

[0206] Signals in a specific wavelength band are obtained through calculations based on the RGB or CMY color information contained in a normal light image.

[0207] (Postscript 18)

[0208] A medical image processing device, wherein,

[0209] The system includes a feature quantity image generation unit that generates a feature quantity image by performing calculations based on at least one of a general light image obtained by illuminating light in the white frequency band, or light in multiple wavelength frequency bands as white frequency band, and a special light image obtained by illuminating light in a specific wavelength frequency band.

[0210] Medical images are characteristic images.

[0211] (Postscript 19)

[0212] An endoscopic device comprising:

[0213] The medical image processing apparatus described in any one of Appendices 1 to 18; and

[0214] An endoscope that acquires images by illuminating light in a white wavelength band or at least one of a specific wavelength band.

[0215] (Postscript 20)

[0216] A diagnostic aid device comprising the medical image processing apparatus described in any one of Appendices 1 to 18.

[0217] (Postscript 21)

[0218] A medical business assistance device, comprising the medical image processing device described in any one of Appendices 1 to 18.

[0219] The embodiments and other examples of the present invention have been described above, but the present invention is not limited to the above-described manner. Various modifications can be made without departing from the spirit of the present invention.

[0220] Symbol Explanation

[0221] 10 Endoscopic Systems

[0222] 100 Endoscopic Observation Devices

[0223] 102 Hands-on Operations Department

[0224] 104 Insertion Section

[0225] 106 General Purpose Cable

[0226] 108 Optical Wire Connector

[0227] 112 Soft parts

[0228] 114 Bend

[0229] 116 Front-end rigid part

[0230] 116A Front Side Face

[0231] 123 Lighting Department

[0232] 123A Illumination Lens

[0233] 123B Illumination Lens

[0234] 126 Pliers opening

[0235] 130 photographic optical system

[0236] 132 Photographic Lens

[0237] 134 camera elements

[0238] 136 drive circuit

[0239] 141 Gas and water supply buttons

[0240] 142 Attraction Button

[0241] 143 Function Buttons

[0242] 144 Camera button

[0243] 170 optical waveguide

[0244] 200 Medical Image Processing Department

[0245] 202 Image Input Controller

[0246] 205 Communications Control Department

[0247] 206 Video Output Department

[0248] 207 Records Department

[0249] 208 Operations Department

[0250] 209 Speech Processing Department

[0251] 209A Speaker

[0252] 210 processor

[0253] 211 ROM

[0254] 212 RAM

[0255] 220 Image Acquisition Unit

[0256] 222 Focus Area Testing Department

[0257] 223 detector

[0258] 223A Pharyngeal Detector

[0259] 223B esophageal detector

[0260] 223C gastric detector

[0261] 223D Duodenal Detector

[0262] 224 Switching Control Unit

[0263] 226 Detection Count Calculation Department

[0264] 228 Characteristic Quantity Calculation Section

[0265] 230 Identity Determination Section

[0266] 232 Display Control Unit

[0267] 234 First Notification Department

[0268] 236 Second Notification Department

[0269] 238 Record Control Department

[0270] 240 Communications Control Department

[0271] 250 Input Layer

[0272] 252 Intermediate Layer

[0273] 254 Output Layer

[0274] 256 convolutional layers

[0275] 258 pooling layer

[0276] 260 Fully Connected Layer

[0277] 300 Light Source Device

[0278] 310 Light Source

[0279] 310B Blue Light Source

[0280] 310G Green Light Source

[0281] 310R Red Light Source

[0282] 310V purple light source

[0283] 330 aperture

[0284] 340 Condensing Lens

[0285] 350 Light Source Control Unit

[0286] 400 monitor

[0287] 500 frames

[0288] 502 Endoscopic image

[0289] 504 Area of ​​Concern

[0290] 504A Area of ​​Concern

[0291] 504B Area of ​​Concern

[0292] 506 icon

[0293] 507 Area of ​​Concern

[0294] 508 bounding box

[0295] 509 Bounding Box

[0296] 510 Arrow

[0297] 511 Bounding Box

[0298] Area 512

[0299] 520 icon

[0300] 522 icon

[0301] F1 filter

[0302] F2 filter

[0303] Steps of the medical image processing method (S100-S260)

Claims

1. A medical image processing device, comprising a processor, wherein, The processor executes: Image acquisition and processing; acquiring time-series medical images. Region of interest detection processing is performed to detect regions of interest from the acquired medical images; Display control processing, displaying the medical image on a display device; as well as In the first notification processing, when a region of interest is detected through the region of interest detection processing, the display device overlays the medical image and information related to the detected region of interest. During the processing of the first notification, a determination is made as to whether the detection is continuous. If the detection is determined to be a continuous detection, the first notification processing is performed, and the second notification processing, which outputs voice from the voice output device when a region of interest is detected by the region of interest detection processing, is performed. If it is determined that the detection is not the continuous detection, the second notification process will not be executed.

2. The medical image processing apparatus according to claim 1, wherein, When a region of interest is detected in the region of interest detection process and then detected again after a first period, the processor determines that the detection is a continuous detection and executes the second notification process. When no region of interest is detected after the first period, the processor determines that the detection is not a continuous detection and does not execute the second notification process.

3. The medical image processing apparatus according to claim 1 or 2, wherein, In the first notification processing, the processor causes the display device to overlay the information corresponding to the position of the region of interest in the medical image.

4. The medical image processing apparatus according to claim 1 or 2, wherein, The processor performs a detection count calculation process, calculating the number of consecutive detections for the region of interest detected through the region of interest detection process. When the number of consecutive detections exceeds a predetermined number, the detection is determined to be a continuous detection and the second notification process is executed.

5. The medical image processing apparatus according to claim 4, wherein, The processor also performs: Feature preservation processing preserves the feature values ​​of the detected region of interest; and The similarity determination process compares the feature values ​​of a first region of interest detected in a medical image taken at a first time point with the feature values ​​of a second region of interest detected in a second medical image taken at a time earlier than the first time point in which the data is stored, to determine the similarity between the first and second regions of interest. In the detection count calculation process, the number of consecutive detections for the first region of interest is calculated based on the determination result in the sameness determination process.

6. The medical image processing apparatus according to claim 5, wherein, When the processor determines that the first region of interest and the second region of interest are the same in the sameness determination process, in the detection count calculation process, after adding the number of consecutive detections recorded for the second region of interest, it calculates the number of consecutive detections for the first region of interest.

7. The medical image processing apparatus according to claim 5, wherein, In the sameness determination process, the processor compares the feature values ​​of the feature values ​​preserved by the feature value preservation process with the feature values ​​of the first region of interest with the feature values ​​of the period before the first time.

8. The medical image processing apparatus according to claim 1 or 2, wherein, In the second notification processing, after the processor causes the voice output device to output voice, it does not output voice for a defined period of time.

9. The medical image processing apparatus according to claim 1 or 2, wherein, In the first notification processing, the processor changes the method of the first notification processing according to the voice output status in the second notification processing.

10. The medical image processing apparatus according to claim 1 or 2, wherein, In the first notification processing, the processor uses at least one of characters, graphics, and symbols as information to cause the display device to perform the overlapping display.

11. An endoscope system comprising: The medical image processing apparatus according to any one of claims 1 to 10; An endoscopic observer, inserted into the subject, has a camera unit that captures the medical images; The display device; and The voice output device.

12. A medical image processing method, executed by a medical image processing device equipped with a processor, wherein, The processor executes: The image acquisition process involves acquiring time-series medical images. The focus area detection process involves detecting the region of interest from the acquired medical image. The display control process causes the display device to display the medical image; as well as In the first notification process, when a region of interest is detected in the region of interest detection process, the display device overlays the medical image and information related to the detected region of interest. During the first notification process, a determination is made as to whether the detection is a continuous detection. If the detection is determined to be the continuous detection, the first notification step is executed, and the second notification step, which outputs voice from the voice output device when a region of interest is detected in the region of interest detection step, is executed. If it is determined that the detection is not the continuous detection, the second notification process is not executed.

13. A recording medium that is non-transitory and computer-readable, wherein, The system contains a program that causes a computer to perform the medical image processing method of claim 12.

Images