Medical image processing device, endoscope system, medical image processing method, and recording medium

Abstract

The present invention provides a medical image processing device, an endoscope system, a medical image processing method, and a program that can assist in searching using information about an unobserved region of interest. The medical image processing device comprises: an image acquisition unit (40) that acquires an observed image (38); a display signal transmission unit (44) that transmits a first display signal representing the observed image to a display device (400); a region of interest detection unit (50) that detects a region of interest from the observed image; and an unobserved image storage unit (71) that stores, among the observed images in which the region of interest is detected, unobserved images that satisfy an unobserved condition indicating that the region of interest is unobserved.

Description

Technical Field

[0001] The present invention relates to a medical image processing device, an endoscope system, a medical image processing method and a program. Background Art

[0002] Systems that incorporate AI technology into endoscopic images to detect and assist in lesion diagnosis are attracting attention due to their potential to reduce missed lesions and unnecessary biopsies. These systems operate in real time during endoscopic examinations, providing notification to physicians through on-screen displays of recognition results. AI stands for Artificial Intelligence.

[0003] Patent Documents 1 to 3 describe endoscope systems that detect a lesion candidate region in an observation image based on a feature amount calculated from the observation image and display the detected lesion candidate region on a display device.

[0004] The endoscope system described in Patent Document 1 adds a marker image to the observed image to emphasize the location of a candidate lesion region. The system changes the display of the marker image based on the duration of the candidate lesion region detection. Specifically, the system begins highlighting the candidate lesion region when the candidate lesion region detection duration reaches a predetermined period.

[0005] The endoscope system described in Patent Document 2 stores observation images sequentially output from a video processor during a continuous detection period. If the continuous detection period is less than a predetermined period, the system displays the stored observation images in a display area different from the display area displaying the observation images, in the reverse order of their storage order, after the predetermined period has elapsed.

[0006] The endoscope system described in Patent Document 3 temporarily stores a still image in memory when a candidate lesion area is detected and notification processing begins. When an operating switch is turned on, the system adds a marker image to the still image temporarily stored in memory and displays the still image on a display unit.

[0007] Patent Document 4 describes an electronic endoscope device that is inserted into a subject to observe its interior. Upon detecting an abnormal pixel, the device automatically and temporarily stores a still image in memory. Furthermore, the device displays a thumbnail of the temporarily stored still image in a thumbnail display area.

[0008] Previous technical literature

[0009] Patent Literature

[0010] Patent Document 1: International Publication No. 2018 / 198161

[0011] Patent Document 2: International Publication No. 2017 / 216922

[0012] Patent Document 3: International Publication No. 2017 / 073337

[0013] Patent Document 4: Japanese Patent Application Laid-Open No. 2006-198106 Summary of the Invention

[0014] Technical issues to be solved by the invention

[0015] A doctor who has missed a region of interest detection can recognize the missed region of interest by receiving notification of the region of interest detection. When searching for a missed region of interest, the doctor needs to use information related to the missed region of interest. However, Patent Documents 1 to 4 do not describe or suggest information related to the missed region of interest.

[0016] The system described in Patent Document 1 emphasizes a lesion or other area when the detection period reaches a predetermined time, thereby reducing the burden on doctors who visually confirm candidate lesions. However, the system described in this document lacks a component for storing information about missed candidate lesions.

[0017] The system described in Patent Document 2 stores one or more observation images from the start of ROI detection until it is interrupted. After ROI detection is interrupted, the stored observation images are automatically displayed, thereby reducing the chance of missing lesions. However, the automatic re-display of observation images can lead to a decrease in physician attention, potentially leading to missed ROIs. Furthermore, the system described in this document lacks a component for storing information about missed candidate lesion regions.

[0018] The system described in Patent Document 3 temporarily stores a still image containing a candidate lesion region upon detection, and then displays the still image upon turning on an operating switch. This document does not mention selectively storing still images containing detected lesion regions, suggesting that the system described in this document stores all still images containing candidate lesion regions. This makes it difficult to distinguish whether a doctor is actively observing the temporarily stored still images.

[0019] Patent Document 4 does not mention selectively temporarily storing still images when an abnormal pixel is detected. It is believed that the device described in this document temporarily stores all still images when an abnormal pixel is detected. This makes it difficult to distinguish whether a doctor is observing the temporarily stored still images.

[0020] That is, it is difficult for the systems described in Patent Documents 1 to 4 to assist a doctor in detecting a missed, unobserved region of interest.

[0021] The present invention has been made in view of such circumstances, and its object is to provide a medical image processing device, an endoscope system, a medical image processing method, and a program that can assist in searching using information of an unobserved area of ​​interest.

[0022] Means for solving technical problems

[0023] In order to achieve the above-mentioned object, the following invention aspects are provided.

[0024] The medical image processing device involved in the first method is a medical image processing device comprising the following components: an image acquisition unit, which acquires an observation image of a subject; a display signal sending unit, which sends a first display signal representing the observation image to a display device; a region of interest detection unit, which detects a region of interest from a frame image constituting the observation image; and an unobserved image storage unit, which stores an unobserved image in the frame image in which the region of interest is detected that satisfies an unobserved condition indicating that the region of interest is not observed.

[0025] According to the first aspect, a frame image including an unobserved region of interest that satisfies the unobserved condition is stored in an observed image in which the region of interest is detected. This assists the search for the unobserved region of interest using the unobserved image.

[0026] The observation image of the subject may be a moving image or a still image group including a plurality of still images. An example of the observation image is an endoscopic image captured using an endoscope.

[0027] The frame image is equivalent to one still image included in a still image group including a plurality of still images.

[0028] An example of an unobserved area of ​​interest is an area of ​​interest where the user does not perform any operation of the imaging device such as zooming.

[0029] A second embodiment may be configured such that the medical image processing apparatus of the first embodiment is provided with a non-observation condition determination unit that determines whether the non-observation condition is satisfied for a frame image in which the region of interest is detected by the region of interest detection unit.

[0030] According to the second aspect, it is possible to determine an unobserved image from a plurality of frame images in which the attention region is detected based on the unobserved condition.

[0031] In the second aspect, a preferred aspect includes a non-observation condition setting unit for setting the non-observation condition.

[0032] A third embodiment may be configured such that, in the medical image processing apparatus of the second embodiment, the non-observation condition determination unit determines that the non-observation condition is satisfied when the number of frame images including the same region of interest within a predetermined period is less than a predetermined number.

[0033] According to the third aspect, the number of frame images including the same region of interest can be applied as the non-observation condition.

[0034] A fourth embodiment may be configured such that, in the medical image processing apparatus of the second embodiment or the third embodiment, the non-observation condition determination unit determines that the non-observation condition is satisfied when the amount of change between frame images is greater than a predetermined threshold value.

[0035] According to the fourth aspect, the amount of change between frame images can be applied as the non-observation condition.

[0036] The fifth embodiment can be configured as follows: in the medical image processing device of any one of the second to fourth embodiments, the non-observation condition determination unit determines that the non-observation condition is satisfied when the same area of ​​interest stays in any area within the screen within a specified period.

[0037] According to the fifth aspect, the position of the same region of interest within the screen can be applied as the non-observation condition.

[0038] The sixth embodiment can be configured as follows: in the medical image processing device of any one of the second to fifth embodiments, the display signal sending unit sends a second display signal to the display device, indicating an unobserved image determined by the unobserved condition determination unit to satisfy the unobserved condition.

[0039] According to the sixth aspect, the unobserved image is displayed on the display device, thereby assisting the search for the unobserved region of interest using the unobserved image displayed on the display device.

[0040] The seventh method can be configured as follows: in the medical image processing device of the sixth method, there is a user input signal acquisition unit that acquires a user input signal sent in response to a user operation, and the display signal sending unit sends a second display signal representing an unobserved image to the display device when the user input signal acquisition unit acquires a user input signal representing a display of an unobserved area of ​​interest.

[0041] According to the seventh aspect, a display signal representing an unobserved image is transmitted to the display device in response to acquisition of a user input signal. This assists in searching for an unobserved region of interest using the unobserved image displayed on the display device.

[0042] The eighth method can be configured as follows: in the medical image processing device of the seventh method, there is a first image processing unit, which performs first image processing on the area of ​​interest detected by the area of ​​interest detection unit to generate a first image, and a display signal sending unit sends a display signal representing the first image to the display device as a first display signal.

[0043] According to the eighth aspect, the first image subjected to the first image processing can be displayed on the display device.

[0044] In the eighth aspect, a first image processing result storage unit for storing a result of the first image processing may be provided.

[0045] The processing result of the first image processing may be applied to the first image, or may be applied to a combination of the observation image and information indicating the processing result of the first image processing associated with the observation image.

[0046] The ninth embodiment can be configured as follows: in the medical image processing device of the eighth embodiment, a second image processing unit is provided, which performs second image processing on the unobserved image to generate a second image, and a display signal sending unit sends a display signal representing the second image to the display device as a second display signal.

[0047] According to the ninth aspect, the second image obtained by performing the second image processing on the unobserved image can be displayed on the display device.

[0048] The tenth method can be configured as follows: in the medical image processing device of the ninth method, the first image processing unit performs emphasis processing on the focus area of ​​the observed image, and the second image processing unit performs emphasis processing on the unobserved image that increases the degree of emphasis more than the emphasis processing on the focus area of ​​the observed image.

[0049] According to the tenth aspect, the second image having a higher degree of emphasis than that of the observed image can be displayed on the display device, thereby facilitating the visual recognition of the target area in the second image.

[0050] The eleventh method can be configured as follows: in the medical image processing device of the ninth method or the tenth method, the second image processing unit performs second image processing on each of a plurality of unobserved images, and the display signal sending unit sends a display signal corresponding to each of the plurality of second images to the display device as a second display signal.

[0051] According to the eleventh aspect, a plurality of unobserved images or a plurality of second images can be displayed on the display device, thereby allowing the user to search for a region of interest in an observed image using the plurality of unobserved images or the plurality of second images.

[0052] The twelfth embodiment can be configured as follows: in the medical image processing device of the eleventh embodiment, there is provided a selection unit which selects one or more unobserved images from a plurality of unobserved images displayed on the display device, or selects one or more second images from a plurality of second images displayed on the display device.

[0053] According to the twelfth aspect, the user can set an unobserved image to be used in searching for a region of interest in an observed image from among a plurality of unobserved images, or set a second image to be used in searching for a region of interest in an observed image from among a plurality of second images.

[0054] The thirteenth embodiment can be configured as follows: in the medical image processing device of any one of the ninth to twelfth embodiments, there is provided a third image processing unit which performs a third image processing on the unobserved image displayed on the display device or the processing result of the second image processing.

[0055] According to the thirteenth aspect, the third image processing can be performed on the unobserved image or the second image.

[0056] The fourteenth embodiment may be configured such that the medical image processing apparatus of the thirteenth embodiment further comprises a third image processing result storage unit that stores the processing result of the third image processing.

[0057] According to the fourteenth aspect, the processing result of the third image processing can be stored.

[0058] The third image processing result storage unit may store the third image as the processing result of the third image processing. The third image processing result storage unit may also store a combination of the processing target image and information indicating the processing result of the third image processing associated with the processing target image as the processing result of the third image processing.

[0059] The fifteenth embodiment can be configured as follows: in the medical image processing device of the thirteenth embodiment or the fourteenth embodiment, the third image processing unit performs editing of the unobserved image when the user input signal acquisition unit acquires a user input signal indicating editing of the unobserved image, or performs editing of the second image when the user input signal acquisition unit acquires a user input signal indicating editing of the second image.

[0060] According to the fifteenth aspect, in response to acquisition of a user input signal, editing of the unobserved image or the second image can be performed.

[0061] The sixteenth embodiment can be configured as follows: in the medical image processing device of any one of the thirteenth to fifteenth embodiments, the third image processing unit sends the unobserved image to the external device when the user input signal acquisition unit acquires a user input signal indicating that the unobserved image is to be sent to the external device, or sends the second image to the external device when the user input signal acquisition unit acquires a user input signal indicating that the second image is to be sent to the external device.

[0062] According to the sixteenth aspect, it is possible to transmit the unobserved image or the second image to an external device.

[0063] The seventeenth embodiment can be configured as follows: in the medical image processing device of any one of the thirteenth to sixteenth embodiments, the third image processing unit performs the rearrangement of multiple unobserved images when the user input signal acquisition unit acquires a user input signal indicating the rearrangement of multiple unobserved images, or performs the rearrangement of multiple second images when the user input signal acquisition unit acquires a user input signal indicating the rearrangement of multiple second images.

[0064] According to the seventeenth aspect, in an aspect in which a plurality of unobserved images or a plurality of second images are displayed on a display device, it is possible to rearrange the plurality of unobserved images or rearrange the plurality of second images.

[0065] The eighteenth embodiment can be configured as follows: in the medical image processing device of any one of the thirteenth to seventeenth embodiments, the third image processing unit deletes the unobserved image when the user input signal acquisition unit acquires a user input signal indicating deletion of the unobserved image, or deletes the second image when the user input signal acquisition unit acquires a user input signal indicating deletion of the second image.

[0066] According to the eighteenth aspect, in response to acquisition of a user input signal, the unobserved image or the second image may be deleted.

[0067] A nineteenth aspect may be configured such that, in the medical image processing apparatus of any one of the sixth to eighteenth aspects, the display signal sending unit stops sending the second display signal after a predetermined period has elapsed from the timing of sending the second display signal.

[0068] According to the nineteenth aspect, non-display of the unobserved image or the second image can be automatically performed.

[0069] A twentieth aspect may be configured such that, in the medical image processing apparatus of any one of the sixth to eighteenth aspects, the display signal sending unit stops sending the second display signal when the unobserved region of interest is displayed on the display device.

[0070] According to the twentieth aspect, when the target area of ​​interest is found, the unobserved image or the second image can be hidden from display.

[0071] The twenty-first embodiment can be configured as follows: in a medical image processing device of any one of the first to twentieth embodiments, there is a user operation unit operated by a user, and the user input signal acquisition unit acquires the user input signal sent when the user operates the user operation unit.

[0072] According to the twenty-first aspect, a user input signal can be transmitted in response to a user operation.

[0073] The twenty-second embodiment can be configured as follows: in a medical image processing device of any one of the first to twenty-first embodiments, there is a sound acquisition unit for acquiring the user's voice, and the user input signal acquisition unit acquires a user input signal representing the user's voice acquired using the sound acquisition unit.

[0074] According to the twenty-second aspect, the user input signal can be sent in response to the user's voice.

[0075] The twenty-third embodiment can be configured as follows: in a medical image processing device of any one of the first embodiment to the twenty-second embodiment, there is an observed image storage unit, which stores an observed image indicating that the interest area has been observed in a frame image in which the interest area is detected using the interest area detection unit.

[0076] According to the twenty-third aspect, the observed image is stored, and thus the observed image can be used to assist in searching for the observed region of interest.

[0077] A twenty-fourth aspect may be configured such that the medical image processing apparatus according to any one of the first to twenty-third aspects includes a notification unit that notifies detection of a region of interest in an observation image.

[0078] According to the twenty-fourth aspect, the user can recognize the detection of the attention area.

[0079] The endoscope system involved in the twenty-fifth method comprises: an endoscope; an endoscope control device which controls the endoscope; and a medical image processing device which processes the endoscopic image acquired using the endoscope, wherein the medical image processing device comprises: an image acquisition unit which acquires an observation image of the subject; a display signal sending unit which sends a first display signal representing the observation image to the display device; an area of ​​interest detection unit which detects the area of ​​interest from the frame image constituting the observation image; and an unobserved image storage unit which stores the unobserved image in the frame image in which the area of ​​interest is detected that satisfies the unobserved condition indicating that the area of ​​interest is not observed.

[0080] According to the twenty-fifth aspect, the same effects as those of the first aspect can be obtained.

[0081] In the twenty-fifth aspect, the same items as those specified in the second to twenty-fourth aspects may be appropriately combined. In this case, the components that perform specific processing or functions in the medical image processing apparatus may be understood as components of the endoscope system that perform the corresponding processing or functions.

[0082] The medical image processing method involved in the twenty-sixth method includes: an image acquisition process, which acquires an observation image of the subject; a display signal sending process, which sends a first display signal representing the observation image to a display device; a region of interest detection process, which detects the region of interest from the frame image constituting the observation image; and an unobserved image storage process, which stores the unobserved image that satisfies the unobserved condition indicating that the region of interest is not observed in the frame image in which the region of interest is detected.

[0083] According to the twenty-sixth aspect, the same effects as those of the first aspect can be obtained.

[0084] In the twenty-sixth embodiment, the same matters as those specified in the second to twenty-fourth embodiments may be appropriately combined. In this case, the components that undertake specific processing or functions in the medical image processing apparatus may be understood as components of the medical image processing method that undertake the corresponding processing or functions.

[0085] The program involved in the twenty-seventh method is a program that enables a computer to implement the following functions: an image acquisition function for acquiring an observed image of a subject; a display signal sending function for sending a first display signal representing the observed image to a display device; a region of interest detection function for detecting a region of interest from a frame image constituting the observed image; and an unobserved image storage function for storing an unobserved image that satisfies an unobserved condition indicating that the region of interest is not observed in the frame image in which the region of interest is detected.

[0086] According to the twenty-seventh aspect, the same effects as those of the first aspect can be obtained.

[0087] In the twenty-seventh embodiment, the same matters as those specified in the second to twenty-fourth embodiments may be appropriately combined. In this case, the components that undertake specific processing or functions in the medical image processing device may be understood as components of a program that undertakes the corresponding processing or functions.

[0088] Effects of the Invention

[0089] According to the present invention, a frame image including an unobserved region of interest that satisfies the unobserved condition in an observed image in which the region of interest is detected is stored, thereby assisting the search for the unobserved region of interest using the unobserved image. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 FIG. 1 is an overall configuration diagram of an endoscope system including an image processing device according to an embodiment.

[0091] Figure 2 This is a functional block diagram of an endoscope system.

[0092] Figure 3 This is a functional block diagram of the image processing device according to the first embodiment.

[0093] Figure 4 This is a flowchart showing the procedure of the image processing method according to the first embodiment.

[0094] Figure 5 is a schematic diagram of the observed image.

[0095] Figure 6 is a schematic diagram of the first image.

[0096] Figure 7 This is a schematic diagram of a screen when a user input signal indicating a display of an unobserved area of ​​interest is acquired.

[0097] Figure 8 Schematic diagram of a non-display area where no image display is observed.

[0098] Figure 9 This is a functional block diagram of an image processing device according to a second embodiment.

[0099] Figure 10 This is a flowchart showing the procedure of the image processing method according to the second embodiment.

[0100] Figure 11 is a schematic diagram of a second image to which image processing is applied to increase the degree of emphasis.

[0101] Figure 12 is a schematic diagram of a method of displaying multiple second images.

[0102] Figure 13 This is a schematic diagram of selecting a second image in a method of displaying a plurality of second images.

[0103] Figure 14 This is a functional block diagram of an image processing device according to a third embodiment.

[0104] Figure 15 This is a flowchart showing the procedure of the image processing method according to the third embodiment.

[0105] Figure 16 This is an explanatory diagram of an example of the third image processing. DETAILED DESCRIPTION

[0106] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, the same components are denoted by the same reference numerals, and duplicate descriptions are omitted as appropriate.

[0107] [Overall structure of the endoscope system]

[0108] Figure 1 1 is an overall configuration diagram of an endoscope system including an image processing device according to an embodiment. The endoscope system 10 includes an endoscope body 100, a processor device 200, a light source device 300, and a monitor 400. In addition, the figure shows a portion of the distal end rigid portion 116 of the endoscope body 100 in an enlarged manner.

[0109] [Example of the structure of the endoscope body]

[0110] The endoscope body 100 includes a handheld operating unit 102 and an insertion unit 104. A user grasps and operates the handheld operating unit 102, inserting the insertion unit 104 into the body of a subject to observe the subject's interior. The term "user" is synonymous with physician and operator. The term "subject" is synonymous with patient and person under examination.

[0111] The hand operation unit 102 includes an air and water supply button 141, a suction button 142, a function button 143, and a camera button 144. The air and water supply button 141 receives an operation for an air supply instruction and a water supply instruction.

[0112] The suction button 142 receives suction instructions. The function button 143 is assigned various functions. The function button 143 receives instructions for various functions. The shooting button 144 receives shooting instructions. Shooting includes moving image shooting and still image shooting.

[0113] The handheld operation unit 102 functions as a user input unit. Although not shown, a foot switch may also be provided as the user input unit. The foot switch includes a pedal, a pedal indicator, and a cable. The cable is connected to the processor device 200.

[0114] The user can operate the foot switch to send a user input signal to the processor device 200. The processor device 200 receives the user input signal sent from the foot switch and performs processing corresponding to the user input signal.

[0115] The insertion section 104 includes a flexible section 112, a curved section 114, and a distal rigid section 116. These sections are arranged in this order, starting from the side of the hand-side operation section 102. Specifically, the proximal side of the distal rigid section 116 is connected to the curved section 114, the proximal side of the curved section 114 is connected to the flexible section 112, and the proximal side of the insertion section 104 is connected to the hand-side operation section 102.

[0116] The user can operate the hand operation unit 102 to bend the bending portion 114 and change the direction of the distal rigid portion 116 up, down, left, and right. The distal rigid portion 116 includes an imaging unit, an illumination unit, and a forceps opening 126.

[0117] exist Figure 1 The photographic lens 132 constituting the imaging unit is shown in FIG. In addition, the lighting lens 123A and the lighting lens 123B constituting the illumination unit are shown in FIG. Figure 2 In addition, the lighting unit is denoted by 123. Figure 2 Shown in.

[0118] When observing and handling, respond to Figure 2 The operation of the illustrated operation unit 208 causes at least one of white light and narrowband light to be output via the illumination lens 123A and the illumination lens 123B.

[0119] When the air and water supply button 141 is operated, cleaning water is discharged from the water supply nozzle, or air is discharged from the air supply nozzle. The cleaning water and air are used to clean the lighting lens 123A and other components. The water supply nozzle and air supply nozzle are not shown in the figure. Alternatively, they can be shared.

[0120] The forceps opening 126 is connected to a conduit. A treatment instrument is inserted into the conduit. The treatment instrument is supported so that it can move forward and backward appropriately. When removing a tumor, etc., the treatment instrument can be used to perform the necessary treatment. In addition, the conduit connected to the forceps opening 126 is omitted from the illustration.

[0121] Figure 2 : is a functional block diagram of an endoscope system. The endoscope body 100 includes an imaging unit 130. The imaging unit 130 is arranged inside the distal rigid portion 116. The imaging unit 130 includes a photographic lens 132, an imaging element 134, a driving circuit 136, and an analog front end (AFE) 138. Figure 2 The AFE shown is an abbreviation for Analog Front End.

[0122] The imaging lens 132 is disposed on the distal end surface 116A of the distal rigid portion 116. An imaging element 134 is disposed on the opposite side of the imaging lens 132 from the distal end surface 116A. A CMOS image sensor can be used as the imaging element 134. Alternatively, a CCD image sensor can be used as the imaging element 134. CMOS stands for Complementary Metal-Oxide Semiconductor. CCD stands for Charge Coupled Device.

[0123] A color image sensor can be applied to the image sensor 134. An example of a color image sensor is an image sensor equipped with color filters corresponding to RGB. RGB is the initials of Red, Green, and Blue, which are the English symbols for red, green, and blue, respectively.

[0124] A monochrome imaging element can be applied to the imaging element 134. When a monochrome imaging element is applied to the imaging element 134, the imaging unit 130 can switch the wavelength band of incident light on the imaging element 134 to perform plane-sequential or color-sequential imaging.

[0125] The drive circuit 136 supplies the imaging element 134 with various timing signals necessary for the operation of the imaging element 134 based on a control signal transmitted from the processor device 200 .

[0126] The analog front end 138 includes an amplifier, a filter, and an analog-to-digital converter. AD stands for the initials of analog and digital, respectively. The analog front end 138 performs amplification, noise reduction, and analog-to-digital conversion on the output signal from the imaging element 134. The output signal from the analog front end 138 is transmitted to the processor device 200.

[0127] The optical image of the observation object is formed on the light receiving surface of the imaging element 134 via the photographic lens 132. The imaging element 134 converts the optical image of the observation object into an electrical signal. The electrical signal output from the imaging element 134 is transmitted to the processor device 200 via a signal line.

[0128] The lighting unit 123 is disposed in the distal rigid portion 116. The lighting unit 123 includes a lighting lens 123A and a lighting lens 123B. The lighting lens 123A and the lighting lens 123B are disposed adjacent to the imaging lens 132 on the distal end surface 116A.

[0129] The lighting unit 123 includes a light guide 170. The emission end of the light guide 170 is arranged at a position on the opposite side of the distal end surface 116A of the lighting lens 123A and the lighting lens 123B.

[0130] The light guide 170 is inserted into Figure 1 The insertion portion 104, the hand operation portion 102, and the universal cable 106 are shown. The incident end of the light guide 170 is arranged inside the light guide connector 108. The endoscope body 100 described in the embodiment corresponds to an example of an endoscope.

[0131] [Configuration Example of Processor Device]

[0132] The processor device 200 includes an image input controller 202, an image processing unit 204, and a video output unit 206. The image input controller 202 acquires an electrical signal corresponding to an optical image of an observation object, which is transmitted from the endoscope body 100.

[0133] The image processing unit 204 generates an endoscopic image of the object to be observed based on an electrical signal corresponding to the optical image of the object to be observed, i.e., an imaging signal. In addition, the term "image" in this specification may include the image itself and image data representing the image. An image may include a dynamic image and a still image. In addition, an endoscopic image is shown in FIG. 38 with the addition of symbol 38. Figure 3 middle.

[0134] The image processing unit 204 can perform image quality correction on the imaging signal using digital signal processing such as white balance processing and shading correction processing. The image processing unit 204 can also attach supplementary information specified in the DICOM standard to the endoscopic image. DICOM stands for Digital Imaging and Communications in Medicine.

[0135] The video output unit 206 transmits a display signal representing an image generated by the image processing unit 204 to the monitor 400. The monitor 400 displays an image of an observation target.

[0136] The processor device 200 Figure 1 When the imaging button 144 shown is operated, the image input controller 202 and the image processing unit 204 are operated in response to an imaging instruction signal transmitted from the endoscope body 100 .

[0137] When the processor device 200 receives a freeze instruction signal indicating still image capture from the endoscope body 100, it generates a still image of a frame image based on the operation timing of the capture button 144 using the image processing unit 204. The processor device 200 displays the still image on the monitor 400. The frame image is denoted by a reference numeral 38B and is shown in FIG. Figure 3 The still image is shown in the figure with the symbol 39 attached. Figure 3 middle.

[0138] The processor device 200 includes a communication control unit 205. The communication control unit 205 controls communications with devices connected via in-hospital systems and the hospital's LAN. The communication control unit 205 can employ a communication protocol based on the DICOM standard. An example of an in-hospital system is an HIS (Hospital Information System). LAN stands for Local Area Network.

[0139] The processor device 200 includes a storage unit 207. The storage unit 207 stores endoscopic images captured using the endoscope body 100. The storage unit 207 may also store various information accompanying the endoscopic images.

[0140] The processor device 200 includes an operation unit 208. The operation unit 208 outputs an instruction signal in response to a user's operation. The operation unit 208 can be implemented by a keyboard, a mouse, a joystick, or the like.

[0141] The user input signal sent from the operation unit 208 is sent to the CPU 210. The CPU 210 receives the user input signal sent from the operation unit 208 and performs control corresponding to the received user input signal. The operation unit 208 may include the above-mentioned foot switch.

[0142] CPU 210 obtains Figure 1 The user input signal sent by the hand operation unit 102 is used to control the endoscope body 100 , the processor device 200 , and the light source device 300 according to the acquired user input signal.

[0143] The processor device 200 includes a sound processing unit 209 and a speaker 209A. The sound processing unit 209 generates a sound signal representing information to be reported as sound. The speaker 209A converts the sound signal generated by the sound processing unit 209 into sound. Examples of the sound output from the speaker 209A include messages, voice guidance, and warning sounds.

[0144] The processor device 200 includes a CPU 210, a ROM 211, and a RAM 212. CPU stands for Central Processing Unit, ROM stands for Read Only Memory, and RAM stands for Random Access Memory.

[0145] The CPU 210 functions as an overall control unit of the processor device 200. The CPU 210 functions as a memory controller that controls the ROM 211 and the RAM 212. The ROM 211 stores various programs and control parameters used in the processor device 200.

[0146] The RAM 212 is used as a temporary storage area for data in various processes and as a processing area for calculation processing using the CPU 210. The RAM 212 can be used as a buffer memory for acquiring endoscopic images.

[0147] The processor device 200 performs various processes on the endoscopic image captured by the endoscope body 100, and displays the endoscopic image and various information accompanying the endoscopic image on the monitor 400. The processor device 200 stores the endoscopic image and various information accompanying the endoscopic image.

[0148] That is, during endoscopic inspection using the endoscope body 100 , the processor device 200 performs display of endoscopic images and the like using the monitor 400 , output of audio information using the speaker 209A, and various processes on endoscopic images.

[0149] The processor device 200 can function as an image processing device that performs predetermined processing on medical images using components such as the image processing unit 204. Examples of the predetermined processing include region of interest detection processing, region of interest emphasis processing, region of interest classification processing, medical image identification processing, and region of interest measurement processing.

[0150] [Hardware Structure of Processor Device]

[0151] A computer can be applied to the processor device 200. The computer can implement the functions of the processor device 200 by applying the following hardware and executing a predetermined program. In addition, the program is synonymous with software.

[0152] The processor device 200 can use various processors as signal processing units for performing signal processing. Examples of processors include CPUs and GPUs (Graphics Processing Units). CPUs are general-purpose processors that execute programs and function as signal processing units. GPUs are processors specifically designed for image processing. Processor hardware can use circuits that combine circuit elements such as semiconductor devices. Each control unit has a ROM for storing programs and a RAM that serves as a work area for various operations.

[0153] Two or more processors may be applied to a single signal processing unit. The two or more processors may be of the same type or different types. Furthermore, a single processor may be applied to multiple signal processing units. Furthermore, the processor device 200 described in the embodiments corresponds to an example of an endoscope control device.

[0154] [Configuration Example of Light Source Device]

[0155] The light source device 300 includes a light source 310, an aperture 330, a condenser lens 340, and a light source control unit 350. The light source device 300 allows observation light to enter the light guide 170. The light source 310 includes a red light source 310R, a green light source 310G, and a blue light source 310B. The red light source 310R, the green light source 310G, and the blue light source 310B emit red, green, and blue narrowband light, respectively.

[0156] Light source 310 can generate illumination light by combining any combination of red, green, and blue narrowband lights. For example, light source 310 can combine red, green, and blue narrowband lights to generate white light. Furthermore, light source 310 can combine any two of red, green, and blue narrowband lights to generate narrowband light.

[0157] Light source 310 can generate narrowband light using any of red, green, and blue narrowband light. Light source 310 can selectively switch between emitting white light and narrowband light. Narrowband light is synonymous with special light. Light source 310 can include an infrared light source that emits infrared light, an ultraviolet light source that emits ultraviolet light, and so on.

[0158] The light source 310 may be a light source that emits white light, a filter that transmits white light, and a filter that transmits narrowband light. This light source 310 can switch between the filter that transmits white light and the filter that transmits narrowband light, selectively emitting either white light or narrowband light.

[0159] The filter for passing the narrowband light may include a plurality of filters corresponding to different frequency bands. The light source 310 may selectively switch the plurality of filters corresponding to different frequency bands to selectively emit a plurality of narrowband lights having different frequency bands.

[0160] The light source 310 can be of a type and wavelength band corresponding to the type of observation object and the purpose of observation. Examples of the type of light source 310 include laser light sources, xenon light sources, and LED light sources. LED is an abbreviation for Light-Emitting Diode.

[0161] When the light guide connector 108 is connected to the light source device 300, observation light emitted from the light source 310 reaches the incident end of the light guide 170 via the aperture 330 and the condenser lens 340. The observation light is irradiated onto the observation object via the light guide 170 and the illumination lens 123A.

[0162] The light source control unit 350 sends control signals to the light source 310 and the aperture 330 based on the instruction signal sent from the processor device 200. The light source control unit 350 controls the illumination of the observation light emitted from the light source 310, switching of the observation light, and on / off of the observation light.

[0163] [Configuration Example of Image Processing Device According to First Embodiment]

[0164] 〔summary〕

[0165] If a doctor operating the endoscope body 100 overlooks a region of interest such as a lesion in an observation image and proceeds to endoscopic examination, the doctor may notice that the region of interest was missed due to triggers such as an audio notification or an enhanced image of the region of interest. If the doctor notices that the region of interest was missed, he or she may operate the endoscope body 100 to search for the unobserved region of interest.

[0166] When a doctor sends a user input signal indicating redisplay of an unobserved region of interest, the image processing apparatus shown in this embodiment reads a frame image including the unobserved region of interest, that is, an unobserved image, and displays it on the screen. Figure 1 The unobserved image is displayed in an area different from the observed image display area where the observed image is displayed. This allows the doctor to compare the observed image with the unobserved image and search for a missed area of ​​interest in the observed image.

[0167] The so-called observation image is a real-time endoscopic image of the observation object displayed on the monitor 400 during endoscopic examination. The observation image can be a dynamic image or a still image. In addition, the observation object described in the embodiment is equivalent to an example of a subject. The monitor 400 described in the embodiment is equivalent to an example of a display device.

[0168] The real-time endoscopic image may be displayed on the monitor 400 after a certain delay period due to signal processing, etc. from the timing of imaging. The image processing apparatus described in this embodiment is synonymous with a medical image processing apparatus and a medical image processing apparatus. The same applies to the second and third embodiments.

[0169] Figure 3 This is a functional block diagram of an image processing device according to the first embodiment. Figure 2 The image processing device 14 is implemented by components such as the image processing unit 204 in the processor device 200. The image processing device 14 includes an image acquisition unit 40, a user input signal acquisition unit 42, a display control unit 44, and a notification unit 46.

[0170] The image processing device 14 includes a region of interest detection unit 50, an unobserved condition determination unit 51, a first image processing unit 52, an endoscopic image storage unit 70, an unobserved image storage unit 71, an observed image storage unit 71A, and a first image processing result storage unit 72. Each unit will be described below.

[0171] [Image acquisition unit]

[0172] The image acquisition unit 40 acquires the image using Figure 1 The endoscope body 100 shown here captures an endoscopic image 38 . Hereinafter, acquisition of the endoscopic image 38 may include acquisition of a dynamic image 38A, acquisition of a frame image 38B, and acquisition of a still image 39 .

[0173] The image acquisition unit 40 stores the endoscopic image 38 in the endoscopic image storage unit 70. The image acquisition unit 40 may acquire the endoscopic image 38 from the processor device 200 via a cable that transmits a signal representing the endoscopic image 38.

[0174] The image acquisition unit 40 may acquire the endoscopic image 38 from the processor device 200 via an information storage medium such as a memory card. The image acquisition unit 40 may acquire the endoscopic image 38 via a communication network.

[0175] The image acquisition unit 40 can acquire a moving image 38A composed of time-series frame images 38B. The image acquisition unit 40 can acquire a still image 39 when still image capture is performed during the capture of the moving image 38A. The image acquisition unit 40 corresponds to Figure 2 The image input controller 202 is shown.

[0176] [User input signal acquisition unit]

[0177] The user input signal acquisition section 42 acquires a user input signal transmitted from the operation section 30 in response to a user's operation of the operation section 30 . Figure 3 The operating unit 30 shown may include Figure 1 The hand operation unit 102 shown, Figure 2 The operating unit 208 shown and the foot switch not shown are also shown.

[0178] The user input signal acquisition unit 42 may include a microphone for acquiring the user's voice. In other words, the user input signal acquisition unit 42 may acquire a user input signal based on the user's voice.

[0179] The operation unit 30 described in the embodiment corresponds to an example of a user operation unit. The microphone described in the embodiment corresponds to an example of a voice acquisition unit that acquires the user's voice.

[0180] [Display control unit]

[0181] The display control unit 44 transmits a display signal representing the endoscopic image 38 to the monitor 400. The monitor 400 displays the endoscopic image 38 as an observation image. When the display control unit 44 detects a region of interest from the endoscopic image 38, it transmits a display signal representing a first image obtained by performing first image processing on the endoscopic image 38 using the first image processing unit 52 to the monitor 400.

[0182] When the user input signal acquisition unit 42 acquires a user input signal indicating the display of the unobserved region of interest, the display control unit 44 transmits a display signal indicating the unobserved image to the monitor 400. The monitor 400 displays the unobserved image.

[0183] The display control unit 44 performs non-display of the unobserved image based on the non-display condition of the unobserved image displayed on the monitor 400. Figure 2 The video output section 206 is shown.

[0184] Furthermore, the display control unit 44 described in the embodiment corresponds to an example of a display signal transmitting unit that transmits a first display signal representing an observed image to the display device. The display signal representing the endoscopic image 38 described in the embodiment corresponds to an example of the first display signal. The display signal representing an unobserved image described in the embodiment corresponds to an example of the second display signal.

[0185] [Hoshin-kumi-bu]

[0186] When the region of interest is detected from the frame image 38B constituting the endoscopic image 38, the notification unit 46 issues a notification indicating the detection of the region of interest in the endoscopic image 38. The notification unit 46 may issue a notification using Figure 2 The notification unit 46 can perform notifications such as text messages using the monitor 400.

[0187] [Region of Interest Detection Department]

[0188] The attention area detection unit 50 uses a learner such as a CNN (Convolutional Neural Network) to detect an attention area from the frame image 38B acquired by the image acquisition unit 40. The attention area detection unit 50 derives a feature value of the frame image 38B and determines the presence of an attention area based on the feature value.

[0189] The region of interest detection unit 50 applies a learned learner that has learned the pair of the frame image 38B and the region of interest in the frame image 38B as learning data. Examples of region of interest detection include lesion detection and specific organ detection.

[0190] [Unobserved Condition Judgment Section]

[0191] The unobserved condition determination unit 51 determines whether the frame image 38B in which the attention region is detected satisfies the conditions of the frame image 38B stored in the unobserved image storage unit 71. In other words, the unobserved condition determination unit 51 determines whether the frame image 38B in which the attention region is detected is an unobserved image.

[0192] When observing the region of interest, it is considered that the doctor stops or slows down the movement of the imaging unit 130. As a result, when observing the region of interest, the number of frame images 38B including the same region of interest increases compared to when not observing the region of interest.

[0193] Therefore, a predetermined number of frame images 38B containing the same region of interest is pre-set. If the number of frame images 38B containing the same region of interest falls below the predetermined number within a predetermined period, the non-observation condition determination unit 51 determines the image as an unobserved image. The predetermined number can be determined based on the frame rate of the imaging unit 130 and the movement speed of the imaging unit 130. Any integer greater than 1 can be used as the predetermined number. While frame images 38B containing the same region of interest are often continuous, there may be cases where the detection of the region of interest is interrupted due to screen flickering. The predetermined number can be the number of continuous or discontinuous frame images 38B.

[0194] The movement of the imaging unit 130 means the movement of the imaging unit 130 along the movement path of the endoscope body 100 during endoscopic inspection. In addition, the movement speed may include the concept of speed indicating the absolute value of the movement speed.

[0195] When observing a region of interest, it is thought that the imaging direction of the imaging unit 130 changes more frequently than when not observing the region of interest. Thus, the non-observation condition determination unit 51 can determine whether a frame image 38B has not been observed by the doctor based on the amount of change between frame images 38B, such as a change in the motion vector of the imaging unit 130. Specifically, the non-observation condition determination unit 51 can determine that an image is an unobserved image if the amount of change between frame images 38B is greater than a predetermined threshold. Other examples of the amount of change between frame images 38B include the value of a cross-correlation function between frame images 38B.

[0196] Furthermore, observation of the region of interest is often performed in the center of the image. Therefore, if the region of interest remains outside the image and leaves the frame (frame-out), the image may be determined as unobserved. In other words, the unobserved condition determination unit 51 may determine that the image is unobserved if the same region of interest remains outside the center of the image for a predetermined period of time. Here, the center of the image is defined as an area that includes the center or center of gravity of the image and excludes the edges of the image. On the other hand, the outer edge of the image is defined as an area that excludes the center or center of gravity of the image and includes the edges of the image.

[0197] That is, the unobserved condition determination unit 51 may determine that the image is an unobserved image if at least any one of the number of frame images 38B including the same region of interest, the amount of change between frame images 38B, and the position of the region of interest on the screen indicates that observation has not been performed.

[0198] When the frame image 38B in which the attention region is detected satisfies the condition of the unobserved image stored in the unobserved image storage unit 71 , the unobserved condition determination unit 51 stores the frame image 38B as an unobserved image in the unobserved image storage unit 71 .

[0199] On the other hand, when the frame image 38B detecting the attention area does not satisfy the conditions for storing the unobserved image in the unobserved image storage 71 , the unobserved condition determination unit 51 stores the frame image 38B as an observed image in the observed image storage 71A.

[0200] [First Image Processing Unit]

[0201] When the first image processing unit 52 detects the region of interest from the frame image 38B constituting the endoscopic image 38, it performs first image processing on the frame image 38B including the region of interest. The endoscopic image 38 and the frame image 38B including the region of interest correspond to the observation image described above.

[0202] The first image processing can be applied to highlight the region of interest. The first image processing unit 52 generates an emphasized image that emphasizes the region of interest detected from the frame image 38B. Examples of the emphasized image include a bounding box that overlaps the region of interest, a predetermined color or pattern that is applied to the region of interest, and the like.

[0203] The first image processing unit 52 identifies the position of the target region and defines the position of the enhanced image based on the position of the target region. The first image processing unit 52 identifies the size of the target region and defines the size of the enhanced image based on the size of the target region.

[0204] The first image processing unit 52 stores the results of the first image processing in the first image processing result storage unit 72. The first image processing result storage unit 72 may store the first image in which the emphasis image is added to the frame image 38B in which the region of interest is detected. The first image processing result storage unit 72 may also store a combination of information about the frame image 38B containing the region of interest and the emphasis image associated with the frame image 38B containing the region of interest. In other words, the first image processing unit 52 may store the file for the frame image 38B and the file for the emphasis image in separate files.

[0205] The first image processing unit 52 uses the display control unit 44 to display the frame image 38B superimposed with the emphasis image as the observation image on the monitor 400. The notification unit 46 can use the frame image 38B superimposed with the emphasis image to issue a notification indicating the detection of the target region in the endoscopic image 38.

[0206] [Endoscope image storage unit]

[0207] The endoscopic image storage unit 70 stores the endoscopic image 38 acquired by the image acquisition unit 40. When still image capturing is performed, the endoscopic image storage unit 70 stores the still image 39 generated by the still image capturing.

[0208] [Unobserved image storage unit]

[0209] The unobserved image storage unit 71 stores unobserved images. When a user input signal indicating the display of an unobserved region of interest is acquired, the display control unit 44 reads out the unobserved images stored in the unobserved image storage unit 71 .

[0210] When a new unobserved image is generated while an unobserved image is already stored, the unobserved image storage unit 71 may store the new unobserved image in place of the stored unobserved image.

[0211] The unobserved image storage unit 71 may also specify the number of unobserved images to be stored. In addition, the unobserved image storage unit 71 may also specify the storage period of unobserved images. That is, the unobserved image storage unit 71 may store unobserved images that are likely to be displayed again and delete unobserved images that are less likely to be displayed again. Figure 3 The illustration of unobserved images is omitted.

[0212] [Observed image storage unit]

[0213] The observed image storage unit 71A stores observed images. When a user input signal indicating the display of an observed region of interest is acquired, the display control unit 44 reads out the observed images stored in the observed image storage unit 71A.

[0214] [First Image Processing Result Storage Unit]

[0215] The first image processing result storage unit 72 stores the processing result of the first image processing performed by the first image processing unit 52. The first image processing result may be applied to the first image or to a combination of the frame image 38B and the information of the target region corresponding to the frame image 38B.

[0216] One or more storage elements may be used for each of the endoscopic image storage unit 70, the unobserved image storage unit 71, the observed image storage unit 71A, and the first image processing result storage unit 72. Specifically, the image processing apparatus 14 may include four storage elements corresponding to each of the endoscopic image storage unit 70, the unobserved image storage unit 71, the observed image storage unit 71A, and the first image processing result storage unit 72.

[0217] The endoscopic image storage unit 70, the unobserved image storage unit 71, the observed image storage unit 71A, and the first image processing result storage unit 72 can be configured as any two to four components using a single storage element. For example, the endoscopic image storage unit 70, the unobserved image storage unit 71, and the observed image storage unit 71A can be configured using a single storage element.

[0218] [Procedure of the Image Processing Method According to the First Embodiment]

[0219] Figure 4 This is a flowchart showing the steps of the image processing method according to the first embodiment. The image processing method described in this embodiment is synonymous with medical image processing methods and medical image processing methods. This also applies to the second and third embodiments.

[0220] In the endoscopic image acquisition step S10, Figure 3 The image acquisition unit 40 shown acquires the endoscopic image 38. In the endoscopic image acquisition step S10, the image acquisition unit 40 stores the endoscopic image 38 in the endoscopic image storage unit 70. After the endoscopic image acquisition step S10, the process proceeds to the region of interest detection step S12.

[0221] In the region of interest detection step S12, the region of interest detection unit 50 detects the region of interest from each frame image 38B constituting the endoscopic image 38. In the region of interest detection step S12, the region of interest detection unit 50 may detect the region of interest from all frame images 38B of the moving image 38A, or may detect the region of interest from frame images 38B at predetermined intervals. Following the region of interest detection step S12, the process proceeds to the notification step S14.

[0222] In the notification step S14, the notification unit 46 performs notification indicating the detection of the attention area in the frame image 38B. A notification stop step for stopping the notification may also be performed. After the notification step S14, the process proceeds to the first image processing step S16.

[0223] In the first image processing step S16, the first image processing unit 52 performs the first image processing on the frame image 38B in which the region of interest has been detected. After the first image processing step S16, the process proceeds to the first image processing result storage step S17. Alternatively, the notification step S14 may be performed after the first image processing step S16.

[0224] In the first image processing result storage step S17, the first image processing unit 52 stores the processing results of the first image processing in the first image processing result storage unit 72. After the first image processing result storage step S17, the process proceeds to the unobserved condition determination step S18. A display step for displaying the first image may be performed after the first image processing result storage step S17. Furthermore, the first image processing step S16 and the first image processing result storage step S17 may be omitted.

[0225] In the unobserved condition determination step S18 , the unobserved condition determination unit 51 determines whether the frame image 38B in which the attention region is detected satisfies the unobserved condition. That is, in the unobserved condition determination step S18 , the unobserved condition determination unit 51 determines whether the frame image 38B in which the attention region is detected is an unobserved image.

[0226] In the non-observation condition determination step S18 , if the non-observation condition determination unit 51 determines that the non-observation condition is not satisfied, the determination is No. If the determination is No, the process proceeds to the user input signal acquisition determination step S20 .

[0227] On the other hand, in the unobserved condition determination step S18, if the unobserved condition determination unit 51 determines that the unobserved condition is satisfied, the determination is Yes. If the determination is Yes, the process proceeds to the unobserved image storage step S19.

[0228] In the unobserved image storage step S19, the unobserved condition determination unit 51 stores the unobserved image in the unobserved image storage unit 71. After the unobserved image storage step S19, the process proceeds to the user input signal acquisition determination step S20. The observed image storage step of storing the observed image in the observed image storage unit 71A can be performed before, during, or in parallel with the unobserved image storage step S19.

[0229] In the user input signal acquisition determination step S20, the user input signal acquisition unit 42 determines whether a user input signal indicating the display of the unobserved region of interest has been acquired. In the user input signal acquisition determination step S20, if it is determined that the user input signal acquisition unit 42 has not acquired a user input signal indicating the display of the unobserved region of interest, the determination is "No." If the determination is "No," the process proceeds to the final frame image determination step S32.

[0230] On the other hand, in the user input signal acquisition determination step S20, if it is determined that the user input signal acquisition unit 42 has acquired a user input signal indicating a display of an unobserved region of interest, the determination is Yes. If the determination is Yes, the process proceeds to the display signal transmission step S26. The user input signal acquisition determination step S20 is an example of a user input signal acquisition step.

[0231] In the display signal transmission step S26, the display control unit 44 transmits a display signal indicating non-observation to the monitor 400. The monitor 400 displays non-observation. After the display signal transmission step S26, the process proceeds to the non-display condition determination step S28.

[0232] In the non-display condition determination step S28, the display control unit 44 determines whether the non-display condition for the unobserved image displayed on the monitor 400 is satisfied. In the non-display condition determination step S28, if the display control unit 44 determines that the non-display condition for the unobserved image is not satisfied, the determination is No. If the determination is No, the non-display condition determination step S28 is continued until the determination is Yes in the non-display condition determination step S28.

[0233] On the other hand, in the non-display condition determination step S28, if the display control unit 44 determines that the non-display condition for the unobserved image is satisfied, the determination is yes. If the determination is yes, the process proceeds to the display signal transmission stop step S30. The details of the non-display condition for the unobserved image will be described later.

[0234] In the display signal transmission stop step S30, the display control unit 44 stops transmitting the display signal representing the unobserved image displayed on the monitor 400. Specifically, in the display signal transmission stop step S30, the display control unit 44 sets the unobserved image to non-display. After the display signal transmission stop step S30, the process proceeds to the final frame image determination step S32.

[0235] In the final frame image determination step S32 , the region of interest detection unit 50 determines whether or not the region of interest detection process has been performed on the final frame image 38B of the endoscopic image 38 acquired by the image acquisition unit 40 .

[0236] In the final frame image determination step S32, if it is determined that the region of interest detection unit 50 has not performed the detection process for the region of interest on the final frame image 38B, the determination is No. If the determination is No, the process proceeds to the endoscopic image acquisition step S10, and the steps from the endoscopic image acquisition step S10 to the final frame image determination step S32 are repeatedly performed until a Yes determination is made in the final frame image determination step S32.

[0237] On the other hand, in the final frame image determination step S32 , if it is determined that the attention area detection unit 50 has performed attention area detection processing on the final frame image 38B, the determination is Yes. If the determination is Yes, the image processing device 14 ends the image processing method.

[0238] [Example of display of endoscopic image]

[0239] Figure 5 Schematic diagram of an observation image. In this figure, a schematic diagram of an observation image of the large intestine is shown as an endoscopic image 38. Figures 6 to 8 、 Figures 11 to 13 ,as well as Figure 16 The same applies to the respective figures of FIG. If the region of interest is not detected from the endoscopic image 38 , the endoscopic image 38 is displayed in real time as an observation image in the observation image display area 404 on the screen 402 of the monitor 400 .

[0240] Right now, Figure 5 The screen 402 of the monitor 400 shown displays a moving image 38A of an endoscopic image 38 during endoscopic examination as an observation image in real time. Figure 5 The illustrated observation image shows a case where the region of interest is not detected from the frame image 38B constituting the endoscopic image 38 .

[0241] Figure 6 38B is a schematic diagram of the first image. When the region of interest 510 is detected from the frame image 38B constituting the endoscopic image 38, Figure 3 The illustrated first image processing unit 52 performs first image processing on the frame image 38B based on the detection result of the attention region 510 .

[0242] Figure 6 The screen 402 of the monitor 400 shown displays a first image 500 in which emphasis processing of a notice region 510 is applied as first image processing, and a boundary frame 512 surrounding the notice region 510 is superimposed on the frame image 38B in which the notice region 510 is detected.

[0243] When a plurality of first images 500 are generated for the same region of interest 510 , any one or more first images 500 may be stored. Figure 6 A first image 500 corresponding to the frame image 38B at an arbitrary timing is shown.

[0244] Figure 7402 is a schematic diagram of a screen when a user input signal indicating the display of an unobserved region of interest is received. When a doctor recognizes the detection of the region of interest 510 due to the notification of the detection of the region of interest 510 , he or she may want to display the missed region of interest 510 on the screen 402 .

[0245] exist Figure 3 When the user input signal acquisition unit 42 receives a user input signal indicating the display of the unobserved region of interest 510, the display control unit 44 reads the unobserved image 39A from the unobserved image storage unit 71 and displays the read unobserved image 39A in the unobserved image display area 406 of the screen 402. The display control unit 44 displays the endoscopic image 38 in real time in the observed image display area 404 of the screen 402. Thus, the doctor can observe the endoscopic image 38 using the unobserved image 39A displayed in the unobserved image display area 406 and find the region of interest 510 that was missed.

[0246] exist Figure 7 , a second image 520 is shown in which a region of interest 510 and a bounding box 512 are superimposed on the unobserved image 39A. That is, the unobserved image display area 406 can display the unprocessed unobserved image 39A, or it can display the second image 520 after the second image processing has been applied to the unobserved image 39A. Details of the second image processing will be described later.

[0247] [Non-display conditions for unobserved image display areas]

[0248] Figure 8 4 is a schematic diagram of non-display of the unobserved image display area. In the screen 402 shown in the figure, the second image 520 displayed in the unobserved image display area 406 is set to non-display. Figure 8 The second image 520 set to be non-displayed is shown using a dotted line.

[0249] The non-display condition for second image 520 can be applied when a predetermined period of time has elapsed since the display start timing of second image 520. Specifically, image processing device 14 includes a timer. The timer measures the period of time since the display start timing of second image 520. Display control unit 44 can non-display second image 520 based on the timer's measured value. The timer is not shown in the illustration.

[0250] Furthermore, the display control unit 44 described in the embodiment corresponds to an example of a display signal transmitting unit that stops transmitting the second display signal after a predetermined period of time has elapsed from the timing of transmitting the second display signal.

[0251] The non-display condition of the second image 520 may apply visual recognition of the same region of interest 510 as the region of interest 510 in the second image 520 . Figure 3The illustrated region of interest detection unit 50 determines the identity of the region of interest 510 in the second image 520 and the region of interest 510 detected from the endoscopic image 38 .

[0252] Figure 3 The illustrated region of interest detection unit 50 can compare the feature quantity of the region of interest 510 in the second image 520 with the feature quantity of the region of interest 510 detected from the endoscopic image 38, and determine the identity of the two based on the comparison result. The region of interest detection unit 50 can use information such as pixel value, number of pixels, and shape as the feature quantity of the region of interest 510.

[0253] When obtaining a determination result indicating that the region of interest 510 in the second image 520 is identical to the region of interest 510 in the endoscopic image 38 , the display control unit 44 hides the second image 520 displayed in the unobserved image display area 406 .

[0254] Figure 3 When the notification unit 46 determines that the region of interest 510 in the second image 520 is identical to the region of interest 510 in the endoscopic image 38, it can notify the determination result by using a voice notification, a text notification, etc. The non-display condition of the unobserved image display area 406 can also be applied to the unobserved image 39A when the unobserved image 39A is displayed in the unobserved image display area 406.

[0255] [Effects of the First Embodiment]

[0256] The image processing device 14 and the image processing method according to the first embodiment can achieve the following effects.

[0257] [1]

[0258] The region of interest detection unit 50 detects the region of interest 510 from the frame image 38B of the moving image 38A constituting the endoscopic image 38. The unobserved image storage unit 71 stores the unobserved image 39A containing the unobserved region of interest 510 in the endoscopic image 38 in which the region of interest 510 was detected. This allows the doctor to search for the region of interest 510 using the unobserved image 39A or the like, if the doctor recognizes that the region of interest 510 has been missed due to notification of the detection of the region of interest 510.

[0259] [2]

[0260] When receiving a user input signal indicating that the unobserved region of interest 510 is to be displayed, the display control unit 44 reads out the unobserved image 39A stored in the unobserved image storage unit 71 and displays the read unobserved image 39A or the second image 520 on the monitor 400. Thus, if the doctor recognizes that the region of interest 510 has been missed due to notification of detection of the region of interest 510, the doctor can search for the region of interest 510 using the unobserved image 39A displayed on the monitor 400.

[0261] 〔3〕

[0262] The display control unit 44 displays the unobserved image 39A or the second image 520 in the unobserved image display area 406, which is a region different from the observed image display area 404. This makes it easier for the doctor to search for the region of interest 510 in the observed image.

[0263] [4]

[0264] When receiving a user input signal indicating that the unobserved image 39A or the second image 520 should not be displayed, the display control unit 44 performs non-display of the unobserved image 39A or the second image 520. Thus, when the search target region of interest 510 is found, the unobserved image 39A, which may hinder observation, can be non-displayed.

[0265] 〔5〕

[0266] The display control unit 44 hides the unobserved image 39A or the second image 520 after a predetermined period has passed since the display start timing of the unobserved image 39A or the second image 520. This automatically hides the unobserved image 39A or the second image 520 that may interfere with observation.

[0267] [6]

[0268] The user input signal acquisition unit 42 acquires a user input signal sent from the operation unit and a user input signal corresponding to a user's voice using a microphone. Therefore, the user input signal acquisition unit 42 can use user input signals in various formats.

[0269] [7]

[0270] The first image processing unit 52 performs first image processing on the frame image 38B in which the region of interest 510 is detected. The display control unit 44 displays a first image 500 representing the processing result of the first image processing unit 52 in the observation image display area 404. This allows the doctor to visually recognize the detection of the region of interest in the observation image.

[0271] 〔8〕

[0272] The first image processing unit 52 generates a second image 520 by applying a second image processing, which is the same image processing as the first image processing, to the unobserved image 39A. The display control unit 44 displays the second image 520 in the unobserved image display area 406. This improves the visibility of the region of interest 510 in the second image 520.

[0273] [Modification of the Image Processing Device and Method According to the First Embodiment]

[0274] In the image processing apparatus and method according to the first embodiment, the unobserved image 39A is displayed using the monitor 400 based on the user input signal. However, the unobserved image 39A may be displayed using the monitor 400 based on the generation of the unobserved image 39A.

[0275] According to this modification, when the unobserved image 39A is generated, the unobserved image 39A can be automatically displayed on the monitor 400 .

[0276] [Image Processing Device According to Second Embodiment]

[0277] Next, an image processing device according to a second embodiment will be described. Figure 9 This is a functional block diagram of an image processing device according to the second embodiment. Figure 3 The image processing apparatus 14 shown in the figure is further provided with a second image processing unit 54 and a second image processing result storage unit 74 .

[0278] [Second Image Processing Unit]

[0279] The second image processing unit 54 is Figure 7 Specifically, when displaying the unobserved image 39A in the unobserved image display area 406 , the second image processing unit 54 changes the display mode of the second image 520 relative to the display mode of the first image 500 displayed in the observed image display area 404 .

[0280] The second image processing may apply an emphasis process that increases the emphasis level compared to the first image 500. The second image processing may apply a unified display of a plurality of different second images 520. The second image processing may apply a process of selecting one of the plurality of different second images 520.

[0281] The second image processing unit 54 stores the processing result of the second image processing in the second image processing result storage unit 74. The processing result of the second image processing may be applied to the second image 520. The processing result of the second image processing may also be applied to a combination of the unobserved image 39A and information indicating the processing result of the second image processing on the unobserved image 39A.

[0282] [Second Image Processing Result Storage Unit]

[0283] The second image processing result storage unit 74 stores the processing result of the second image processing performed by the second image processing unit 54. The second image processing result storage unit 74 may store the processing result of the second image processing selected from a plurality of processing results of the second image processing.

[0284] [Procedure of the Image Processing Method According to the Second Embodiment]

[0285] Figure 10 This is a flowchart showing the procedure of the image processing method according to the second embodiment. Figure 10 The flowchart shown is relative to Figure 4 The flowchart shown in FIG. 1 adds a second image processing step S22 and a second image processing result storage step S24 .

[0286] Figure 10 The steps from the endoscopic image acquisition step S10 to the user input signal acquisition and determination step S20 are shown in FIG. Figure 4 The steps from the endoscopic image acquisition step S10 to the user input signal acquisition determination step S20 are the same. If the user input signal acquisition determination step S20 determines "Yes," the process proceeds to the second image processing step S22.

[0287] In the second image processing step S22, Figure 9 The second image processing unit 54 shown performs the second image processing on the unobserved image 39A. After the second image processing step S22, the process proceeds to the second image processing result storage step S24.

[0288] In the second image processing result storage step S24, the second image processing unit 54 stores the result of the second image processing in the second image processing result storage unit 74. After the second image processing result storage step S24, the process proceeds to the display signal transmission step S26. Alternatively, the second image processing step S22 and the second image processing result storage step S24 may be performed between the first image processing result storage step S17 and the non-observation condition determination step S18.

[0289] [Specific Example of Second Image Processing]

[0290] Figure 11 is a schematic diagram of a second image to which image processing to enhance the degree of emphasis is applied. The unobserved image display area 406 shown in this figure displays the same Figure 8 The emphasis processing of the first image 500 is shown to increase the emphasis level of the second image 530 .

[0291] The second image 530 uses Figure 6 Bounding box 512 is shown as bounding box 532 that emphasizes region of interest 510. Compared to bounding box 512 applied to first image 500, bounding box 532 has a solid line instead of a dotted line, a thicker line, and a bold color such as red.

[0292] The processing of enhancing the degree of emphasis is not limited to Figure 11 For example, you can also change Figure 6 At least one of the type of frame line, the line width of the frame line, and the color of the frame line of the illustrated boundary box 512.

[0293] In the screen 402 where the second image 530 is displayed in the non-observed image display area 406, the real-time endoscopic image 38 is displayed in the observed image display area 404. When the doctor notices the notification indicating that the region of interest 510 has been detected, he / she operates Figure 1 The endoscope body 100 shown may refer to the second image 530 when searching for the region of interest 510 .

[0294] Figure 11 The second image 530 is shown with Figure 7 Compared with the unobserved image 39A displayed in the unobserved image display area 406 shown, the visibility of the attention area 510 can be improved.

[0295] Figure 12 is a schematic diagram of a method of displaying multiple second images. Figure 12 The second image 540 and the second image 542 shown correspond to the unobserved image 39A generated at different timings.

[0296] The second image 540 displays a bounding box 512A superimposed on the target area 510A. The second image 542 displays a bounding box 512B superimposed on the target area 510B. When multiple second images 540 are displayed, each of the second images 540 may be displayed in a time-division manner.

[0297] Figure 13 1 is a schematic diagram of selecting a second image in a method of displaying a plurality of second images. Figure 13 In the illustrated screen 402, the second image 542 selected from the second images 540 and 542 is highlighted. That is, when a user input signal indicating selection of the second image 540 is obtained, the second image processing unit 54 highlights the selected second image 542 based on the user input signal.

[0298] exist Figure 13In the screen 402 shown, the non-selected second image 540 can be set to non-display according to the selection of the second image 542. The highlighted display of the second image 542 can also be set to non-display according to the non-display of the non-selected second image 540. Figure 9 The user input signal acquisition unit 42 shown corresponds to an example of a selection unit.

[0299] Although Figure 12 and Figure 13 While the display method shows two second images 540, etc., three or more second images may also be displayed. When displaying multiple second images 540, etc., information indicating the timing of each acquisition may also be added. Preferably, the multiple second images 540, etc. are arranged in the order of the timings at which they were acquired. Furthermore, when selecting the second image by providing multiple second images 540, etc. as options, more than two second images may be selected.

[0300] according to Figure 12 and Figure 13 In the manner shown, a plurality of second images 540 and the like may be used as a plurality of search candidates. In addition, more than one search object may be selected from the plurality of search candidates.

[0301] As Figures 11 to 13 Other examples of the second image processing shown include changing at least one of the hue, lightness, and saturation of the region of interest 510 relative to the first image 500. Other examples of the second image processing include resolution conversion, such as lowering or increasing the resolution. Furthermore, other examples of the second image processing include displaying various information related to the second image 530, such as information related to the storage timing of the unobserved image 39A and the identification results of the region of interest 510 in the unobserved image 39A. Text and sound can be used to display these various information.

[0302] [Operation and Effect of the Second Embodiment]

[0303] The image processing device 14A and the image processing method according to the second embodiment can achieve the following effects.

[0304] [1]

[0305] The second image processing unit 54 generates a second image 530 or the like that has a different display format than the first image 500. The display control unit 44 displays the second image 530 or the like on the monitor 400. Thus, if the doctor recognizes that the region of interest 510 has been missed due to the notification of the detection of the region of interest 510, the doctor can search for the region of interest 510 using the second image 530 or the like.

[0306] [2]

[0307] The first image processing applies emphasis to the region of interest 510. The second image processing enhances the emphasis compared to the first image processing, thereby improving the visibility of the region of interest 510 in the second image 530 and the like.

[0308] 〔3〕

[0309] The monitor 400 displays a plurality of second images 540 etc. acquired at different timings. This allows searching for the region of interest 510 that was missed in the endoscopic image 38 from among the plurality of second images 540 etc.

[0310] [4]

[0311] One or more second images 542 etc. can be selected from a plurality of second images 540 etc. acquired at different timings. Thus, the second images 542 etc. used for searching the target region 510 can be set from a plurality of second images 540 etc. acquired at different timings.

[0312] [Image Processing Device According to Third Embodiment]

[0313] Next, an image processing device according to a third embodiment will be described. Figure 14 This is a functional block diagram of an image processing device according to the third embodiment. Figure 9 The image processing device 14A shown in the figure is additionally provided with a third image processing unit 56 and a third image processing result storage unit 76 .

[0314] The third image processing unit 56 performs third image processing on the unobserved image 39A or the second image 520 displayed in the unobserved image display area 406. The second image 520, etc., referred to herein, refers to Figure 7 The second image 520 shown, Figure 11 The second image 530 shown, Figure 12 At least one of the second image 540 and the second image 542 shown. The same applies to the following description.

[0315] The third image processing can apply at least any one of editing the unobserved image 39A or the second image 520, etc., storing the third image generated by editing the unobserved image 39A or the second image 520, etc., sending the unobserved image 39A or the second image 520, etc. to the external device 32, deleting the unobserved image 39A or the second image 520, etc., and rearranging multiple unobserved images 39A or multiple second images 520, etc.

[0316] When the third image processing performed by the third image processing unit 56 is editing of the unobserved image 39A or the second image 520, the third image processing result storage unit 76 stores a third image 560 generated by editing the unobserved image 39A or the second image 520. The editing of the unobserved image 39A or the second image 520 may include enlarging or reducing the unobserved image 39A or the second image 520.

[0317] When the user input signal acquisition unit 42 acquires a user input signal indicating that the unobserved image 39A or the second image 520 is to be transmitted to the external device 32, the third image processing unit 56 transmits the unobserved image 39A or the second image 520 to the external device 32. The external device 32 may be a storage device or the like connected to the image processing device 14B in a communicable manner. Figure 14 , the second image 520 and the like are shown being sent to the external device 32 .

[0318] When the user input signal acquisition unit 42 acquires a user input signal indicating deletion of the unobserved image 39A or the second image 520 , the third image processing unit 56 deletes the unobserved image 39A or the second image 520 .

[0319] When the user input signal acquisition unit 42 acquires a user input signal indicating rearrangement of the unobserved images 39A or the second images 520 , the third image processing unit 56 rearranges the arrangement of the unobserved images 39A or the second images 520 .

[0320] [Procedure of the Image Processing Method According to the Third Embodiment]

[0321] Figure 15 : is a flowchart showing the procedure of the image processing method involved in the third embodiment. Figure 15 , a process of applying the editing method of the second image 530 as the third image processing is shown. Figure 15 The flowchart shown is relative to Figure 10 The flowchart shown in the figure adds a third image processing instruction step S40 , a third image processing step S42 , and a third image processing result storage step S44 .

[0322] Figure 15 The steps from the endoscopic image acquisition step S10 to the display signal transmission step S26 are similar to those in FIG. Figure 10 The steps from the endoscopic image acquisition step S10 to the display signal transmission step S26 are the same. After the display signal transmission step S26, the process proceeds to the third image processing instruction step S40.

[0323] In the third image processing instruction step S40, Figure 14 The user input signal acquisition unit 42 shown in FIG. 4 acquires a third image processing instruction signal indicating the processing content of the third image processing. As an example of acquiring the third image processing instruction signal, there is an example of acquiring the third image processing instruction signal from FIG. Figure 14 The user input signal is sent from the operation unit 30. After the third image processing instruction step S40, the process proceeds to the third image processing step S42.

[0324] In the third image processing step S42, the third image processing unit 56 performs the third image processing on the unobserved image 39A or the second image 520 based on the third image processing instruction signal. After the third image processing step S42, the process proceeds to the third image processing result storage step S44.

[0325] In the third image processing result storage step S44, Figure 14 The third image processing result storage unit 76 shown stores the processing results of the third image processing. After the third image processing result storage step S44, the process proceeds to the non-display condition determination step S28. If the third image processing involves transmitting the unobserved image 39A or the second image 520 to the external device 32 or deleting the unobserved image 39A or the second image 520, the third image processing result storage step S44 is omitted.

[0326] Figure 15 The steps from the non-display condition determination step S28 to the final frame image determination step S32 are similar to those shown in FIG. Figure 10 The steps from the non-display condition determination step S28 to the final frame image determination step S32 shown are the same.

[0327] [Specific Example of the Third Image Processing]

[0328] Figure 16 This is an explanatory diagram of an example of the third image processing. The screen 402 shown in this figure replaces Figure 7 The third image 560 is displayed in addition to the second image 520 etc. The screen 402 may also display both the second image 520 etc. and the third image 560.

[0329] In the third image 560, Figure 7 The region of interest 510 of the second image 520 is shown to be magnified. Figure 16 The third image 560 shown contains Figure 7 The region of interest 510 is shown as an enlarged region of interest 510C and Figure 7 Bounding box 512 is shown as an enlarged bounding box 512C.

[0330] [Effects of the Third Embodiment]

[0331] The image processing device 14B and the image processing method according to the third embodiment can achieve the following effects.

[0332] [1]

[0333] The image processing device 14B includes a third image processing unit 56 that performs third image processing on the unobserved image 39A or the second image 520. This allows editing of the unobserved image 39A or the second image 520.

[0334] [2]

[0335] The third image processing can be applied to editing the unobserved image 39A or the second image 520, etc., storing the edited results of the unobserved image 39A or the second image 520, etc., transmitting the unobserved image 39A or the second image 520, etc. to an external device, deleting the unobserved image 39A or the second image 520, etc., and rearranging the unobserved image 39A or the second image 540, etc. in a manner such that a plurality of unobserved images 39A or a plurality of second images 540 are displayed. Thus, editing of the unobserved image 39A or the second image 520, etc. can be performed.

[0336] [Modification of the Endoscope System]

[0337] [Modification of Illumination Light]

[0338] An example of a medical image that can be acquired using the endoscope system 10 shown in the embodiment is a normal light image obtained by irradiating light in a white band or irradiating light in a plurality of wavelength bands as the white band.

[0339] Another example of a medical image that can be acquired using the endoscope system 10 described in the embodiment is an image obtained by irradiating light of a specific wavelength band. The specific wavelength band can be a band narrower than the white band. The following modifications are applicable.

[0340] <First Modification>

[0341] A first example of a specific wavelength band is a blue band or a green band within the visible light range. The wavelength band of the first example includes a wavelength band of 390 to 450 nanometers, or a wavelength band of 530 to 550 nanometers, and the light of the first example has a peak wavelength within the wavelength band of 390 to 450 nanometers, or a wavelength band of 530 to 550 nanometers.

[0342] <Second Modification>

[0343] A second example of a specific wavelength band is a red band in the visible range. The wavelength band in this second example includes a wavelength band between 585 nanometers and 615 nanometers, or between 610 nanometers and 730 nanometers. The light in this second example has a peak wavelength within this wavelength band.

[0344] <Third Modification>

[0345] A third example of a specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light of the third example has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients. The wavelength band of this third example includes 400±10 nanometers, 440±10 nanometers, 470±10 nanometers, or a wavelength band between 600 nanometers and 750 nanometers, and the light of the third example has a peak wavelength in 400±10 nanometers, 440±10 nanometers, 470±10 nanometers, or a wavelength band between 600 nanometers and 750 nanometers.

[0346] <Fourth Modification>

[0347] A fourth example of a specific wavelength band is the wavelength band used to observe fluorescence emitted by fluorescent substances in a living body and the excitation light that excites the fluorescent substances. For example, this is a wavelength band between 390 nanometers and 470 nanometers. Observation of fluorescence is sometimes referred to as fluorescence observation.

[0348] <Fifth Modification>

[0349] A fifth example of a specific wavelength band is a wavelength band of infrared light. The wavelength band of this fifth example includes a wavelength band of 790 to 820 nanometers, or a wavelength band of 905 to 970 nanometers. The light of this fifth example has a peak wavelength in the wavelength band of 790 to 820 nanometers, or a wavelength band of 905 to 970 nanometers.

[0350] [Example of generating special light images]

[0351] The processor device 200 can also generate a special light image containing information about a specific wavelength band based on a normal light image captured using white light. Furthermore, the term "generating" here includes acquiring. In this case, the processor device 200 functions as a special light image acquisition unit. Furthermore, the processor device 200 obtains a signal in the specific wavelength band by performing calculations based on the color information of red, green, and blue, or cyan, magenta, and yellow, contained in the normal light image.

[0352] In addition, cyan, magenta, and yellow are sometimes expressed as CMY using the initials of their respective English symbols, ie, Cyan, Magenta, and Yellow.

[0353] [Example of generating a feature image]

[0354] As a medical image, a feature value image can be generated by calculation based on at least one of an ordinary light image obtained by irradiating light of a white band, or irradiating light of multiple wavelength bands as white band light, and a special light image obtained by irradiating light of a specific wavelength band.

[0355] [Application Example of a Program for Making a Computer Function as an Image Processing Device]

[0356] The image processing device and the image processing method described above may be configured as a program that realizes the functions corresponding to the respective units in the image processing device or the respective steps in the image processing method using a computer.

[0357] For example, a program may be constructed to cause a computer to realize an image acquisition function, a display signal transmission function, a region of interest detection function, an unobserved image storage function, and a user input signal acquisition function.

[0358] The image acquisition function corresponds to the image acquisition unit 40 and the endoscopic image acquisition step S10. The display signal transmission function corresponds to the display control unit 44 and the display signal transmission step S26. The region of interest detection function corresponds to the region of interest detection unit 50 and the region of interest detection step S12.

[0359] The unobserved image storage function corresponds to the unobserved image storage unit 71 and the unobserved condition determination step S18. The user input signal acquisition function corresponds to the user input signal acquisition unit 42 and the user input signal acquisition determination step S20.

[0360] The program may include a first image processing function corresponding to the first image processing unit 52 and the first image processing step S16. The program may include a first image processing result storage function corresponding to the first image processing result storage unit 72 and the first image processing result storage step S17.

[0361] The program may include a second image processing function corresponding to the second image processing unit 54 and the second image processing step S22. The program may include a second image processing result storage function corresponding to the second image processing result storage unit 74 and the second image processing result storage step S24. The program may include a third image processing function corresponding to the third image processing unit 56 and the third image processing step S42.

[0362] The program for causing a computer to realize the above-described image processing function may be stored in a computer-readable information storage medium, which is a tangible, non-transitory information storage medium, and provided via the information storage medium.

[0363] In addition, instead of storing and providing the program in a non-transitory information storage medium, a program signal may be provided via a communication network.

[0364] [Regarding Combinations of Embodiments and Modifications]

[0365] The components described in the above embodiment and the components described in the modified examples may be used in combination as appropriate, and some of the components may be replaced.

[0366] The embodiments of the present invention described above can be appropriately changed, added, or deleted without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and those skilled in the art can make various modifications within the technical concept of the present invention.

[0367] Explanation of symbols

[0368] 10 Endoscopic system

[0369] 14 Image processing device

[0370] 14A Image processing device

[0371] 14B Image processing device

[0372] 38 Endoscopic images

[0373] 38A Moving Image

[0374] 38B frame image

[0375] 39 still images

[0376] 39A Unobserved Image

[0377] 40 Image acquisition unit

[0378] 42 User input signal acquisition unit

[0379] 44 Display control unit

[0380] 46 Ministry of Public Information

[0381] 50 Focus Area Detection Department

[0382] 51 Unobserved Condition Judgment Unit

[0383] 52 First image processing unit

[0384] 54 Second image processing unit

[0385] 56 Third Image Processing Unit

[0386] 70 Endoscope image storage unit

[0387] 71 Unobserved image storage unit

[0388] 71A Observed image storage unit

[0389] 72 First image processing result storage unit

[0390] 74 Second image processing result storage unit

[0391] 76 Third image processing result storage unit

[0392] 100 Endoscope body

[0393] 102 Hand operation unit

[0394] 104 Insertion

[0395] 106 General Cable

[0396] 108 optical connector

[0397] 112 Soft Department

[0398] 114 Bend

[0399] 116 top hard part

[0400] 116A top side end face

[0401] 123 Lighting Department

[0402] 123A Lighting Lens

[0403] 123B Lighting Lens

[0404] 126 Clamping mouth

[0405] 130 Camera Department

[0406] 132 Photographic lens

[0407] 134 Camera Components

[0408] 136 drive circuit

[0409] 138 Analog Front End

[0410] 141 Gas and water supply buttons

[0411] 142 Attraction Button

[0412] 143 function buttons

[0413] 144 Camera button

[0414] 170 Light Guide

[0415] 200 processor units

[0416] 202 Image Input Controller

[0417] 204 Image Processing Department

[0418] 205 Communication Control Department

[0419] 206 Video Output Unit

[0420] 207 Storage Department

[0421] 208 Operation Department

[0422] 209 Sound Processing Department

[0423] 209A Speaker

[0424] 210 CPU

[0425] 211 ROM

[0426] 212 RAM

[0427] 300 Light Source Device

[0428] 310 light source

[0429] 310B blue light source

[0430] 310G green light source

[0431] 310R red light source

[0432] 330 aperture

[0433] 340 Condenser Lens

[0434] 350 Light source control unit

[0435] 400 monitors

[0436] 402 screen

[0437] 404 Observe the image display area

[0438] 406 Image display area not observed

[0439] 500 First Image

[0440] 510 Focus Area

[0441] 510A Area of ​​Concern

[0442] 510B Area of ​​Concern

[0443] 510C Focus Areas

[0444] 512 bounding boxes

[0445] 512A Bounding Box

[0446] 512B bounding box

[0447] 512C Bounding Box

[0448] 520 Second Image

[0449] 530 Second Image

[0450] 532 bounding boxes

[0451] 540 Second Image

[0452] 542 Second Image

[0453] 560 Third Image

[0454] S10 to S44: Steps of the Image Processing Method

Claims

1. A medical image processing device comprising: an image acquisition unit that acquires an observation image of a subject; a region-of-interest detection unit configured to detect a region of interest from a frame image constituting the observation image; a non-observation condition determination unit for determining, with respect to a frame image in which a region of interest is detected, whether a non-observation condition indicating that the region of interest that the user has overlooked is included is satisfied; an unobserved image storage unit that stores unobserved images that satisfy the unobserved condition; as well as a display signal transmitting unit that transmits a first display signal representing the observed image and a second display signal representing the unobserved image to a display device; The attention area detection unit determines whether the attention area of ​​the unobserved image and the attention area of ​​the observed image are identical. The display signal transmitting unit stops transmitting the second display signal when a determination result is obtained that the region of interest in the unobserved image is identical to the region of interest in the observed image.

2. The medical image processing apparatus according to claim 1, wherein: The attention region detection unit determines identity between the attention region of the unobserved image and the attention region of the observed image based on a comparison result of a feature amount of the attention region of the unobserved image and a feature amount of the attention region of the observed image.

3. The medical image processing apparatus according to claim 1, wherein: The non-observation condition determination unit determines that the non-observation condition is satisfied when the number of frame images including the same region of interest within a predetermined period is equal to or smaller than a predetermined number.

4. The medical image processing apparatus according to claim 1 or 3, wherein: The non-observation condition determination unit determines that the non-observation condition is satisfied when the amount of change between frame images is equal to or greater than a predetermined threshold value.

5. The medical image processing apparatus according to claim 1 or 3, wherein: The non-observation condition determination unit determines that the non-observation condition is satisfied when the same region of interest stays in any region within the screen within a predetermined period. The medical image processing apparatus according to claim 1 , wherein: The medical image processing device comprises: a user input signal acquiring unit for acquiring a user input signal transmitted in response to a user operation, The display signal transmitting unit transmits a second display signal indicating the unobserved image to the display device when the user input signal acquiring unit acquires the user input signal indicating the display of the unobserved region of interest.

7. The medical image processing apparatus according to claim 6, wherein: The medical image processing device comprises: a first image processing unit that performs first image processing on the attention area detected by the attention area detection unit to generate a first image; The display signal transmitting unit transmits a display signal representing the first image to the display device as the first display signal.

8. The medical image processing apparatus according to claim 7, wherein: The medical image processing device comprises: a second image processing unit that performs second image processing on the unobserved image to generate a second image; The display signal transmitting section transmits a display signal representing the second image to the display device as the second display signal.

9. The medical image processing apparatus according to claim 8, wherein: The first image processing unit performs emphasis processing on the attention area of ​​the observation image. The second image processing unit performs, on the unobserved image, an emphasis process that is performed to enhance the emphasis level more than the emphasis process performed on the attention area of ​​the observed image.

10. The medical image processing apparatus according to claim 8, wherein: The second image processing unit performs the second image processing on each of the plurality of unobserved images. The display signal transmitting section transmits a display signal corresponding to each of the plurality of second images to the display device as the second display signal.

11. The medical image processing apparatus according to claim 10, wherein: The medical image processing device comprises: The selection unit selects one or more of the unobserved images from the plurality of unobserved images displayed on the display device, or selects one or more of the second images from the plurality of second images displayed on the display device.

12. The medical image processing apparatus according to claim 8, wherein: The medical image processing device comprises: A third image processing unit performs third image processing on the unobserved image displayed on the display device or the processing result of the second image processing.

13. The medical image processing apparatus according to claim 12, wherein: The medical image processing device comprises: A third image processing result storage unit stores a processing result of the third image processing.

14. The medical image processing apparatus according to claim 12, wherein: The third image processing unit edits the unobserved image when the user input signal acquisition unit acquires the user input signal indicating editing of the unobserved image, or edits the second image when the user input signal acquisition unit acquires the user input signal indicating editing of the second image.

15. The medical image processing apparatus according to claim 12, wherein: The third image processing unit transmits the unobserved image to the external device when the user input signal acquisition unit acquires the user input signal indicating that the unobserved image is to be transmitted to the external device, or transmits the second image to the external device when the user input signal acquisition unit acquires the user input signal indicating that the second image is to be transmitted to the external device.

16. The medical image processing apparatus according to claim 12, wherein: The third image processing unit rearranges the plurality of unobserved images when the user input signal acquisition unit acquires a user input signal indicating rearrangement of the plurality of unobserved images, or rearranges the plurality of second images when the user input signal acquisition unit acquires a user input signal indicating rearrangement of the plurality of second images.

17. The medical image processing apparatus according to claim 12, wherein: The third image processing unit deletes the unobserved image when the user input signal acquisition unit acquires the user input signal indicating deletion of the unobserved image, or deletes the second image when the user input signal acquisition unit acquires the user input signal indicating deletion of the second image.

18. The medical image processing apparatus according to claim 1, wherein: The display signal transmitting section stops transmitting the second display signal after a predetermined period of time has elapsed from a timing of transmitting the second display signal.

19. The medical image processing apparatus according to claim 1 or 3, wherein: The medical image processing device comprises: A user operation unit operated by a user, The user input signal acquisition unit acquires a user input signal transmitted when the user operates the user operation unit.

20. The medical image processing apparatus according to claim 1 or 3, wherein: The medical image processing device comprises: a voice acquisition unit that acquires a user's voice, The user input signal acquisition unit acquires a user input signal representing the user's voice acquired using the voice acquisition unit.

21. The medical image processing apparatus according to claim 1 or 3, wherein: The medical image processing device comprises: An observed image storage unit stores an observed image indicating that the attention area has been observed, among the frame images in which the attention area has been detected by the attention area detection unit.

22. The medical image processing apparatus according to claim 1 or 3, wherein: The medical image processing device comprises: A notification unit notifies detection of the region of interest in the observation image.

23. An endoscope system comprising: Endoscope; an endoscope control device that controls the endoscope; and A medical image processing device that processes an endoscopic image acquired using the endoscope, in, The medical image processing device comprises: an image acquisition unit that acquires an observation image of a subject; a region-of-interest detection unit configured to detect a region of interest from a frame image constituting the observation image; a non-observation condition determination unit for determining, with respect to a frame image in which a region of interest is detected, whether a non-observation condition indicating that the region of interest that the user has overlooked is included is satisfied; an unobserved image storage unit that stores unobserved images that satisfy the unobserved condition; as well as a display signal transmitting unit that transmits a first display signal representing the observed image and a second display signal representing the unobserved image to a display device; The attention area detection unit determines whether the attention area of ​​the unobserved image and the attention area of ​​the observed image are identical. The display signal transmitting unit stops transmitting the second display signal when a determination result is obtained that the region of interest in the unobserved image is identical to the region of interest in the observed image.

24. A medical image processing method, comprising: An image acquisition step of acquiring an observation image of the subject; a region of interest detection step of detecting a region of interest from a frame image constituting the observation image; a non-observation condition determination step of determining whether the frame image in which the attention area is detected satisfies a non-observation condition indicating that the attention area that the user has overlooked is included, an unobserved image storage step of storing unobserved images that meet the unobserved condition; as well as a display signal sending step of sending a first display signal representing the observed image and a second display signal representing the unobserved image to a display device; The attention region detection step determines whether the attention region of the unobserved image is identical to the attention region of the observed image. The display signal transmission step stops transmission of the second display signal when a determination result is obtained that the region of interest in the unobserved image is identical to the region of interest in the observed image.

25. A recording medium which is a non-transitory computer-readable recording medium and records a computer command, wherein: The computer command causes the computer to execute the medical image processing method according to claim 24 when the computer reads the instructions stored in the recording medium.

Images