Image processing device for endoscope and method for operating image processing device for endoscope

Abstract

An endoscope image processing device (3) is provided with: a first image creation unit (3b1) that creates a first image on the basis of a first signal acquired by a first signal acquisition unit (3a1); a second image creation unit (3b2) that creates a second image on the basis of the second signal acquired by the second signal acquisition unit (3a2); a visibility determination unit (3d) that determines whether the visibility of at least a portion of the second image is greater than or equal to a second threshold value or less than the second threshold value; a notification information creation unit (3e) that creates different notification information on the basis of the visibility determination result; and an output unit (3h) that outputs an image based on at least one of the first image and the second image to a display (4), and outputs notification information to a notification device.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for an endoscope that performs processing according to the visibility of at least a portion of an image and an operating method of the image processing apparatus for an endoscope. Background Art

[0002] Traditionally, endoscopes have acquired ordinary light images by illuminating the subject with ordinary light, such as white light. Furthermore, endoscopes sometimes acquire special light images by illuminating the subject with special light, whose spectral distribution differs from that of white light. Furthermore, special light images are not limited to acquisition through irradiation with special light; they can also be acquired by performing image processing different from that used for ordinary light images on the signals acquired through irradiation with ordinary light.

[0003] For example, Japanese Patent Application Laid-Open No. 2022-63129 discloses a technology for displaying a normal light image and a special light image on a single display included in an endoscope system.

[0004] Special light images are sometimes effective for detecting candidate lesion areas (at least a portion of an image) that are candidates for lesion areas. Furthermore, AI (artificial intelligence) is sometimes used in detecting candidate lesion areas. Because ordinary light images are suitable for comprehensively observing the entire image, many users display ordinary light images on the main monitor before operating an endoscope and only check the special light image when special observation is required.

[0005] When shifting the line of sight to the special light image or switching to the special light image, if the user's visibility of the candidate lesion area in the special light image does not improve compared to the normal light image, this wastes time and prolongs the examination. To reduce the burden on the patient and improve the efficiency of the examination, it is preferable to perform an endoscopic examination in the shortest possible time.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing apparatus for endoscope and an operating method of the image processing apparatus for endoscope that can generate notification information that allows a user to easily visually confirm at least a portion of an image. Summary of the Invention

[0007] An image processing device for an endoscope according to one embodiment of the present invention comprises: a first signal acquisition unit that acquires a first signal related to an image under a first condition; a second signal acquisition unit that acquires a second signal related to an image under a second condition, the second condition being different from the first condition; a first image production unit that produces a first image based on the first signal; a second image production unit that produces a second image based on the second signal; a visibility judgment unit that determines whether the visibility of at least a portion of the second image is above a second threshold value or below the second threshold value; a notification information production unit that produces different notification information between a case where the visibility is above the second threshold value and a case where the visibility is below the second threshold value; and an output unit that outputs an image based on at least one of the first image and the second image to a display, and outputs the notification information to a notification device.

[0008] An image processing device for an endoscope in one embodiment of the present invention includes a processor that performs the following processing: acquiring a first signal related to an image under a first condition; acquiring a second signal related to an image under a second condition, the second condition being different from the first condition; producing a first image based on the first signal; producing a second image based on the second signal; determining whether the visibility of at least a portion of the second image is above a second threshold or below the second threshold; producing different notification information between a case where the visibility is above the second threshold and a case where the visibility is below the second threshold; and outputting an image based on at least one of the first image and the second image to a display, and outputting the notification information to a notification device.

[0009] In an operating method of an image processing device for an endoscope according to one embodiment of the present invention, a processor of the image processing device for an endoscope performs the following processing: acquiring a first signal related to an image of a first condition; acquiring a second signal related to an image of a second condition, the second condition being different from the first condition; producing a first image based on the first signal; producing a second image based on the second signal; determining whether the visibility of at least a portion of the second image is above a second threshold or below the second threshold; producing different notification information between a case where the visibility is above the second threshold and a case where the visibility is below the second threshold; and outputting an image based on at least one of the first image and the second image to a display, and outputting the notification information to a notification device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 These are diagrams showing the appearance of an endoscope system according to each embodiment of the present invention.

[0011] Figure 2It is a block diagram showing a configuration example of an endoscope image processing device according to each embodiment.

[0012] Figure 3 This is a block diagram showing a configuration example of a functional unit of an endoscope image processing apparatus according to each embodiment.

[0013] Figure 4 : is a diagram showing an example of how the first image and the second image are displayed with respect to the structure of the display in each embodiment.

[0014] Figure 5 This is a flowchart showing a first example of basic processing performed by the image processing apparatus for an endoscope in each embodiment.

[0015] Figure 6 This is a flowchart showing a second example of basic processing performed by the image processing apparatus for endoscope in each embodiment.

[0016] Figure 7 This is a flowchart showing a third example of basic processing performed by the image processing apparatus for an endoscope in each embodiment.

[0017] Figure 8 This is a flowchart showing a fourth example of basic processing performed by the endoscope image processing apparatus in each embodiment.

[0018] Figure 9 This is a table showing several examples of different notification information according to the second visibility of the second image in each embodiment.

[0019] Figure 10 This is a flowchart showing specific processing of the endoscope image processing apparatus in the first embodiment of the present invention.

[0020] Figure 11 This is a flowchart showing specific processing of the endoscope image processing apparatus in the second embodiment of the present invention.

[0021] Figure 12 This is a flowchart showing specific processing of the endoscope image processing apparatus in the third embodiment of the present invention.

[0022] Figure 13 This is a flowchart showing specific processing of the endoscope image processing apparatus in the fourth embodiment of the present invention.

[0023] Figure 14 1 is a diagram showing an example of how image display and notification are performed on the first display and the second display according to visibility in each embodiment.

[0024] Figure 15This is a diagram showing an example of how image display and notification are performed on one display according to visibility in each embodiment.

[0025] Figure 16 This is a graph showing an example of changes in the display on the monitor when an endoscopic examination is being performed in each embodiment.

[0026] Figure 17 This is a graph showing another example of changes in the display on the monitor when an endoscopic examination is being performed in each embodiment.

[0027] Figure 18 Graphs showing display examples of images according to visibility when one image is displayed on a display in each embodiment.

[0028] Figure 19 Graphs showing display examples of images according to visibility when two images are displayed on the main screen and the sub screen, respectively, in each embodiment. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described below.

[0030] In the description of the drawings, identical or corresponding elements are appropriately denoted by the same reference numerals. It should also be noted that the drawings are schematic, and the length relationships, length ratios, and number of elements within a single drawing may sometimes differ from the actual situation for the sake of simplicity. Furthermore, multiple drawings may sometimes include portions with different length relationships or ratios.

[0031] Figures 1 to 19 Various embodiments of the present invention are shown. Figure 1 These are diagrams showing the appearance of the endoscope system 1 according to each embodiment.

[0032] The endoscope system 1 includes an endoscope 2 , an endoscope image processing device 3 , and a display 4 .

[0033] The endoscope 2 includes an insertion portion 5 , an operation portion 6 , and a universal cable 7 .

[0034] The insertion portion 5 is an elongated portion inserted into the subject. As an example, the subject inserted by the insertion portion 5 is assumed to be a human body, but is not limited to a human body and may be a living thing such as an animal, or an inanimate object such as a machine or a building.

[0035] The insertion portion 5 includes a distal end structure portion 5 a , a bending portion 5 b , and a flexible tube portion 5 c in this order from the distal end side to the proximal end side.

[0036] The endoscope 2 is configured as an electronic endoscope, with an imaging system installed within the distal end structure 5a. The imaging system includes an objective lens for forming an optical image of the subject, and an imaging element for photoelectrically converting the optical image formed by the objective lens into an electrical signal. The imaging element generates an image signal in units of frames and transmits this image signal to the endoscope image processing device 3.

[0037] Furthermore, the imaging element is not limited to being provided in the distal end structural portion 5a of the insertion portion 5. For example, a structure may be adopted in which a relay optical system is provided in the insertion portion 5 and the operating portion 6, and a camera head is mounted on the operating portion 6. In the case of adopting this structure, the optical image formed by the objective lens is transmitted through the relay optical system, and the optical image is captured by the imaging element in the camera head.

[0038] The bending portion 5b is a portion that can bend, for example, in two directions, vertically or horizontally. It is located at the base end of the distal end structure 5a. When the bending portion 5b bends, the orientation of the distal end structure 5a changes, and the direction of illumination and the imaging system's observation also change. Furthermore, the bending portion 5b may be bent to improve the insertability of the insertion portion 5 within the subject.

[0039] The flexible tubular portion 5c is a flexible tubular portion. The flexible tubular portion 5c is disposed on the proximal end side of the curved portion 5b. Here, the example of the endoscope 2 being a soft endoscope having the flexible tubular portion 5c is given. However, the endoscope 2 may also be a rigid endoscope in which the portion corresponding to the flexible tubular portion 5c is rigid. Furthermore, the endoscope 2 may be any of the following: an entirely disposable endoscope, an entirely reprocessed endoscope for reuse, and an endoscope with a portion being disposable.

[0040] The operation portion 6 is disposed on the proximal end side of the insertion portion 5. The operation portion 6 includes a grip portion 6a, a bending operation knob 6b, an operation button 6c, and a treatment instrument insertion port 6d.

[0041] The grip portion 6 a is a portion for the user to grip the endoscope 2 with the palm of his or her hand.

[0042] The bending operation knob 6b is an operating device for bending the bending portion 5b. The bending operation knob 6b is operated using, for example, the thumb of the hand holding the grip 6a. The bending operation knob 6b is connected to the bending portion 5b via a bending wire. Operating the bending operation knob 6b pulls the bending wire, causing the bending portion 5b to bend.

[0043] The operation button 6c includes a plurality of buttons for operating the endoscope 2. Some examples of the operation button 6c include an air / water supply button, a suction button, and buttons related to imaging.

[0044] The treatment instrument insertion port 6d is an opening on the proximal side of the treatment instrument channel provided within the insertion portion 5 and the operation portion 6. When a treatment instrument is inserted through the treatment instrument insertion port 6d, the distal end of the treatment instrument protrudes from the distal end opening of the treatment instrument channel provided in the distal end structure 5a. In this state, various treatments can be performed on the subject using the treatment instrument.

[0045] The universal cable 7 extends from a side surface of the operation portion 6 , for example, on the proximal side, and is connected to the endoscope image processing device 3 .

[0046] The endoscope image processing device 3 receives image signals from the imaging element in frames. The endoscope image processing device 3 processes the acquired image signals and outputs the processed image signals to the display 4. The endoscope image processing device 3 also serves as an endoscope control device for controlling the endoscope 2.

[0047] The endoscopic image processing device 3 may also serve as an illumination device that emits illumination light. Furthermore, the illumination device may be provided separately from the endoscopic image processing device 3. The illumination device can emit ordinary light, such as white light, as well as special light having a spectral distribution different from that of ordinary light. As described later, the endoscopic image processing device 3 may be equipped with computer-aided detection (CADe) or computer-aided diagnosis (CADx) functionality. CADe or CADx may also be incorporated into another processor capable of communicating with the endoscopic image processing device 3.

[0048] The display 4 is a display device (display unit) that receives image signals and displays endoscopic images. Furthermore, the display 4 need not be a component unique to the endoscope system 1. For example, the display 4, which is separate from the endoscope system 1, may be connected to the endoscope image processing device 3 for use. Furthermore, as described later, the display 4 is not limited to a single unit; multiple units may be provided.

[0049] The endoscope system 1 may include a sound generating device such as a speaker or a buzzer for generating sound or voice, either integrated with the endoscope image processing device 3 or the display 4, or separately from the endoscope image processing device 3 or the display 4. Furthermore, the endoscope system 1 may include a vibration device for generating vibration, either integrated with the display 4, or separately from the display 4.

[0050] Display 4 is an example of a notification device (notification unit) that outputs notification information as visual information. A sound-generating device is an example of a notification device that outputs notification information as sound or voice information. A vibration device is an example of a notification device that outputs notification information as vibration information. Specifically, examples of notification information include one or more of visual information, sound information, voice information, and vibration information.

[0051] Figure 2 It is a block diagram showing a configuration example of the endoscope image processing device 3 according to each embodiment.

[0052] Each functional unit of the endoscope image processing device 3 (see Figure 3 ) can also be composed of electronic circuits. In addition, all or part of the functional parts of the endoscope image processing device 3 can also be composed of Figure 2 The image processing device 3 for endoscopes is composed of a processor 30a and a memory 30b. The processor 30a is composed of an ASIC (Application Specific Integrated Circuit) including a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or the like. The memory 30b stores computer programs for implementing the various functional units. The various functional units within the image processing device 3 for endoscopes are implemented by the processor 30a reading and executing the computer programs stored in the memory 30b.

[0053] Figure 3 1 is a block diagram showing a configuration example of the functional unit of the endoscope image processing device 3 according to each embodiment. Figure 3 The functional units involved in each embodiment are listed in the figure, but some functional units may be omitted or other functional units may be added as needed. In addition, each structure does not need to be installed in a single device, and the structure installed in other devices may be connected via a communication device to form the endoscope image processing device 3.

[0054] The endoscope image processing device 3 includes a signal acquisition unit 3a, an image generation unit 3b, a target region detection unit 3c, a visibility determination unit 3d, a notification information generation unit 3e, an image synthesis unit 3f, an output switching unit 3g, and an output unit 3h.

[0055] The signal acquisition unit 3 a includes a first signal acquisition unit 3 a 1 , a second signal acquisition unit 3 a 2 , and a detection signal acquisition unit 3 a 3 .

[0056] The first signal acquisition unit 3a1 acquires a first signal related to an image under a first condition. An example of a first condition is a condition in which the first signal is acquired by irradiating the subject with ordinary light (such as white light). The first condition may also include various setting conditions associated with imaging (such as the luminous intensity of ordinary light, exposure time, and signal amplification factor). The first condition may also include a condition for performing first image processing on the first signal in order to generate an ordinary light image.

[0057] The second signal acquisition unit 3a2 acquires a second signal associated with an image of a second condition that is different from the first condition. An example of the second condition is a condition in which the second signal is acquired by irradiating the subject with special light. As the second condition, various setting conditions associated with photography (luminous intensity of special light, exposure time, signal amplification rate, presence or absence of optical filters, type, etc.) may also be included. As the second condition, a condition in which a second image processing (image processing different from the first image processing) is performed on the second signal in order to generate a special light image may also be included. In addition, the second condition is not limited to the condition in which special light is irradiated. For example, the second condition may also be a condition in which a special image processing (image processing different from the first image processing and the second image processing) is performed on the first signal obtained by irradiating ordinary light to obtain an image corresponding to the special light image.

[0058] The detection signal acquisition unit 3a3 acquires a detection signal. The detection signal is used by the AI ​​(Artificial Intelligence) (described later) of the target region detection unit 3c to detect target regions such as candidate lesion regions. The detection signal acquisition unit 3a3 and the second signal acquisition unit 3a2 may be integrated. In this case, the second signal also serves as the detection signal. Alternatively, the detection signal acquisition unit 3a3 and the first signal acquisition unit 3a1 may be integrated. In this case, the first signal also serves as the detection signal.

[0059] The image generator 3b includes a first image generator 3b1 and a second image generator 3b2. The first image generator 3b1 performs first image processing on the first signal to generate a first image. The first image (normal light image) generated based on the first signal obtained by irradiating the subject with normal light can be used as a normal observation image.

[0060] The second image generator 3b2 performs second image processing on the second signal to generate a second image. The second image (special light image) generated based on the second signal acquired by irradiating the subject with special light can be used as an identification image for detecting the target area.

[0061] The target region detection unit 3c detects the target region (at least a portion of the image) based on the detection signal. An example of a target region is a lesion candidate region that is a candidate for a lesion region (in addition, when the lesion candidate region is determined to be a lesion region, it can also be a lesion region). However, the target region is not limited to the lesion candidate region, and can also be a normal region. For example, in cases where obesity treatment, nasal mucosal cauterization, endoscopic diverticulotomy, etc. are not accompanied by detection of a lesion candidate region, a normal region can also be used as a target region. As for the target region that is a normal region, for example, fat, blood vessels, bleeding points, nerves, ureters, or urethra are cited.

[0062] When the second signal also serves as the detection signal, the target area detection unit 3c detects the target area from the second image generated based on the second signal. Alternatively, the target area detected based on the detection signal or the second image may be used as the target area for the first image. Alternatively, the target area detection unit 3c may further detect the target area from the first image.

[0063] For example, AI is used to perform target area detection by the target area detection unit 3c. In this case, the target area detection unit 3c has AI for target area detection. When AI detects the target area of ​​the first image, AI may further detect a first score representing the accuracy of the target area of ​​the first image. When AI detects the target area of ​​the second image, AI may further detect a second score representing the accuracy of the target area of ​​the second image. The target area detection unit 3c only needs to include one or more detectors (identifiers). In the case of one, one detector detects the target area of ​​the first image and the target area of ​​the second image. In the case of including two or more detectors, it may also be configured so that the first detector detects the target area of ​​the first image and the second detector detects the target area of ​​the second image.

[0064] The visibility determination unit 3d determines whether the visibility (second visibility) of at least a portion of the second image (specifically, the target area in the second image) is greater than or less than a second threshold (a certain threshold). Alternatively, the visibility determination unit 3d may determine whether the first visibility of at least a portion of the first image (the target area) is greater than or less than the first threshold in addition to the second image. Here, the first threshold is a threshold for the first visibility of the target area in the first image. The second threshold is a threshold for the second visibility of the target area in the second image.

[0065] The visibility determination unit 3d determines that the first visibility is high when the first visibility is greater than or equal to a first threshold value, and determines that the first visibility is low when the first visibility is less than the first threshold value. The visibility determination unit 3d determines that the second visibility is high when the second visibility is greater than or equal to a second threshold value, and determines that the second visibility is low when the second visibility is less than the second threshold value.

[0066] Therefore, in the following, a statement such as "the first visibility is high" means "the first visibility is above the first threshold value," and a statement such as "the first visibility is low" means "the first visibility is below the first threshold value." Similarly, a statement such as "the second visibility is high" means "the second visibility is above the second threshold value," and a statement such as "the second visibility is low" means "the second visibility is below the second threshold value."

[0067] The visibility determination unit 3d may also include a second AI dedicated to visibility estimation. For example, the second AI estimates a visibility score for the target area. The visibility determination unit 3d compares the estimated visibility score with thresholds (first and second thresholds corresponding to the visibility score) to determine the visibility of the target area.

[0068] The notification information generating unit 3e generates different notification information depending on whether the visibility (second visibility) of at least a portion of the second image (specifically, the target area) is greater than a second threshold (a certain threshold) or less than the second threshold. Notification information is information notified to the user, for example, information urging attention. "Generating different notification information" includes generating notification information and not generating notification information (generating no (zero) notification information).

[0069] The notification information generator 3e may generate notification information based not only on the second visibility but also on the first visibility. For example, if the first visibility is less than a first threshold, the notification information generator 3e may generate different notification information depending on whether the second visibility is greater than or equal to the second threshold, or whether the second visibility is less than the second threshold. Furthermore, the notification information generator 3e may generate an icon indicating the location of the target area.

[0070] When the AI ​​creates a score indicating the accuracy of the target area, or when the second AI creates a visibility score, the notification information creating unit 3 e may create notification information indicating the accuracy score or the visibility score.

[0071] The image synthesis unit 3f synthesizes the first image and the second image to create a composite image. For example, the image synthesis unit 3f creates the composite image when the first visibility of the first image is less than a first threshold value and the second visibility of the second image is greater than a second threshold value. Therefore, if the first visibility is greater than the first threshold value or the second visibility is less than the second threshold value, a composite image may not be created (although a composite image may be created as needed).

[0072] The image synthesis unit 3f may synthesize the first image and the second image at a 1:1 ratio, or may set a synthesis ratio for synthesis. For example, the synthesis ratio may be set based on visibility. Furthermore, the image synthesis unit 3f may synthesize only the target area of ​​the second image with the first image (or may synthesize the target area of ​​the second image after setting a synthesis ratio based on visibility) to create a synthesized image.

[0073] For example, when one display 4 displays a single image, the output switching unit 3g switches between the first and second images and outputs them to the output unit 3h along with notification information. Furthermore, when the image synthesis unit 3f creates a synthesized image, the output switching unit 3g switches between one or more of the first, second, and synthesized images and outputs them to the output unit 3h along with notification information. However, if image switching is not necessary, the output switching unit 3g may not be provided.

[0074] The output unit 3h outputs an image based on at least one of the first image and the second image to the display 4 and outputs notification information to the notification device (notification unit). If the image synthesis unit 3f creates a synthesized image, the output unit 3h may also output the synthesized image to the display 4. Therefore, examples of an image based on at least one of the first image and the second image include the first image itself, the second image itself, a synthesized image, and the like.

[0075] The output unit 3h may also output an icon indicating the position of the target area generated by the notification information generating unit 3e together with the first image (and / or the second image (which may also be a composite image)) to the display 4. In this way, the endoscope system 1 can function as a CADe.

[0076] Here, several examples of visibility determination performed by the visibility determination unit 3d are described. The visibility calculation described below is not limited to a single calculation; multiple calculations can be performed, and a comprehensive visibility can be calculated based on the results. Furthermore, the first and second thresholds are each set to values ​​corresponding to the visibility determination method.

[0077] The visibility determination unit 3d may also determine the first visibility based on, for example, information about the target area in the first image and information about the area outside the target area (surrounding area). Similarly, the visibility determination unit 3d may also determine the second visibility based on, for example, information about the target area in the second image and information about the area outside the target area (surrounding area).

[0078] For example, the visibility determination unit 3d calculates a first color difference between the target area and the area outside the target area in the first image, and then compares the first color difference with a first threshold value as the first visibility to determine the first visibility.

[0079] The visibility determination unit 3d also calculates a second color difference between the target area and an area outside the target area in the second image. The visibility determination unit 3d then compares the second color difference with a second threshold value as the second visibility, thereby determining the second visibility. Here, the color difference is defined as, for example, the Euclidean distance in a color space.

[0080] The visibility determination unit 3d detects blood vessels from the target area of ​​the first image and detects blood vessels from an area outside the target area of ​​the first image. The visibility determination unit 3d calculates a first blood vessel volume difference between the target area and the area outside the target area in the first image, and compares the first blood vessel volume difference as a first visibility with a first threshold.

[0081] The visibility determination unit 3d detects blood vessels from the target area of ​​the second image and detects blood vessels from an area outside the target area of ​​the second image. The visibility determination unit 3d calculates a second blood vessel volume difference between the target area and the area outside the target area in the second image, and compares the second blood vessel volume difference as a second visibility with a second threshold value.

[0082] For example, the visibility determination unit 3 d calculates the difference between the ratio of the area of ​​blood vessels detected in the target region and the ratio of the area of ​​blood vessels detected in the region outside the target region for the first and second images as the first and second blood vessel volume differences.

[0083] Alternatively, instead of using the ratio of blood vessel areas as the blood vessel volume, the average of the total areas of blood vessels contained within a unit area in the target region and the region outside the target region may be used as the blood vessel volume. Alternatively, instead of using the area, the average of the total lengths of blood vessels contained within a unit area may be used as the blood vessel volume.

[0084] When the target region is a lesion region such as a tumor, more blood vessels may be detected in the target region than in areas outside the target region. Therefore, this determination method is effective when detecting a lesion candidate region such as a tumor candidate.

[0085] Furthermore, for example, when the AI ​​detects a target area in a first image and a target area in a second image, as well as a first score and a second score, the visibility determination unit 3d may use the first score as the first visibility and compare it with the first threshold to determine the first visibility, and use the second score as the second visibility and compare it with the second threshold to determine the second visibility. If the target area is, for example, a lesion candidate area, the AI ​​detects the first score and the second score, respectively, as the lesion score.

[0086] For example, the visibility determination unit 3d extracts a first edge from the target area of ​​the first image and calculates a first edge quantity. The visibility determination unit 3d compares the first edge quantity as the first visibility with a first threshold value to determine the first visibility. The visibility determination unit 3d extracts a second edge from the target area of ​​the second image and calculates a second edge quantity. The visibility determination unit 3d compares the second edge quantity as the second visibility with a second threshold value to determine the second visibility. Here, for example, the edge quantity can also be calculated based on the total number of edges detected within the target area. If the edge quantity is greater than the threshold value, the target area can be clearly distinguished from areas outside the target area, and thus it can be determined that visibility is high.

[0087] As described above, when the visibility score is estimated by the second AI, the visibility determination unit 3d may determine that the visibility of the target area is high when the visibility score of the target area is greater than the threshold, and may determine that the visibility of the target area is low when the visibility score is less than the threshold.

[0088] For example, the visibility determination unit 3d may determine visibility based on whether the target area is detected. The visibility determination unit 3d may determine that the first visibility is greater than or equal to a first threshold value when the target area is detected from the first image, and that the first visibility is less than or equal to the first threshold value when the target area is not detected from the first image. The visibility determination unit 3d may determine that the second visibility is greater than or equal to a second threshold value when the target area is detected from the second image, and that the second visibility is less than or equal to the second threshold value when the target area is not detected from the second image.

[0089] For example, the visibility determination unit 3d may also determine visibility based on the degree of consistency in the positions of the target areas. Specifically, when the target area is detected at the same position in both the first image and the second image (when the degree of consistency in the positions of the target areas is high), the visibility determination unit 3d determines that the first visibility of the target area in the first image is high, and the second visibility of the target area in the second image is high. Alternatively, when the target area is detected at the same position in only one of the first and second images, the visibility determination unit 3d determines that the detected target area has high visibility, and the target area where the target area is not detected has low visibility.

[0090] Figure 4 4 is a diagram showing an example of how the first image and the second image are displayed with respect to the structure of the display 4 in each embodiment.

[0091] Figure 4Column A shows an example where display 4 is configured as a single display device. A first image display area 4a1 is provided on a portion of the display screen of display 4, displaying a first image, and a second image display area 4a2 is provided on another portion of the display screen, displaying a second image. In this case, for example, the output unit 3h may simply output the first and second images to display 4 in a parallel arrangement.

[0092] Alternatively, the first image display area 4a1 may be set larger than the second image display area 4a2. In this case, the output unit 3h may simply create a reduced second image and output the first image and the reduced second image to the display 4 in a parallel arrangement.

[0093] Furthermore, the first image display area 4a1 and the second image display area 4a2 are not limited to being independently provided. For example, the second image display area 4a2 may be arranged within the first image display area 4a1. In this case, the output unit 3h only needs to create a reduced second image and output it to the display 4 in a manner that superimposes the reduced second image on the portion of the first image that does not overlap with the target area.

[0094] Figure 4 Column B of FIG. 4 shows an example in which the display 4 is configured to include two display devices, a first display 4A and a second display 4B. In this case, the first image is displayed on the first display 4A, and the second image is displayed on the second display 4B.

[0095] Furthermore, the screen sizes of the first display 4A and the second display 4B may differ. If the screen sizes are different, the screen size of the first display 4A is preferably larger than that of the second display 4B. This is because the first image, which is a normal light image, is the primary image primarily viewed by the user. In this case, the first display 4A functions as the primary display, and the second display 4B functions as the secondary display.

[0096] In the following, the first image display area 4a1 or the screen of the first display 4A is appropriately referred to as a main screen, and the second image display area 4a2 or the screen of the second display 4B is appropriately referred to as a sub screen.

[0097] Figure 5 This is a flowchart showing a first example of basic processing performed by the endoscope image processing device 3 in each embodiment.

[0098] When the process starts, the first signal acquisition unit 3a1 acquires a first signal (step S1), and the second signal acquisition unit 3a2 acquires a second signal (step S2).

[0099] The first image creating unit 3b1 creates a first image based on the first signal acquired by the first signal acquiring unit 3a1 (step S3).

[0100] The second image creating unit 3b2 creates a second image based on the second signal acquired by the second signal acquiring unit 3a2 (step S4).

[0101] The visibility determination unit 3 d determines whether the second visibility of the target area of ​​the second image is equal to or greater than a second threshold value or is lower than the second threshold value (step S5 ).

[0102] The notification information generating unit 3e generates different notification information depending on whether the second visibility of the target area of ​​the second image is greater than the second threshold or less than the second threshold (step S6). Figure 9 To explain.

[0103] The output unit 3h outputs an image based on at least one of the first image and the second image to the display 4, and outputs notification information to the notification device (step S8).

[0104] At this time, when the display 4 is composed of one display device, the output switching unit 3g may switch which of the first image and the second image is to be output to the display 4 by the output unit 3h (step S7).

[0105] After step S8, the process is terminated. In addition, when the next frame image is output, the process is performed again. Figure 5 The processing shown (for Figure 6 The same applies to subsequent processing).

[0106] Figure 6 This is a flowchart showing a second example of the basic processing performed by the endoscope image processing device 3 in each embodiment.

[0107] When the process starts, the processes of steps S1 to S4 described above are performed.

[0108] The visibility determination unit 3d determines whether the second visibility of the target area in the second image is greater than or less than the second threshold value. Furthermore, the visibility determination unit 3d determines whether the first visibility of the target area in the first image is greater than or less than the first threshold value (step S5A).

[0109] The notification information generating unit 3e generates different notification information when the first visibility is less than the first threshold and the second visibility is greater than or equal to the second threshold, and when the first visibility is less than the first threshold and the second visibility is less than the second threshold (step S6A).

[0110] Regarding an example of notification information corresponding to the second visibility when the first visibility is less than the first threshold, similarly to the above, reference will be made later. Figure 9In addition, when the first visibility is above the first threshold, the user does not need to shift his or her sight to the second image. Therefore, the notification information generating unit 3e may not generate notification information, or may generate notification information for notifying that the user can continue to observe the first image (see Figure 18 and Figure 19 ).

[0111] After that, step S8 is performed (and step S7 is performed as needed) and the processing is ended.

[0112] Figure 7 This is a flowchart showing a third example of the basic processing performed by the endoscope image processing device 3 in each embodiment.

[0113] When the process starts, the processes of steps S1 to S4 described above are performed.

[0114] Then, the detection signal acquisition unit 3a3 acquires a detection signal (step S11).

[0115] The target area detection unit 3c detects the target area based on the detection signal acquired by the detection signal acquisition unit 3a3, using, for example, an AI (identifier) ​​(step S12). As described above, the detection signal may be the same as or different from the first and second signals. The detection results of the target area detection unit 3c are transmitted to the first image generation unit 3b.

[0116] The visibility determination unit 3d may also receive the detection result from the target area detection unit 3c. In this case, it may determine whether the second visibility of the detected target area in the second image is greater than or less than the second threshold (step S5B).

[0117] The notification information generating unit 3e generates different notification information between the case where the second visibility is equal to or greater than the second threshold value and the case where the second visibility is less than the second threshold value (step S6B) (see Figure 9 In addition, when the notification information generating unit 3e generates an icon indicating the position of the target area, the icon may be generated in a different shape (or display control may be performed) between when the second visibility is greater than or equal to the second threshold and when the second visibility is less than the second threshold.

[0118] In the process of step S8, the output unit 3h outputs the icon indicating the position of the target area created by the notification information creating unit 3e to, for example, the first display 4A also serving as a notification device, and displays it on the first image of the first display 4A.

[0119] In this way, after step S8 (and step S7 as needed), the processing is terminated.

[0120] Figure 8 This is a flowchart showing a fourth example of the basic processing performed by the endoscope image processing device 3 in each embodiment.

[0121] When the process is started, the processes of steps S1 to S4 and S11 to S12 described above are performed.

[0122] The visibility determination unit 3d determines whether the second visibility of the target area detected by the target area detection unit 3c in the second image is greater than or less than the second threshold value. Furthermore, the visibility determination unit 3d determines whether the first visibility of the target area detected by the target area detection unit 3c in the first image is greater than or less than the first threshold value (step S5C).

[0123] The notification information generating unit 3e generates different notification information for the case where the first visibility is less than the first threshold and the second visibility is greater than the second threshold, and for the case where the first visibility is less than the first threshold and the second visibility is less than the second threshold (step S6C) (see Figure 9 wait).

[0124] After that, step S8 is performed (and step S7 is performed as needed) and the processing is ended.

[0125] Figure 9 : is a diagram showing several examples of different notification information according to the second visibility of the second image in each embodiment. Figure 9 An example in which the notification information is visual information is shown in FIG.

[0126] Figure 9 Column 1 shows an example of notification information when the first image and the second image are displayed simultaneously on the main screen and the sub-screen. When the second visibility is greater than the second threshold, an icon guiding the user to the second image is displayed on the first image ( Figure 9 1A column). When the second visibility is less than the second threshold, the icon leading to the second image is not displayed on the first image ( Figure 9 1B of the table).

[0127] Figure 9 Column 2 shows an example of notification information when the first image and the second image are displayed on the main screen and the sub-screen at the same time. When the second visibility is above the second threshold, the icon that does not recommend the user to view the second image is not displayed on the first image ( Figure 9 Column 2A). When the second visibility is less than the second threshold, an icon indicating that the user is not recommended to view the second image is displayed on the first image ( Figure 9 2B of the table).

[0128] Figure 9 Column 3 shows an example of notification information when the first and second images are displayed in a single screen. In this case, a screen switching button is displayed on the display 4 along with either the first or second image. The screen switching button is operated using an input device provided in or connected to the endoscopic image processing device 3. Examples of input devices connected to the endoscopic image processing device 3 include a touch panel provided on the display 4, a keyboard, a mouse, and the like.

[0129] Generally speaking, if the screen switching button is operated while the first image is being displayed, the second image is displayed instead of the first image. If the screen switching button is operated while the second image is being displayed, the first image is displayed instead of the second image.

[0130] When the second visibility is equal to or greater than the second threshold, the screen switching button is not grayed out ( Figure 9 3A column). Thus, the image displayed on the display 4 can be switched from the first image to the second image, and can be switched from the second image to the first image.

[0131] When the second visibility is less than the second threshold, the screen switching button when the first image is being displayed is grayed out ( Figure 9 In this case, the image displayed on the display 4 cannot be switched from the first image to the second image. Therefore, the image displayed on the display 4 is only the first image (the second image is not displayed).

[0132] Figure 9 Column 4 shows an example of notification information when the first image and the second image are displayed simultaneously on the main screen and the sub-screen. When the second visibility is above the second threshold, the second image is not filled with black (or gray). Figure 9 4A column). Thus, the user can observe the second image as needed. When the second visibility is less than the second threshold, the second image is filled with black (or gray) ( Figure 9 4B column). Therefore, the second image is automatically excluded from the user's observation object.

[0133] [First embodiment]

[0134] Figure 10 This is a flowchart showing specific processing of the endoscope image processing device 3 in the first embodiment.

[0135] When the process starts, the processes of steps S1 to S4 described above are performed.

[0136] The target region detection unit 3c detects the target region from the first image created by the first image creation unit 3b1, for example, using AI. Furthermore, the target region detection unit 3c detects the target region from the second image created by the second image creation unit 3b2, for example, using AI (step S12D). As described above, the AI ​​may further detect a lesion score (first score, second score) when the target region is a candidate lesion region.

[0137] The visibility determination unit 3d determines whether the first visibility of the target area in the first image is greater than or less than a first threshold. Furthermore, the visibility determination unit 3d determines whether the second visibility of the target area in the second image is greater than or less than a second threshold (step S5D). Here, the visibility determination unit 3d calculates the first and second visibilities based on, for example, at least one of the aforementioned color difference, edge amount, and blood vessel amount.

[0138] Only when the first visibility is smaller than the first threshold value and the second visibility is equal to or greater than the second threshold value, the notification information generating unit 3e generates notification information for urging attention (step S6D).

[0139] In the process of step S8, the output unit 3h outputs an image based on at least one of the first image and the second image to the display 4, and outputs the notification information to the notification device. By outputting the notification information for urging attention, for example, by displaying a mark (see Figure 16 、 Figure 18 、 Figure 19 Notification by sound, voice, characters (see Figure 18 、 Figure 19 At least one of notifications made by at least one of the messages MSG1 to MSG3 shown, and notifications when the lesion score detected by AI exceeds a threshold.

[0140] In this way, after step S8 (and step S7 as needed), the processing is terminated.

[0141] [Second embodiment]

[0142] Figure 11 This is a flowchart showing specific processing of the endoscope image processing device 3 in the second embodiment.

[0143] When the process starts, the processes of steps S1 to S4, S12D, and S5D described above are performed.

[0144] The notification information generating unit 3e generates notification information for urging attention only when the first visibility is smaller than the first threshold value and the second visibility is equal to or greater than the second threshold value (step S6E).

[0145] For example, the notification information generating unit 3e performs at least one of the following operations: generating a mark indicating the position of the target area detected from the first image (position information as one of the notification information) and superimposing the mark on at least the first image; and generating a mark indicating the position of the target area detected from the second image and superimposing the mark on at least the second image. In the former case, the mark may be further superimposed on the second image, and in the latter case, the mark may be further superimposed on the first image.

[0146] Alternatively, the notification information generating unit 3e performs at least one of the following operations: generating a flag (see FIG. 1 ) as notification information indicating that the target area is detected from the first image; Figure 16 、 Figure 18 、 Figure 19 The present invention also provides a method for creating a marker (e.g., a mark FG) and superimposing the mark on at least the first image; and creating a mark indicating that the target area has been detected from the second image and superimposing the mark on at least the second image. In the former case, the mark may be further superimposed on the second image, or in the latter case, the mark may be further superimposed on the first image. Furthermore, either or both of the marker and the mark may be created.

[0147] Furthermore, for example, the notification information generating unit 3e further sets a third threshold value lower than the first threshold value, and sets at least one of the color of the mark and the color of the symbol according to whether the first visibility is less than the third threshold value, or is greater than the third threshold value and less than the first threshold value, or is greater than the first threshold value (for example, refer to Figure 17 ).

[0148] Furthermore, the notification information generating unit 3e may further classify the second visibility into three levels and set at least one of the color of the marker and the color of the symbol based on the classification result. Furthermore, the notification information generating unit 3e is not limited to classifying the first and second visibility into two or three levels and may also classify them into four or more levels.

[0149] When it is determined that the second visibility is equal to or greater than the second threshold, the notification information generating unit 3e may further generate notification information for notifying the second condition.

[0150] In step S8, the output unit 3h outputs the first image, on which at least one of the color-coded mark and logo is superimposed, to the main screen, and outputs the second image (on which at least one of the color-coded mark and logo is superimposed, as needed) to the sub-screen. Alternatively, the output unit 3h may output to the display 4 an image obtained by superimposing the reduced size of the second image on a portion of the first image that does not overlap with the target area.

[0151] In this way, after step S8 (and step S7 as needed), the processing is terminated.

[0152] [Third embodiment]

[0153] Figure 12 This is a flowchart showing specific processing of the endoscope image processing device 3 in the third embodiment.

[0154] When the process starts, the processes of steps S1 to S4 and S12D described above are performed.

[0155] The visibility determination unit 3d determines whether the first visibility of the target area in the first image is greater than or less than a first threshold. Furthermore, the visibility determination unit 3d determines whether the second visibility of the target area in the second image is greater than or less than a second threshold (step S5F). Here, the visibility determination unit 3d calculates the first visibility and the second visibility based on, for example, at least one of the lesion scores (first score, second score) calculated by the AI, the degree of coincidence between the positions of the target area in the first and second images, and the visibility score calculated by the second AI.

[0156] If the first visibility is less than the first threshold and the second visibility is greater than or equal to the second threshold, the image synthesis unit 3f synthesizes the first image and the second image to create a synthesized image (step S9). At this time, the image synthesis unit 3f performs at least one of the following: setting a synthesis ratio for synthesis; synthesizing with a higher synthesis ratio for the first image than for the second image; or synthesizing only the target area of ​​the second image with the first image.

[0157] As a specific example, when the first visibility of the first image is less than a first threshold and the second visibility of the second image is greater than or equal to the second threshold, the image synthesis unit 3f sets the synthesis ratio of the first image to a lower synthesis ratio than when the first visibility of the first image is greater than or equal to the first threshold (since the visibility of the first image is high, the visibility of the second image is irrelevant). Furthermore, generally speaking, when the synthesis ratio of the first image is set to r1 and the synthesis ratio of the second image is set to r2, they are normalized to (r1 + r2) = 1. Therefore, when the synthesis ratio of the first image is set to a low level, the synthesis ratio of the second image is set to a high level.

[0158] In general, the first image (observation image) is more suitable for user observation than the second image (recognition image). Therefore, the image synthesis unit 3f may set the synthesis ratio so that the synthesis ratio r1 of the first image is higher than the synthesis ratio r2 of the second image (that is, r1>r2).

[0159] Furthermore, the image synthesis unit 3 f may synthesize only the target region of the second image with the first image to improve the visibility of the target region while maintaining the visibility of the first image.

[0160] The notification information generating unit 3e generates a mark indicating the position of the target area (for example, referring to Figure 17 The mark PI shown in the column D of FIG) and the mark indicating that the target area is detected (for example, referring to Figure 16 、 Figure 18 、 Figure 19 Furthermore, the notification information generating unit 3e generates notification information indicating that the image has been switched to a highly visible composite image (see Figure 18 2B column of the message MSG2) (step S6F).

[0161] In the processing of step S8, the output unit 3h outputs a composite image superimposed with at least one of a mark and a logo and notification information indicating that the composite image has been switched to, for example, a display 4 only when the first visibility is less than the first threshold and the second visibility is greater than the second threshold.

[0162] In this way, after step S8 (and step S7 as needed), the processing is terminated.

[0163] [Fourth embodiment]

[0164] Figure 13 This is a flowchart showing specific processing of the endoscope image processing device 3 in the fourth embodiment.

[0165] When the process starts, the processes of steps S1 to S4, S12D, S5F, and S9 described above are performed.

[0166] The notification information generator 3e generates at least one of a mark indicating the location of the target area and a mark indicating that the target area has been detected, only when the first visibility is less than the first threshold and the second visibility is greater than the second threshold (step S6G). In this case, if the first visibility is lower than the second visibility and the difference between the first and second visibility is large, the notification information generator 3e may also reduce the transmittance of the mark. Furthermore, the notification information generator 3e may generate a mark urging confirmation of the image displayed on the sub-screen (an image other than the first image, such as a composite image) and display it on the first image.

[0167] In step S8, if the first visibility is less than the first threshold and the second visibility is greater than or equal to the second threshold, the output unit 3h outputs the first image superimposed with at least one of the marker and the logo to the main screen, and outputs the composite image to the sub-screen. Thus, by viewing the logo, the user is urged to shift their gaze from the main screen to the sub-screen.

[0168] In this way, after step S8 (and step S7 as needed), the processing is terminated.

[0169] Figure 14 4A and 4B are diagrams showing examples of how images are displayed and notified on the first display 4A and the second display 4B according to visibility in each embodiment. Figure 14 and the following Figure 15 In the description of FIG, an example of notifying by displaying visual information is described. However, as described above, notification may also be performed by information other than visual information (sound information, voice information, vibration information, etc.).

[0170] In such Figure 14 When the first visibility is greater than or equal to the first threshold value, as shown in column 1 of [ ], the target area, such as the lesion candidate area, can be easily identified by observing the first image. In this case, regardless of whether the second visibility is greater than or less than the second threshold value, the image is displayed and notified as follows.

[0171] The first image is displayed on the first display 4A, and the second image or a composite image is displayed on the second display 4B. Alternatively, notification may not be performed. Alternatively, at least one of the first display 4A and the second display 4B may display only the detection result of the target area (e.g., markers PI and PI' described later).

[0172] In such Figure 14 As shown in the second column of , when the first visibility is less than the first threshold and the second visibility is greater than or equal to the second threshold, the target area is difficult to recognize even when observing the first image, but is easy to recognize when observing the second image.

[0173] In this case, the first image is displayed on the first display 4A, and the second image or the composite image is displayed on the second display 4B. Alternatively, the composite image may be displayed on the first display 4A, and the second image on the second display 4B. In addition, the detection results and visibility information of the target area are displayed.

[0174] In such Figure 14 As shown in the third column of , when the first visibility is smaller than the first threshold value and the second visibility is smaller than the second threshold value, it is difficult to recognize the target area regardless of whether the first image or the second image is observed.

[0175] In this case, the first image is displayed on the first display 4A, and the second image or the composite image is displayed on the second display 4B. In addition, only the detection result of the target area is displayed on at least one of the first display 4A and the second display 4B.

[0176] Figure 15 1 and 2 are diagrams showing examples of how image display and notification are performed on one display 4 according to visibility in each embodiment.

[0177] In such Figure 15 When the first visibility is greater than or equal to the first threshold value, as shown in column 1 of FIG, the first image is displayed on the display 4. Furthermore, in addition to the first image, a second image or a composite image may be displayed side by side on the display 4. Notification may not be performed. Alternatively, only the detection result of the target area may be displayed.

[0178] In such Figure 15 As shown in the second column of FIG, when the first visibility is less than the first threshold and the second visibility is greater than or equal to the second threshold, the first image, the second image, or the synthesized image is displayed on the display 4. Alternatively, the first image and the second image or the synthesized image may be displayed side by side on the display 4. In addition, the detection result and visibility information of the target area are displayed.

[0179] In such Figure 15 As shown in the third column of FIG, when the first visibility is less than the first threshold and the second visibility is less than the second threshold, the first image is displayed on the display 4. In addition to the first image, the second image or the composite image may be displayed in parallel on the display 4. In addition, only the detection result of the target area is displayed.

[0180] In addition, not limited to Figure 14 、 Figure 15 The examples shown are examples, but other image displays and notifications may also be performed.

[0181] Figure 16 4 is a graph showing an example of changes in the display on the display 4 when an endoscopic examination is being performed in each embodiment.

[0182] When an endoscopic examination is started, the endoscopic image processing device 3 acquires the first signal and the second signal from the endoscope 2 and creates a first image P1 and a second image P2. The created first image P1 and second image P2 are as follows: Figure 16 As shown in column A of FIG. 4 , the images are displayed side by side on one display 4 , or are displayed separately on the first display 4A and the second display 4B.

[0183] The target area detection unit 3c detects Figure 16The visibility determination unit 3 d determines the visibility of the detected target area TG.

[0184] When the first visibility of the target area TG of the first image P1 is greater than the first threshold, or when the first visibility is less than the first threshold and the second visibility of the target area TG of the second image P2 is less than the second threshold, as shown in FIG. Figure 16 As shown in column C of the main screen, a marker PI (position information) indicating the position of the target area TG is displayed in the first image P1 of the main screen and the second image P2 of the sub-screen. Here, a square icon surrounding the target area TG is used as the marker PI. However, the marker PI may be displayed in other ways.

[0185] In addition, the shape of the icon representing the marker PI may be changed according to the size of visibility (at least one of the first visibility and the second visibility). Figure 16 In the example shown in the column C of FIG, the marker PI is displayed in both the first image P1 and the second image P2. However, the marker PI may be displayed in only one of the two images.

[0186] In addition, when the first visibility is less than the first threshold and the second visibility is greater than the second threshold, as shown in FIG. Figure 16 As shown in the D column of FIG. 1 , a mark FG and a mark PI indicating the position of the target area TG are displayed on the main screen. Here, the mark FG is used to urge confirmation of the second image displayed on the sub-screen. The mark PI is displayed on the sub-screen.

[0187] The mark FG is used to urge the user to shift their gaze from the main screen displaying the first image to the sub-screen displaying an image other than the first image. Therefore, the mark FG is displayed only on the main screen.

[0188] The mark FG is, for example, a triangular graphic (icon) formed by filling the four corners of the first image P1 with a specific color (yellow, for example). However, the mark FG may be displayed in other ways.

[0189] Figure 17 This is a graph showing another example of changes in the display on the display 4 when an endoscopic examination is being performed in each embodiment.

[0190] When starting endoscopy, if Figure 17 As shown in column A of Figure 16 Similarly to column A of FIG), the first image P1 and the second image P2 are displayed side by side on one display 4, or are displayed on the first display 4A and the second display 4B respectively.

[0191] like Figure 17As shown in column B (with Figure 16 Similarly to column B of FIG), the target region detecting unit 3c detects the target region TG, and the visibility determining unit 3d determines the visibility of the target region TG.

[0192] When the first visibility is equal to or greater than the first threshold, the user can recognize the target area TG by observing the first image P1. Figure 17 As shown in the column C of FIG, no notification information is displayed in either the first image P1 or the second image P2.

[0193] In addition, when the first visibility is less than the first threshold and the second visibility is greater than the second threshold, as shown in FIG. Figure 17 As shown in the D column of FIG. 1 , a mark PI indicating the position of the target area TG is displayed in each of the first image P1 of the main screen and the second image P2 of the sub screen.

[0194] Figure 18 : is a graph showing an example of image display according to visibility when one image is displayed on the display 4 in each embodiment. Figure 18 In columns 1 to 2 of , column A shows an example in which the first visibility is equal to or greater than the first threshold, and column B shows an example in which the first visibility is less than the first threshold and the second visibility is equal to or greater than the second threshold.

[0195] Figure 18 Column 1 shows the first display example. In the case where the first visibility is high, as shown in FIG. Figure 18 As shown in column 1A of FIG. 1A, the first image P1 is displayed on the display 4. A marker FG is superimposed on the first image P1. Since the first visibility is high, the marker FG is used here to notify that the first image P1 can continue to be viewed. The marker FG also serves to notify that the target has been captured.

[0196] The mark FG is a triangular figure formed by filling the four corners of the first image P1 with a specific color (green, for example). Figure 16 The color of the mark FG for urging confirmation of the sub-screen displaying an image other than the first image is as shown in the D column of FIG.

[0197] like Figure 18 As shown in column 1B of FIG, when the first visibility is low and the second visibility is high, the first image P1 is displayed on the display 4. The first image P1 is superimposed with a sign FG and a message MSG1. Since the first visibility is low, the sign FG is used here to urge confirmation of images other than the first image P1.

[0198] The mark FG is, for example, a triangular figure in which the four corners of the first image P1 are filled with a specific color (for example, green) and the boundary line BL of the mark FG is filled with another characteristic color (for example, red). Therefore, the color of the boundary line BL of the mark FG urging confirmation of images other than the first image P1 is different from that of the first image P1. Figure 18 The notification shown in column 1A can continue to be observed in the color of the boundary line BL of the mark FG of the first image P1.

[0199] Message MSG1 is configured as notification information using characters such as "Please switch to special light." When a user sees message MSG1, they switch the illumination light from normal light to special light, and the image displayed on display 4 switches from the first image (normal light image) to the second image (special light image). However, message MSG1 may be displayed as an icon instead of character information, or may be displayed as an icon along with character information.

[0200] Figure 18 Column 2 shows the second display example. In the case where the first visibility is high, as shown in FIG. Figure 18 As shown in column 2A of FIG. 2A, a first image P1 is displayed on the display 4. A marker PI indicating the position of the target area is superimposed on the first image P1. Here, the marker PI is displayed as a square frame. When the first visibility is greater than or equal to the first threshold, the marker PI is, for example, a square frame surrounded by a line of a specific color (for example, green).

[0201] In the case where the first visibility is low and the second visibility is high, as Figure 18 As shown in the column 2B of FIG. 2B, a composite image PS is displayed on the display 4. A marker PI' indicating the position of the target area and a message MSG2 are superimposed on the composite image PS. When the first visibility is less than the first threshold, the marker PI' is a square frame surrounded by lines of another specific color (for example, red). Therefore, the color of the line indicating the marker PI' with low first visibility is different from that of the first threshold. Figure 18 Column 2A shows the color of the line of the marker PI with high first visibility.

[0202] Message MSG2 is configured as notification information using characters such as "Switch to Composite Image." Alternatively, message MSG2 may be displayed as an icon instead of the character information, or may be displayed as an icon along with the character information. By viewing message MSG2, the user can recognize that the image displayed on display 4 has automatically switched from the first image (normal light image) to the highly visible composite image.

[0203] Figure 19is a diagram showing an example of display of images corresponding to visibility when two images are displayed on the main screen and the sub-screen, respectively, in each embodiment. Figure 19 In the example, the screen on the left side that displays an image larger than the main screen is the main screen, and the screen on the right side that displays an image smaller than the main screen is the sub screen.

[0204] exist Figure 19 In columns 1 to 3 of , column A shows an example in which the first visibility is equal to or greater than the first threshold, and column B shows an example in which the first visibility is less than the first threshold and the second visibility is equal to or greater than the second threshold.

[0205] Figure 19 Column 1 shows the first display example. In the case where the first visibility is high, as shown in FIG. Figure 19 As shown in the column 1A of FIG, the first image P1 is displayed on the main screen and the second image P2 is displayed on the sub-screen. A mark PI indicating the position of the target area is superimposed on each of the first image P1 and the second image P2. Figure 18 Similarly, in column 2A, when the first visibility is equal to or greater than the first threshold value, the mark PI is, for example, a square frame surrounded by a line of a specific color (for example, green).

[0206] In the case where the first visibility is low and the second visibility is high, as Figure 19 As shown in column 1B of FIG, the first image P1 is displayed on the main screen and the second image P2 is displayed on the sub-screen. A mark PI' indicating the position of the target area and a message MSG3 are superimposed on the first image P1. A mark PI' indicating the position of the target area is superimposed on the second image P2.

[0207] and Figure 18 Similarly, in the case where the first visibility is less than the first threshold, the mark PI' is a square frame surrounded by a line of another specific color (for example, red). Therefore, the color of the line of the mark PI' indicating low first visibility is different from Figure 19 The color of the line of the marker PI indicating first high visibility is shown in column 1A.

[0208] Message MSG3 is configured as notification information using characters such as "Please confirm the sub-screen." Alternatively, message MSG3 may be displayed as an icon instead of the character information, or may be displayed as an icon along with the character information. By viewing message MSG3, the user can recognize that they are being urged to shift their gaze from the main screen to the sub-screen.

[0209] Figure 19 Column 2 shows the second display example. In the case where the first visibility is high, as shown in FIG. Figure 19As shown in the 2A column of FIG, the first image P1 is displayed on the main screen and the second image P2 is displayed on the sub-screen. Since the first visibility is high, a mark FG is superimposed on the first image P1 to notify that the first image P1 can be continued to be observed. In addition, the mark FG can also serve as a notification that the target has been photographed. Figure 18 Similarly, the mark FG shown in the column 1A is a triangular figure formed by filling the four corners of the first image P1 with a specific color (green, for example).

[0210] A marker PI indicating the position of the target area is superimposed on the second image P2. Figure 19 Similarly to column 1A, when the first visibility is equal to or greater than the first threshold value, the mark PI is, for example, a square frame surrounded by a line of a specific color (for example, green).

[0211] In the case where the first visibility is low and the second visibility is high, as Figure 19 As shown in the column 2B of FIG. 1 , the first image P1 is displayed on the main screen, and the second image P2 is displayed on the sub-screen. A mark FG and a message MSG3 for urging confirmation of the sub-screen are superimposed on the first image P1.

[0212] The mark FG is a triangular figure formed by filling the four corners of the first image P1 with another specific color (for example, yellow). Therefore, the color of the mark FG for prompting confirmation of the sub-screen is different from that of the first image P1. Figure 19 The color of the flag FG is shown in column 2A. Message MSG3 is Figure 19 Column 1B shows the same message MSG3.

[0213] A marker PI' indicating the position of the target area is superimposed on the second image P2. Figure 19 Similarly, in column 1B, when the first visibility is less than the first threshold, the mark PI′ is a square frame surrounded by lines of another specific color (for example, red).

[0214] Figure 19 Column 3 shows the third display example. In the case where the first visibility is high, as shown in FIG. Figure 19 As shown in the 3A column of FIG, the first image P1 is displayed on the main screen and the composite image PS is displayed on the sub-screen. Since the first visibility is high, a mark FG is superimposed on the first image P1 to notify that the first image P1 can be continued to be observed. In addition, the mark FG can also serve as a notification that the target is photographed. Figure 19 The same mark FG is shown in the main screen of column 2A.

[0215] The marker PI indicating the position of the target area is superimposed on the composite image PS. Figure 19The sub-screen of column 2A shows the same green square box labeled PI.

[0216] In the case where the first visibility is low and the second visibility is high, as Figure 19 As shown in the column 3B of FIG, the first image P1 is displayed on the main screen and the composite image PS is displayed on the sub-screen. The first image P1 is superimposed with a mark FG and a message MSG3 for urging confirmation of the sub-screen. The mark FG and the message MSG3 are respectively Figure 19 The logo and message shown in column 2B of the main screen are the same.

[0217] The marker PI' indicating the position of the target area is superimposed on the composite image PS. The marker PI' is Figure 19 The sub-picture of column 2B shows the same red square frame marked PI'.

[0218] Furthermore, the visual information displayed on the display 4 in the notification information includes, as described above, the markers PI, PI', the marker FG, and the messages MSG1 to MSG3. Of these, the markers PI and PI', which indicate the location of the target area, are preferably displayed within the endoscopic image. In contrast, the marker FG and the messages MSG1 to MSG3 only need to be within the display screen of the display 4 and may also be displayed outside the endoscopic image.

[0219] According to these embodiments, the user's line of sight is directed to the second image, or switched to the second image, only when the second image, where the target area is easily discernible to the human eye, is generated. Therefore, the user's line of sight is not pointlessly diverted to the second image (or switched to the second image) where the target area is difficult to discern. This allows for endoscopic examinations to be performed in the shortest possible time, reducing the burden on the patient and improving examination efficiency.

[0220] Furthermore, only when the visibility of the first image is low and the visibility of the second image is high, a notification message urging attention is generated and sent. This allows the user to find a target area that is difficult to visually identify in the first image by observing the highly visible second image, thus reducing the chance of missing the target area.

[0221] In this way, it is possible to create notification information that allows the user to easily visually recognize at least a portion of the image.

[0222] Furthermore, while the above description primarily addresses the case of an endoscopic image processing device, the present invention is not limited thereto. For example, the present invention may also address an endoscope system including an endoscopic image processing device. The present invention may also address an operating method for operating the endoscopic image processing device as described above. The present invention may also address a computer program for causing a computer to perform the same processing as the endoscopic image processing device. The present invention may also address a computer-readable, non-transitory recording medium, etc., that stores the computer program.

[0223] Here, some examples of recording media storing computer program products are portable recording media such as floppy disks, CD-ROMs (Compact Disc Read only memory), DVDs (Digital Versatile Discs), or recording media such as hard disks. The program stored in the recording medium is not limited to the entire computer program, but may also be a part of the computer program. In addition, all or part of the computer program may also be circulated or provided via a communication network. The user installs the computer program from the recording medium into the computer, or downloads the computer program via a communication network and installs it into the computer, so that the computer program is read by the computer and all or part of the actions are executed, thereby enabling the above-mentioned actions of the endoscopic image processing device to be executed.

[0224] Furthermore, the present invention is not limited to the above-mentioned embodiments per se. During the implementation phase, the present invention can be embodied by deforming the constituent elements within the scope of the main purpose of the invention. In addition, various inventive methods can be formed by appropriately combining the multiple constituent elements disclosed in the above-mentioned embodiments. For example, some constituent elements can be deleted from all the constituent elements disclosed in the embodiments. Furthermore, constituent elements of different embodiments can be appropriately combined. Of course, various modifications and applications can be made in this way within the scope of the main purpose of the invention.

Claims

1. An image processing device for endoscope, characterized in that have: a first signal acquiring unit configured to acquire a first signal associated with an image under a first condition; a second signal acquiring unit configured to acquire a second signal associated with an image under a second condition, the second condition being different from the first condition; a first image producing unit for producing a first image according to the first signal; a second image producing unit for producing a second image according to the second signal; a visibility determination unit configured to determine whether visibility of at least a portion of the second image is greater than or equal to a second threshold value or less than the second threshold value; a notification information generating unit configured to generate different notification information between a case where the visibility is greater than or equal to the second threshold and a case where the visibility is less than the second threshold; as well as An output unit outputs an image based on at least one of the first image and the second image to a display, and outputs the notification information to a notification device.

2. The endoscope image processing device according to claim 1, wherein Also features: a detection signal acquisition unit that acquires a detection signal; and a target area detection unit configured to detect a target area based on the detection signal; The visibility determination unit determines whether visibility of the target area in the second image is equal to or greater than the second threshold or less than the second threshold.

3. The endoscope image processing device according to claim 2, wherein: The detection signal acquisition unit and the second signal acquisition unit are integrated, and the second signal also serves as the detection signal. The target region detection unit detects the target region from the second image.

4. The endoscope image processing device according to claim 3, wherein: When the threshold value of the target area of ​​the first image is set to a first threshold value, The visibility determination unit further determines whether the first visibility of the target area in the first image is greater than or less than the first threshold value. When the first visibility is less than the first threshold, the notification information generating unit generates different notification information depending on whether the second visibility of the target area of ​​the second image is greater than or equal to the second threshold or less than the second threshold.

5. The endoscope image processing device according to claim 4, wherein: The visibility determination unit determines whether the second visibility is equal to or greater than the second threshold value or less than the second threshold value based on information about the target area and information about an area outside the target area in the second image.

6. The endoscope image processing device according to claim 5, wherein: The visibility determination unit determines whether the first visibility is equal to or greater than the first threshold value or less than the first threshold value based on information about the target area and information about an area outside the target area in the first image.

7. The endoscope image processing device according to claim 6, wherein: The visibility judgment unit performs the following processing: calculating a first color difference between the target area and an area outside the target area in the first image, and using the first color difference as the first visibility to compare with the first threshold; as well as A second color difference between the target area and an outer area of ​​the target area in the second image is calculated, and the second color difference is used as the second visibility to be compared with the second threshold.

8. The endoscope image processing device according to claim 6, wherein: The visibility judgment unit performs the following processing: calculating a first blood vessel amount difference between the target area and an area outside the target area in the first image, and using the first blood vessel amount difference as the first visibility to compare with the first threshold; as well as A second blood vessel amount difference between the target area and an area outside the target area in the second image is calculated, and the second blood vessel amount difference is used as the second visibility to be compared with the second threshold.

9. The endoscope image processing device according to claim 4, wherein: The target area detection unit performs the following processing: further detecting the target area from the first image; as well as further detecting a first score indicating the accuracy of the target region of the first image and a second score indicating the accuracy of the target region of the second image, The visibility judgment unit performs the following processing: comparing the first score as the first visibility with the first threshold; as well as The second score is used as the second visibility and is compared with the second threshold.

10. The endoscope image processing device according to claim 4, wherein The target area detection unit further detects the target area from the first image. The visibility judgment unit performs the following processing: If the target area is detected from the first image, determining that the first visibility is greater than or equal to the first threshold; if the target area is not detected from the first image, determining that the first visibility is less than the first threshold; as well as If the target area is detected from the second image, it is determined that the second visibility is equal to or greater than the second threshold. If the target area is not detected from the second image, it is determined that the second visibility is less than the second threshold.

11. The endoscope image processing device according to claim 2, wherein: The notification information generating unit generates an icon indicating the position of the target area. The output unit outputs the icon together with the first image to the display.

12. The endoscope image processing device according to claim 2, wherein: The target region is a candidate lesion region.

13. The endoscope image processing device according to claim 2, wherein: further comprising an image synthesis unit for synthesizing the first image and the second image to create a synthesized image, The output unit outputs the synthesized image to the display.

14. The endoscope image processing device according to claim 13, wherein: The image synthesis unit sets a synthesis ratio between the first image and the second image to create the synthesized image.

15. The endoscope image processing device according to claim 13, wherein: The image synthesis unit synthesizes the target area of ​​the second image and the first image to create the synthesized image.

16. The endoscope image processing device according to claim 13, wherein: An output switching unit is further provided for causing the output unit to switch and output one or more of the first image, the second image, and the synthesized image to the display.

17. An image processing device for endoscope, characterized in that: With processor, The processor performs the following processing: acquiring a first signal associated with an image of a first condition; acquiring a second signal associated with the image under a second condition, the second condition being different from the first condition; generating a first image according to the first signal; generating a second image based on the second signal; determining whether visibility of at least a portion of the second image is greater than or less than a second threshold; creating different notification information between a case where the visibility is greater than or equal to the second threshold and a case where the visibility is less than the second threshold; as well as An image based on at least one of the first image and the second image is output to a display, and the notification information is output to a notification device.

18. An operating method of an endoscope image processing device, characterized in that: The processor included in the endoscope image processing device performs the following processing: acquiring a first signal associated with an image of a first condition; acquiring a second signal associated with the image under a second condition, the second condition being different from the first condition; generating a first image according to the first signal; generating a second image based on the second signal; determining whether visibility of at least a portion of the second image is greater than or less than a second threshold; creating different notification information between a case where the visibility is greater than or equal to the second threshold and a case where the visibility is less than the second threshold; as well as An image based on at least one of the first image and the second image is output to a display, and the notification information is output to a notification device.

Images