Image processing apparatus, image processing method, and storage medium

Abstract

The invention provides an image processing apparatus, an image processing method, and a storage medium. An image processing device is applied to an endoscope system and processes an image of a subject captured by an endoscope. A processor of the image processing apparatus executes: a process for generating a three-dimensional model of a subject from a group of images captured by an endoscope; detecting an unobserved area which is not shot by the endoscope according to the three-dimensional model; acquiring an operation signal of the endoscope system; and temporarily stopping the detection of the unobserved area when the operation signal matches the predetermined condition. The predetermined condition includes an operation of the endoscope system that causes a loss of a part of the image group to occur.

Description

Technical Field

[0001] The invention relates to an image processing device, an image processing method and a storage medium. Background Art

[0002] In the past, there is a known technique for generating a three-dimensional model of a subject based on a group of images obtained by an endoscope (for example, see Patent Document 1). According to Patent Document 1, a blank area where a three-dimensional model is not generated is determined to be an unobserved area that is not observed by the endoscope, and the unobserved area is displayed in a visually recognizable manner.

[0003] [Prior art literature]

[0004] [Patent Document]

[0005] [Patent Document 1] Japanese Patent No. 6242543 Summary of the invention

[0006] [Problems to be Solved by the Invention]

[0007] Due to the operation of the endoscope system, a part of the image group may be missing. The three-dimensional model generated based on the partially missing image group lacks accuracy, and as a result, problems may occur in determining the unobserved area.

[0008] [Methods for solving the problem]

[0009] One embodiment of the present invention is an image processing device, which is applied to an endoscope system and processes an image of a subject photographed by an endoscope, wherein the image processing device has a processor that performs the following processing: generating a three-dimensional model of the subject based on a group of images photographed by the endoscope; detecting an unobserved area not photographed by the endoscope based on the three-dimensional model; obtaining an action signal of the endoscope system; and temporarily stopping the detection of the unobserved area when the action signal is consistent with a specified condition, wherein the specified condition includes an action of the endoscope system that causes a portion of the image group to be missing.

[0010] One embodiment of the present invention is an image processing method for processing an image of a subject captured by an endoscope, wherein the image processing method includes the following steps: generating a three-dimensional model of the subject based on a group of images captured by the endoscope; detecting an unobserved area not captured by the endoscope based on the three-dimensional model; obtaining an action signal of an endoscope system including the endoscope; and temporarily stopping detection of the unobserved area when the action signal is consistent with specified conditions.

[0011] One embodiment of the present invention is a storage medium, which is a non-temporary storage medium readable by a computer, wherein the storage medium stores an image processing program, which causes a computer to perform image processing on an image of a subject photographed by an endoscope, and the image processing program causes the computer to perform the following processing: generating a three-dimensional model of the subject based on a group of images photographed by the endoscope; detecting an unobserved area that is not photographed by the endoscope based on the three-dimensional model; obtaining an action signal of an endoscope system including the endoscope; and temporarily stopping the detection of the unobserved area when the action signal is consistent with a specified condition, the specified condition including the action of the endoscope system causing a portion of the image group to be missing. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing the overall configuration of the image processing device and the endoscope system according to the first embodiment.

[0013] Figure 2 This is a diagram illustrating colon endoscopy.

[0014] Figure 3 The following is a diagram for explaining the generation of a 3D model during colon endoscopy.

[0015] Figure 4 The diagrams are for explaining images displayed on a display device during colon endoscopy.

[0016] Figure 5 This is a flowchart of the image processing method according to the first embodiment.

[0017] Figure 6 This is a flowchart of a modified example of the image processing method according to the first embodiment.

[0018] Figure 7 This is a diagram showing a selection screen for a user to select a predetermined condition.

[0019] Fig. 8A This is a diagram showing an example of a report after a colon endoscopy.

[0020] Figure 8B This is a diagram showing another example of a report after a colon endoscopy.

[0021] Fig. 9 It is a block diagram showing the overall configuration of an image processing device and an endoscope system according to the second embodiment.

[0022] Fig.10 This is a diagram showing the correspondence between the combination of equipment in the endoscope system and predetermined conditions.

[0023] Fig.11This is a flowchart of the image processing method according to the second embodiment.

[0024] Description of Reference Numerals

[0025] 1. 102 processor

[0026] 2 Storage unit (recording medium)

[0027] 2a Image processing program

[0028] 4 User Interface

[0029] 6 Conditional Storage Unit

[0030] 10.101 Image processing device

[0031] 20 Endoscope

[0032] 22 Zoom mechanism

[0033] 23 Angle sensor (sensor)

[0034] 30 Light source device

[0035] 40 Control device

[0036] 61 Air pump

[0037] 62 Suction pump

[0038] 63 Water pump

[0039] 64 Suction pump

[0040] 65 High frequency treatment equipment

[0041] 66UPD (Sensor)

[0042] C 3D Model

[0043] D Unobserved area DETAILED DESCRIPTION

[0044] (First Embodiment)

[0045] An image processing device, an image processing method, an image processing program, and a storage medium according to a first embodiment of the present invention will be described with reference to the drawings.

[0046] like Figure 1 As shown, the image processing device 10 of the present embodiment is applied to an endoscope system 100 .

[0047] The endoscope system 100 includes an image processing device 10, an endoscope 20, a light source device 30, a control device 40, and a display device 50. In addition, the endoscope system 100 includes peripheral equipment 60 of the endoscope 20 used for endoscopic inspection. The peripheral equipment 60 includes, for example, an air supply pump 61, an air suction pump 62, a water supply pump 63, a water suction pump 64, a high-frequency treatment device 65, and a UPD (endoscope insertion shape observation device) 66.

[0048] The endoscope 20 is a soft endoscope for use in digestive organs such as the large intestine. The endoscope 20 includes a long soft insertion portion 20a, a bending portion 20b provided at the front end of the insertion portion 20a, and an operation portion (not shown) connected to the base end of the insertion portion 20a (see FIG. Figure 2 ).

[0049] The endoscope 20 includes an imaging optical system 21 for photographing a subject, a zoom mechanism 22 for enlarging and reducing the subject in the image, and an angle sensor 23 for detecting the bending angle of the bending portion 20b. The imaging optical system 21 includes an imaging element such as an objective lens and a CMOS image sensor. The zoom mechanism 22 changes the magnification of the subject in the image by optical zoom or digital zoom. For example, the zoom mechanism 22 switches from a normal magnification to a high magnification by turning on a zoom switch provided on the operation portion by the user.

[0050] The light source device 30 is connected to the endoscope 20, and supplies illumination light to the endoscope 20. The light source device 30 includes LEDs of multiple colors, and can output multiple types of illumination light by turning each LED on and off.

[0051] For example, the light source device 30 has five LEDs, namely purple, blue, green, amber and red. The various illumination lights include white light for normal observation and special light for special light observation. White light is composed of five colors of light. Special light is composed of blue and green light for NBI (narrowband imaging).

[0052] The control device 40 includes a processor 41, an input / output unit 42, and a user interface 43. The input / output unit 42 has a known input / output interface, and the control device 40 is connected to the image processing device 10, the endoscope 20, the light source device 30, and the peripheral device 60 via the input / output unit 42. The image captured by the endoscope 20 is input to the display device 50 via the control device 40 and the image processing device 10, and is displayed on the display device 50. The display device 50 is any type of display such as a liquid crystal display.

[0053] The control device 40 controls the operation of the light source device 30 and the peripheral device 60. For example, the processor 41 generates a control signal for controlling the light source device 30 and the peripheral device 60 according to the user's operation on the user interface 43, and sends the control signal to the light source device 30 and the peripheral device 60.

[0054] The image processing device 10 includes a processor 1 such as a central processing unit, a storage unit 2, a memory 3, and a user interface 4. For example, the image processing device 10 is configured by an arbitrary computer such as a personal computer.

[0055] The storage unit 2 is a non-transitory storage medium readable by a computer, such as a known magnetic disk, optical disk, or flash memory, etc. The storage unit 2 stores an image processing program 2 a for causing the processor 1 to execute an image processing method described later.

[0056] The memory 3 is composed of a volatile storage device such as a RAM (random access memory), and is used as a work area of ​​the processor 1 .

[0057] The user interface 4 includes input devices such as a mouse, a keyboard, and a touch panel, and receives operations on the input devices by the user.

[0058] The processor 1 has a three-dimensional (3D) model generating section 11 , a determining section 12 , a data removing section 13 , a non-observed region detecting section 14 , and a display controlling section 15 as functional sections.

[0059] The image processing device 10 also includes an image storage unit 5 that stores an image group used for generating a 3D model, and a condition storage unit 6 that stores a predetermined condition for temporarily stopping detection of an unobserved area. The storage units 5 and 6 are composed of, for example, the storage unit 2, the memory 3, or other storage devices.

[0060] The processor 1 stores the images input from the endoscope 20 to the image processing device 10 via the control device 40 in the image storage unit 5, and generates an image group consisting of a plurality of images for the generation of a 3D model. The processor 1 may store all the images input to the image processing device 10, or may select and store images suitable for the generation of a 3D model.

[0061] The 3D model generation unit 11 obtains the image group from the image storage unit 5 and generates a 3D model representing the 3D shape of the subject in the image group based on the image group. To generate the 3D model, a known three-dimensional reconstruction technology such as visual SLAM (Simultaneous Localization and Mapping) is used.

[0062] Figure 2 and Figure 3Colonoscopy and generation of a 3D model of the large intestine are described respectively.

[0063] In the colon endoscopy, after the endoscope 20 is inserted from the anus to the cecum, the cecum, ascending colon, transverse colon, descending colon, sigmoid colon, and rectum are observed in sequence while the endoscope 20 is pulled out toward the anus. If necessary, in order to observe the mucosa of the large intestine, the mucosa is cleaned by water supply, and air is supplied and suctioned. If a lesion is found, the lesion is observed by magnification or special light, and the high-frequency treatment device 65 is used for treatment as needed.

[0064] like Figure 3 As shown, as the endoscope 20 moves, 3D models C are generated sequentially from the cecum. The 3D model C may include a missing portion D where the model of the object is not created. The missing portion D corresponds to an unobserved area B of the object that is not observed by the endoscope 20. For example, in a colon endoscopy, an area on the inner side of a fold A that is easily a blind spot may become an unobserved area B. Figure 2 and Figure 3 In the figure, an unobserved region B is generated on the inner side of the wrinkle A at time t2, and a missing portion D corresponding to the unobserved region B is formed in the 3D model C.

[0065] In order to generate a 3D model C that is continuous over the entire range of movement of the visual field of the endoscope 20, it is necessary that the subject is continuous in the plurality of images used for generating the 3D model C. However, due to the operation of the endoscope system 100, an image that does not have information on the form of the subject may be input to the image processing device 10, or an image may not be input to the image processing device 10 temporarily. In such a case, a portion of the image group is missing, for example, a portion of the images in the image group does not have information on the form of the subject, or the image group does not include an image of a portion of the subject. Therefore, the image group that is missing a portion includes a discontinuous portion of the subject that does not have information on the form of the subject.

[0066] For example, when switching from white light to special light for NBI, the intensity of the illumination light is temporarily reduced, and a dark image without information on the shape of the object is temporarily acquired through the endoscope 20. In the image group including the dark image, a part of the object corresponding to the dark image becomes a discontinuous portion.

[0067] Regardless of whether a part of the image group is missing, the 3D model generation unit 11 continues to generate the 3D model C. The 3D model C generated based on the image group with a part missing includes a missing part corresponding to the missing part of the image group.

[0068] The determination unit 12 acquires an operation signal of the endoscope system 100, and compares the operation signal with the predetermined condition stored in the condition storage unit 6. The determination unit 12 determines whether the operation signal matches the predetermined condition.

[0069] The operation signal is a signal indicating the operation of the devices 20, 30, 61-66 constituting the endoscope system 100, and particularly includes a signal indicating a predetermined operation of the devices 20, 30, 61-66 causing a partial loss of the image group. The determination unit 12 obtains the operation signal from the control device 40 or the devices 20, 30, 61-66.

[0070] An example of the action signal is a signal indicating the switching of the type of illumination light supplied from the light source device 30 to the endoscope 20 , for example, a control signal outputted from the control device 40 to the light source device 30 to switch the type of illumination light. The determination unit 12 obtains the control signal from the control device 40 .

[0071] The predetermined condition is a predetermined operation of the endoscope system 100 that causes a part of the image group to be lost. For example, the predetermined condition includes switching the type of illumination light supplied from the light source device 30 to the endoscope 20 .

[0072] When the motion signal matches the predetermined condition, the data removing unit 13 detects the portion corresponding to the predetermined condition (i.e., the portion corresponding to the missing portion of the image group), that is, the missing portion in the 3D model C, and removes the data of the missing portion from the 3D model C. Thus, the 3D model C is generated without the missing portion caused by the motion of the endoscope system 100.

[0073] The unobserved region detecting unit 14 detects the unobserved region in the 3D model C based on the determination result of the determining unit 12 .

[0074] Specifically, when it is determined that the motion signal does not match the predetermined condition, the unobserved region detecting unit 14 detects the missing portion D in the 3D model C generated by the 3D model generating unit 11 as the unobserved region.

[0075] On the other hand, when it is determined that the motion signal matches the predetermined condition, the unobserved region detection unit 14 temporarily stops detecting the unobserved region. After the missing portion caused by the motion of the endoscope system 100 is removed by the data removal unit 13, the unobserved region detection unit 14 restarts detecting the unobserved region, and detects the missing portion D in the 3D model C after the missing portion caused by the motion of the endoscope system 100 is removed as the unobserved region.

[0076] like Figure 4As shown, the display control unit 15 generates displays E1, E2, and E3 indicating that the unobserved area is detected, and outputs the displays E1, E2, and E3 together with the current image F to the display device 50 and displays them on the display device 50. The display control unit 15 may also output the displays E1, E2, and E3 together with the image F at a position at a certain distance from the unobserved area D, or the image F after a certain time from the detection of the unobserved area D.

[0077] exist Figure 4 In the example of FIG. 5 , the unobserved area D is detected at time t2 c , and at the subsequent time t3 , displays E1 , E2 , and E3 are displayed on the display device 50 .

[0078] One example of display is an image E1 when the unobserved area D is detected, and a mark may be added to the unobserved area B in the image E1. Another example of display is an arrow E2 indicating the position of the unobserved area B in the current image F. Another example of display is a frame E3 of a predetermined color added to the image F. By displaying at least one of the displays E1, E2, and E3 together with the image F, the user is notified of the existence of the unobserved area B.

[0079] Figure 4 These are images obtained when an endoscope is inserted into the digestive tract and then removed. Figure 4 The image when observing the wall surface is omitted.

[0080] Reference numeral 50 denotes a display screen, F denotes a portion for displaying an image obtained by the endoscope, and a guide area for performing a guide display and the like is provided outside the portion.

[0081] “t2b” is when the imaging unit at the distal end of the endoscope is directed toward the deep side of the lumen. In this image, the deep wrinkle H1 and the front wrinkle H2 can be seen.

[0082] like Figure 2 As shown at "t1" and "t2", the lumen wall surface is photographed near the inner surface of the wrinkle of H2 to generate a 3D model.

[0083] After H1 and Figure 2 The observations of t1 and t2 are not missed in H2, so Figure 4 In "t2b", "Nothing is missed" is displayed in the boot area.

[0084] Figure 4 The "t2c" means Figure 2 After t2, after the observation of the inside of the surface of H3 is completed. At this time, because the obtained image is used for 3D modeling, it is displayed as "Focus on wrinkle area detection" and "Missed area detection". Based on the evaluation of the image group, it is determined whether there is an area like B (a part of the captured image that is missing). When making this judgment, the same area as in Figure 2 The image obtained by the camera unit of the endoscope when observing the wrinkles of H4 at time t3 is equivalent to the image obtained by the endoscope, but when the result of the judgment at "t2c" is known, the edge of the frame E3 can be lit or flashed to draw attention, or an arrow can be used to indicate the missed wrinkles or parts as in E2, or a display of "missed" or the missed position converted based on the 3D model can be displayed in the guide area E1 to instruct the doctor to confirm again. In this way, the examination can be carried out without missing any wrinkles or parts.

[0085] Depending on the operation of the endoscope, e.g. Figure 4 Sometimes, the 3D modeling cannot be performed due to the timing of t2c. In this case, a display such as "stopped due to missed detection" can be made to inform the user that there is an operation to stop the missed detection function. In this way, it can be clearly indicated that the function is just stopped. It can also be displayed for what reason the function is stopped, and it can also be conveyed that there is no need to recheck.

[0086] Next, an image processing method executed by the image processing device 10 will be described.

[0087] like Figure 5 As shown, the image processing method of this embodiment includes: step S1, obtaining an image captured by the endoscope 20; step S2, generating a 3D model C; step S3, detecting an unobserved area D in the 3D model C; and step S4, generating and outputting a display representing the unobserved area D. In addition, the image processing method includes: step S5, obtaining an action signal of the endoscope system 100; step S6, determining whether the action signal is consistent with a prescribed condition; step S7, temporarily stopping the detection of the unobserved area or performing image determination and selection for the detection of the unobserved area; and step S8, removing the data of the missing part from the 3D model C as needed.

[0088] The processor 1 starts to obtain images input from the endoscope 20 to the image processing device 10, and stores the images in the image storage unit 5 in sequence (step S1). Next, the 3D model generation unit 11 starts to generate a 3D model C of the subject based on the image group obtained continuously in time sequence by pasting images of similar parts of the images of image frames that are roughly adjacent in time (step S2). When there is a missed view, since the image frame at that time does not exist, it is determined to be a missed view, but even if it is not missed, if there is no similar part in the image, it is sometimes impossible to paste and generate a 3D model. In this case, there is also a case where it is not a missed view, so there is also an idea of ​​not performing missed view detection, and an idea of ​​searching for alternative image frames from frames at roughly adjacent moments (selecting between unusable frames and usable frames) to generate a 3D model in order to perform missed view detection as much as possible.

[0089] The unobserved region detection unit 14 detects the unobserved region in the generated 3D model C (step S3). When the unobserved region D is detected, the display control unit 15 generates displays E1, E2, and E3 indicating the detected unobserved region D, and outputs the displays E1, E2, and E3 together with the image F to the display device 50 (step S4).

[0090] Here, after starting to generate the 3D model C, the determination unit 12 starts to obtain the operation signal of the endoscope system 100 (step S5). When the operation signal is obtained with the operation of the endoscope system 100 (yes in step S5), the determination unit 12 then determines whether the operation signal meets a predetermined condition (step S6).

[0091] When the operation signal does not match the predetermined condition (No in step S6), step S3 is then executed.

[0092] On the other hand, when the action signal is consistent with the prescribed conditions (yes in step S6), the unobserved area detection unit 14 temporarily stops the detection of the unobserved area, or determines and selects the image for the unobserved area detection (step S7). This is because, as described above, the image frame that is not suitable for the generation of the 3D model is not used, and an image is found as a substitute for the unsuitable image (for example, even in the case of special light observation, the (usually white light) image before the special light observation mode is set to observe the same object, and sometimes it can be replaced by comparison). The data removal unit 13 removes the data of the missing part corresponding to the prescribed condition from the 3D model C according to whether the above-mentioned selection is performed (step S8). If there is a replaceable image and the replacement image can be used for 3D modeling, the image can also be removed. After removing the data of the missing part as needed, the unobserved area detection unit 14 starts the detection of the unobserved area again (step S3).

[0093] Thus, according to the image processing device 10 of the present embodiment, by determining whether the action signal of the endoscope system 100 is consistent with the predetermined condition, the predetermined action of the endoscope system 100 that causes a part of the image group to be lost is detected. After the predetermined action is detected, the detection of the unobserved area is temporarily stopped to remove the data of the missing part caused by the predetermined action from the 3D model, and then the detection of the unobserved area is restarted. Therefore, even if a part of the image group is lost, it is possible to prevent the missing part caused by the loss of the image group from being erroneously detected as the unobserved area, thereby preventing the occurrence of a problem in the detection of the unobserved area. In addition, it is possible to provide an image processing device that processes an image of a subject photographed by an endoscope, wherein when generating a 3D model of the subject based on the image group photographed by the endoscope, the action signal of the endoscope system is obtained, and when the action signal is consistent with the predetermined condition (the action of the endoscope system that may cause a part of the image group to be lost), the image used for generating the 3D model is selected and discarded, thereby appropriately generating the 3D model and detecting the unobserved area that is not photographed by the endoscope.

[0094] In the present embodiment, the 3D model generating unit 11 continues to generate the 3D model regardless of the operation of the endoscope system 100 . However, instead, the 3D model generating unit 11 may temporarily stop the generation of the 3D model according to the operation of the endoscope system 100 .

[0095] Specifically, if Figure 6 As shown, when the action signal is consistent with the prescribed condition (Yes in step S6), the 3D model generation unit 11 temporarily stops the generation of the 3D model (step S9), and the unobserved area detection unit 14 temporarily stops the detection of the unobserved area (step S7). The data removal unit 13 removes the image corresponding to the prescribed condition (for example, a dark image caused by the switching of the illumination light) from the image group (step S10), and then the 3D model generation unit 11 restarts the generation of the 3D model (step S11). Therefore, the unobserved area detection unit 14 detects the missing part D in the 3D model C that does not include the missing part caused by the action of the endoscope system 100 as the unobserved area.

[0096] When the 3D model is continuously generated, the data volume of the 3D model becomes huge, and a large amount of data needs to be saved. In addition, the data processing for generating the 3D model takes time. By temporarily stopping the generation of the 3D model when the prescribed action of the endoscope system 100 is detected, these undesirable situations can be eliminated and the processing speed of the processor 1 can be improved.

[0097] In the present embodiment, the processor 1 may temporarily stop detecting the unobserved area when there is no need to detect the unobserved area, in addition to when a part of the image group is missing.

[0098] In this case, the operation signal acquired by the determination unit 12 in step S5 includes a signal indicating that the endoscope system 100 does not need to detect the unobserved area. Furthermore, the predetermined condition compared with the operation signal by the determination unit 12 in step S6 includes the endoscope system 100 does not need to detect the unobserved area.

[0099] The operation signal indicating that the operation of the endoscope system 100 does not need to detect the unobserved area includes at least one of the following signals.

[0100] A first example of the operation signal is a signal indicating a change in the magnification of the zoom mechanism 22, such as an on signal of a zoom switch. In this case, the predetermined condition includes a change in the zoom magnification of the endoscope 20, such as switching from a normal magnification to a high magnification.

[0101] A second example of the action signal is a signal indicating the output of special light from the light source device 30, for example, a control signal output by the control device 40 to the light source device 30 to switch from white light to special light. In this case, the predetermined condition includes switching of the illumination light output by the light source device 30 to special light.

[0102] A third example of the operation signal is a signal indicating the operation of each of the air supply pump 61 and the air suction pump 62, and is, for example, a control signal outputted from the control device 40 to the pumps 61 and 62 to operate the pumps 61 and 62. In this case, the predetermined condition includes the operation of the pumps 61 and 62.

[0103] A fourth example of the operation signal is a signal indicating the operation of the water delivery pump 63 and the water suction pump 64, for example, a control signal outputted from the control device 40 to the pumps 63 and 64 to operate the pumps 63 and 64. In this case, the predetermined condition includes the operation of the pumps 63 and 64.

[0104] A fifth example of the operation signal is a signal indicating the operation of the high-frequency treatment device 65, and is, for example, an on signal of a switch of the high-frequency treatment device 65. In this case, the predetermined condition includes the operation of the high-frequency treatment device 65.

[0105] The sixth example of the operation signal is a signal output by the angle sensor 23 or the UPD 66 that detects the bending angle of the bending portion 20b. In this case, the predetermined condition includes that the bending portion 20b is bent at a bending angle equal to or greater than a predetermined value.

[0106] When identifying or treating a lesion, the user can observe the mucosa at a high magnification, observe the lesion with special light, treat the lesion using the high-frequency treatment device 65, or observe the back by bending the bending portion 20b at a large angle. In identification or treatment, the user does not need to detect the unobserved area.

[0107] During air supply and air suction as well as water supply and water suction, it is not necessary for the user to detect the unobserved area.

[0108] The unobserved region detection unit 14 may restart the unobserved region detection after a certain time has passed after temporarily stopping the unobserved region detection. For example, the unobserved region detection unit 14 may restart the unobserved region detection after a certain time has passed after the pumps 61 and 63 start supplying air or water.

[0109] In the present embodiment, the processor 1 may set a predetermined condition for temporarily stopping the detection of the unobserved area according to the operation of the user interface 4 by the user.

[0110] According to this configuration, the user can set a condition for temporarily stopping the detection of the unobserved area.

[0111] For example, Figure 7 As shown, the processor 1 causes the display device 50 to display a selection screen including a plurality of conditions. A user such as a doctor selects a desired condition from the plurality of conditions by operating the user interface 4. Figure 7 In the example, a condition is selected by checking a check box. The processor 1 sets the selected condition as a predetermined condition, and the determination unit 12 determines whether the operation signal matches the selected condition.

[0112] In the present embodiment, the display control unit 15 may generate a report on the stop of detection of the unobserved area after the end of the endoscopic examination. Fig. 8A and Figure 8B An example of a report is shown.

[0113] Fig. 8A The report includes a schematic diagram of the large intestine, with marks added to indicate where the test was stopped in the unobserved areas. Figure 8B The report is a table showing the number of times detection of the unobserved area at each part of the large intestine was stopped.

[0114] (Second Embodiment)

[0115] Next, an image processing device, an image processing method, an image processing program, and a storage medium according to a second embodiment of the present invention will be described.

[0116] The present embodiment is different from the first embodiment in that a predetermined condition is set for each combination of devices constituting the endoscope system.

[0117] In this embodiment, the configurations different from those of the first embodiment will be described, and the configurations common to the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0118] like Fig. 9 As shown, the image processing device 101 of the present embodiment is applied to an endoscope system 200 including an endoscope 20 , a light source device 30 , a control device 40 , a display device 50 , and peripheral equipment 60 .

[0119] The image processing device 101 includes a processor 102 , a storage unit 2 , a memory 3 , a user interface 4 , an image storage unit 5 , and a condition storage unit 6 .

[0120] Fig.10 The diagram shows combinations of device models that constitute the endoscope system 200 and prescribed conditions corresponding to each combination. The condition storage unit 6 stores the correspondence between the combinations of device models and the prescribed conditions.

[0121] like Fig.10 As shown, multiple models of endoscopes 20 "Y1", "Y2", "Y3" and "Y4" can be used in the endoscope system 200. The functions of the endoscope 20 are different for each model. Similarly, multiple models of light source devices 30 and multiple models of control devices 40 can be used in the endoscope system 200.

[0122] exist Fig.10 In the example, the control device 40 of the model "X1" corresponds to the functions of NBI, RDI (Red Dichromatic Imaging) and EDOF (Extended Depth of Field). RDI is an observation method that irradiates amber, green and red light as illumination light to the subject in order to facilitate observation of deep blood vessels and bleeding. EDOF is a technology that generates an image focused in a wide range from two images focused at the near point and the far point respectively.

[0123] The endoscopes 20 of the models "Y1" and "Y2" have the functions of NBI, RDI, and EDOF. The endoscopes 20 of the models "Y3" and "Y4" do not have the functions of NBI and RDI, and can only perform normal observation using white light.

[0124] The light source device 30 of the model “Z1” has LEDs of five colors: purple, blue, green, amber, and red.

[0125] For example, the predetermined condition corresponding to the combination of “X1”, “Y1”, and “Z1” is the turning-on operation of a switch for executing the function of NBI, RDI, or EDOF.

[0126] There is no prescribed condition corresponding to the combination of "X1", "Y3", and "Z1".

[0127] The processor 102 has a 3D model generating unit 11 , a determining unit 12 , an unobserved region detecting unit 14 , and a display controlling unit 15 as functional units.

[0128] The determination unit 12 obtains system information from, for example, the control device 40. The system information is model information of the devices 20, 30, and 40 that constitute the endoscope system 200 and are directly or indirectly connected to the image processing device 101. The determination unit 12 obtains the prescribed conditions corresponding to the combination of the models of the devices 20, 30, and 40 from the condition storage unit 6, compares the action signal with the prescribed conditions, and determines whether the action signal is consistent with the prescribed conditions.

[0129] When it is determined that the motion signal does not match the predetermined condition, the unobserved region detecting unit 14 detects the missing portion D in the 3D model generated by the 3D model generating unit 11 as the unobserved region.

[0130] When determining that the operation signal matches the predetermined condition, the unobserved region detection unit 14 temporarily stops detecting the unobserved region.

[0131] Next, an image processing method executed by the image processing apparatus 101 will be described.

[0132] like Fig.11 As shown, the image processing method of this embodiment includes step S0 of acquiring system information and steps S1 to S7.

[0133] For example, when the endoscope 20 is connected to the control device 40, the control device 40 inputs the system information to the image processing device 101. The processor 102 obtains the system information input to the image processing device 101 and stores the system information in a storage device in the image processing device 101, such as the condition storage unit 6 (step S0).

[0134] Next, steps S1 to S7 are executed in the same manner as in the first embodiment.

[0135] The unobserved area detection unit 14 may also automatically restart the detection of the unobserved area after temporarily stopping the detection of the unobserved area in the 3D model (step S7). For example, the unobserved area detection unit 14 may restart the detection of the unobserved area based on a signal indicating the off action of the switch of NBI, RDI or EDOF.

[0136] After step S7 , step S8 or steps S10 and S11 described in the first embodiment may be executed.

[0137] Thus, according to the image processing device 101 of the present embodiment, by determining whether the action signal of the endoscope system 200 is consistent with the prescribed condition, the prescribed action of the endoscope system 200 that causes a part of the image group to be missing is detected. After the prescribed action is detected, the detection of the unobserved area is temporarily stopped. Thus, even when a part of the image group is missing, it is possible to prevent the missing part caused by the missing image group from being erroneously detected as the unobserved area, thereby preventing the occurrence of a bad situation in the detection of the unobserved area.

[0138] Furthermore, since the functions of the devices 20, 30, and 40 differ according to the models, the operation of the endoscope system 200 that causes a portion of the image group to be lost differs according to each combination of the devices 20, 30, and 40. According to this embodiment, the prescribed conditions suitable for the combination of the models of the devices 20, 30, and 40 can be automatically set.

[0139] In this embodiment, the modified example described in the first embodiment can also be applied.

[0140] That is, the prescribed condition may also include the action of the endoscope system that does not require detection of the unobserved area. In addition, the prescribed condition may also be set from a plurality of conditions based on the user's operation on the user interface 4 (see Figure 7 ).

[0141] The display control unit 15 may also generate a report after the endoscope examination is completed (see Fig. 8A and Figure 8B ).

[0142] Above, the embodiments of the present invention and its variants are described in detail with reference to the accompanying drawings, but the specific structure of the present invention is not limited to the above embodiments and variants, and various design changes can be made within the scope of the gist of the present invention. In addition, the constituent elements shown in the above embodiments and variants can be appropriately combined.

[0143] For example, the object observed by the endoscope may be an organ other than the large intestine.

Claims

1. An image processing device, applied to an endoscope system, for processing an image of a subject taken by an endoscope, wherein: The image processing device includes a processor. The processor performs the following processing: generating a three-dimensional model of the object based on a group of images captured by the endoscope; Detecting an unobserved area not photographed by the endoscope according to the three-dimensional model; obtaining an action signal of the endoscope system; as well as When the action signal matches a predetermined condition, the detection of the unobserved area is temporarily stopped. The predetermined condition includes an action of the endoscope system that causes a portion of the image group to be lost.

2. An image processing device, applied to an endoscope system, for processing an image of a subject taken by an endoscope, wherein: The image processing device includes a processor. The processor performs the following processing: When generating a three-dimensional model of the object based on the image group captured by the endoscope, obtaining an action signal of the endoscope system, When the action signal is consistent with a predetermined condition, the image when generating the three-dimensional model is selected and discarded to generate a three-dimensional model of the object, and an unobserved area not observed by the endoscope is detected based on the three-dimensional model. The predetermined condition includes an action of the endoscope system that causes a portion of the image group to be lost.

3. The image processing device according to claim 1 or 2, wherein: The operation signal includes a signal indicating switching of the type of illumination light supplied from the light source device to the endoscope. The predetermined condition includes switching of the type of the illumination light.

4. The image processing device according to claim 1 or 2, wherein: The predetermined condition includes an operation of the endoscope system that does not require detection of the unobserved area.

5. The image processing device according to claim 4, wherein: The action signal includes a signal indicating a change in magnification of a zoom mechanism of the endoscope. The predetermined condition includes a change in a zoom factor of the endoscope.

6. The image processing device according to claim 4, wherein: The action signal includes a signal indicating the operation of the air delivery pump or the air suction pump. The prescribed conditions include the operation of the air delivery pump or the air suction pump.

7. The image processing device according to claim 4, wherein: The action signal includes a signal indicating the operation of the water delivery pump or the water suction pump. The prescribed condition includes the operation of the water delivery pump or the water suction pump.

8. The image processing device according to claim 4, wherein: The action signal includes a signal indicating the operation of the high-frequency treatment device. The prescribed conditions include the operation of the high-frequency treatment device.

9. The image processing device according to claim 4, wherein: The action signal includes a signal of a sensor, the sensor detecting a bending angle of a bending portion of the endoscope, The predetermined condition includes that the bending portion is bent at a bending angle equal to or greater than a predetermined value.

10. The image processing device according to claim 1 or 2, wherein: The image processing device further includes a user interface for accepting user operations. The processor selects at least one from a plurality of conditions based on the operation accepted by the user interface, and sets the selected condition as the prescribed condition.

11. The image processing device according to claim 1 or 2, wherein: The image processing device further comprises a condition storage unit. The condition storage unit stores a correspondence relationship between a combination of models of a plurality of devices constituting the endoscope system and the prescribed conditions, the plurality of devices including the endoscope and a light source device for supplying illumination light to the endoscope. The processor performs the following processing: obtaining model information of each of a plurality of devices connected to the image processing apparatus; and The predetermined condition corresponding to the combination of the models is acquired from the condition storage unit.

12. The image processing device according to claim 2, wherein: When selecting images for generating the three-dimensional model, if it is determined that an image in the image group is not suitable for generating the three-dimensional model, an image is found as a substitute for the unsuitable image to be used for generating the three-dimensional model.

13. The image processing apparatus according to claim 12, wherein: The substitute image is an image obtained by photographing the same observation object as that photographed by the unsuitable image in a different observation mode.

14. The image processing apparatus according to claim 1, further comprising: When the operation signal matches a predetermined condition, the generation of the three-dimensional model is temporarily stopped immediately before the detection of the unobserved area is temporarily stopped.

15. An image processing method for processing an image of a subject taken by an endoscope, wherein: The image processing method comprises the following steps: generating a three-dimensional model of the object based on a group of images captured by the endoscope; Detecting an unobserved area not photographed by the endoscope according to the three-dimensional model; obtaining an action signal of an endoscope system including the endoscope; as well as When the action signal matches a predetermined condition, the detection of the unobserved area is temporarily stopped. The predetermined condition includes an action of the endoscope system that causes a portion of the image group to be lost.

16. An image processing method for processing an image of a subject taken by an endoscope, wherein: The image processing method comprises the following steps: When generating a three-dimensional model of the object based on the image group captured by the endoscope, obtaining an action signal of the endoscope system, When the action signal is consistent with a predetermined condition, the image when generating the three-dimensional model is selected and discarded to generate a three-dimensional model of the object, and an unobserved area not observed by the endoscope is detected based on the three-dimensional model. The predetermined condition includes an action of the endoscope system that causes a portion of the image group to be lost.

17. A computer-readable non-transitory storage medium, wherein: The storage medium stores an image processing program that causes a computer to execute image processing for processing an image of a subject captured by an endoscope. The image processing program causes the computer to perform the following processing: generating a three-dimensional model of the object based on a group of images captured by an endoscope; Detecting an unobserved area not photographed by the endoscope according to the three-dimensional model; obtaining an action signal of an endoscope system including the endoscope; as well as When the action signal matches a predetermined condition, the detection of the unobserved area is temporarily stopped. The predetermined condition includes an action of the endoscope system that results in a loss of a portion of the set of images.

Images