Image processing device and program

Abstract

An image processing device (4) comprises a processor (4a) equipped with hardware. The processor (4a) is configured to generate an internal organ model by acquiring endoscopic image information from an endoscope, and then, while continuing acquiring of the endoscopic image information, perform corrections to the shape of the generated internal organ model on the basis of the latest endoscopic image information, and output information of the corrected internal organ model.

Description

[Technical Field]

[0001] The present invention relates to an image processing device that acquires endoscopic image information and generates an organ model. 、 and regarding the program. [Background technology]

[0002] In endoscopic examinations, it is necessary to observe all areas of the organ being examined to prevent overlooking any lesions.

[0003] For example, U.S. Patent Publication No. 10,682,108 describes a technique for creating a three-dimensional organ model based on two-dimensional endoscopic images using direct sparse odometry (DSO) and a neural network. The three-dimensional organ model is used, for example, to determine the position of an endoscope. It is also used to identify unobserved regions by showing unvisualized parts (i.e., unobserved parts) in the organ model.

[0004] However, some organs change shape over time, and the shape and position of the organ within the body may change as an endoscope is inserted and removed.

[0005] Previously created organ models using the technology of U.S. Patent Publication No. 10,682,108 may no longer match the current organ if the shape or position of the organ changes. Specifically, multiple organ models may be generated for the same region, or the display of unobserved regions may become misaligned with the current state. As a result, the guide display for unobserved regions may no longer indicate the correct direction or position.

[0006] The present invention has been made in view of the above circumstances, and provides an image processing device capable of generating an organ model that matches the current state of an organ. 、 and programs. DISCLOSURE OF THE INVENTION [Means for solving the problem]

[0007] An image processing device according to one aspect of the present invention includes a processor having hardware, wherein the processor acquires endoscopic image information from an endoscope and generates an organ model, and then continues acquiring the endoscopic image information, and identifies a changed portion of the generated organ model based on the latest endoscopic image information; Estimating a change amount of the changed portion, which is at least one of an amount of expansion / contraction, an amount of rotation, an amount of extension / contraction, and an amount of movement; Based on the estimated amount of change in the changed area, the shape of at least a part of the organ model including the changed area is corrected, and information on the corrected organ model is output.

[0009] A program according to one aspect of the present invention causes a computer to acquire endoscopic image information from an endoscope and generate an organ model, and then causes the computer to continue acquiring the endoscopic image information and identify a changed portion of the generated organ model based on the latest endoscopic image information; A change amount of the changed portion, which is at least one of an amount of enlargement / reduction, an amount of rotation, an amount of extension / contraction, and an amount of movement, is estimated, and based on the estimated change amount of the changed portion, The shape of at least a part of the organ model including the changed portion is corrected, and information on the corrected organ model is output. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a perspective view showing a configuration of an endoscope system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a diagram mainly showing the structural and functional configuration of an image processing device in the first embodiment. [Figure 3] FIG. 2 is a block diagram showing an example of the configuration of the image processing device of the first embodiment when viewed as a structural unit. [Figure 4] 6 is a flowchart showing processing of the image processing device of the first embodiment. [Figure 5] FIG. 2 is a diagram for explaining generation of an organ model by an organ model generation unit in the first embodiment. [Figure 6] 10 is a diagram showing an overall image of an organ model and an example of an organ model that is generated, corrected, and displayed in the first embodiment. [Figure 7] FIG. 10 is a diagram for explaining an example of detecting a change in an organ model based on feature points in the first embodiment. [Figure 8] 10 is a flowchart showing processing performed by an image processing device according to a second embodiment of the present invention. [Figure 9] 10 is a chart showing how the amount of change in an organ model at different times is estimated in the second embodiment. [Figure 10] FIG. 11 is a diagram showing an example of correcting the shape of an organ model based on the amount of change in the second embodiment. [Figure 11] 9 is a flowchart showing the process of estimating the amount of change in the organ model in step S12 of FIG. 8 in the second embodiment. [Figure 12] 10 is a diagram showing how changes in an organ model include expansion, rotation, and movement in the second embodiment. [Figure 13] 12 is a flowchart showing the process of detecting the amount of enlargement / reduction in step S21 of FIG. 11 in the second embodiment. [Figure 14] 10 is a table for explaining the process of detecting the amount of enlargement / reduction in the second embodiment. [Figure 15] 12 is a flowchart showing the process of detecting the amount of rotation in step S22 of FIG. 11 in the second embodiment. [Figure 16] 10 is a table for explaining the process of detecting the amount of rotation in the second embodiment. [Figure 17] 12 is a flowchart showing the process of detecting the amount of expansion / contraction in step S23 of FIG. 11 in the second embodiment. [Figure 18] 10 is a table for explaining the process of detecting the amount of expansion / contraction in the second embodiment. [Figure 19] 12 is a flowchart showing the process of detecting the amount of movement in step S24 of FIG. 11 in the second embodiment. [Figure 20] 10 is a table for explaining an example of a method for correcting the shape of an organ model in step S3A of FIG. 8 in the second embodiment. [Figure 21] 10 is a flowchart showing processing performed by an image processing device according to a third embodiment of the present invention. [Figure 22] 22 is a flowchart showing the process of estimating the amount of change in the folds in step S12B of FIG. 21 in the third embodiment. [Figure 23] 13 is a diagram for explaining a process for detecting whether or not a fold has passed in the third embodiment. [Figure 24] 13 is a diagram for explaining how folds in an endoscopic image are associated with folds in an organ model in the third embodiment. [Figure 25] 13 is a table for explaining how the amount of change in the same fold is detected in the third embodiment. [Figure 26] 23 is a flowchart showing the process of detecting the amount of change in the same fold in step S73 of FIG. 22 in the third embodiment. [Figure 27] 27 is a flowchart showing another example of processing for detecting the amount of expansion or contraction of the diameter in step S81 of FIG. 26 in the third embodiment. [Figure 28] 13 is a table for explaining another example of processing for detecting the amount of expansion or contraction of the diameter in the third embodiment. [Figure 29] 13 is a graph for explaining an example of a method for correcting the amount of expansion / contraction of the diameter of an organ model in the third embodiment. [Figure 30] FIG. 13 is a diagram for explaining an example in which the amount of expansion / contraction of the diameter of the organ model is corrected within a correction range in the third embodiment. [Figure 31] 13 is a graph for explaining an example of a method for correcting the amount of rotation of an organ model in the third embodiment. [Figure 32] 27 is a table for explaining the process of detecting the amount of expansion / contraction in step S83 of FIG. 26 in the third embodiment. [Figure 33] 13 is a graph for explaining an example of a method for correcting the amount of expansion / contraction of an organ model in the third embodiment. [Figure 34]FIG. 13 is a diagram for explaining an example of correcting the amount of expansion / contraction of an organ model in the third embodiment. [Figure 35] 27 is a flowchart for explaining the process of detecting the amount of movement in step S84 of FIG. 26 in the third embodiment. [Figure 36] FIG. 13 is a diagram showing an example of detecting identical folds in an existing organ model and a new organ model in order to determine the movement of an organ in the third embodiment. [Figure 37] FIG. 13 is a diagram for explaining a method for correcting the shape of an organ model in accordance with movement of the organ in the third embodiment. [Figure 38] 13 is a graph for explaining a method of correcting the shape of an organ model in accordance with the movement of the organ in the third embodiment. [Figure 39] 13 is a diagram showing a display example of an organ model and an unobserved region in the third embodiment. BEST MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the embodiments described below.

[0013] In the drawings, the same or corresponding elements are appropriately designated by the same reference numerals. It should be noted that the drawings are schematic, and that the length relationships, length ratios, and quantities of elements within a single drawing may differ from reality in order to simplify the explanation. Furthermore, there may be parts in which the length relationships and ratios differ between multiple drawings. [First embodiment]

[0014] 1 to 7 show a first embodiment of the present invention, and FIG. 1 is a perspective view showing the configuration of an endoscope system 1 in the first embodiment.

[0015] The endoscope system 1 includes, for example, an endoscope 2, a light source device 3, an image processing device 4, a tip position detection device 5, a suction pump 6, a water tank 7, and a monitor 8. As shown in FIG. 1, all of these devices except the endoscope 2 are mounted on a cart 9 or are fixed in place. The endoscope system 1 is placed, for example, in an examination room where examinations and treatments of subjects are performed. The light source device 3 and the image processing device 4 may be separate entities, or may be integrated into a light source-integrated image processing device. The tip position detection device 5 can be, for example, a technology that detects the position of the endoscope tip by generating a magnetic field. The tip position detection device 5 can also be a device with a known insertion shape. detection A device (UPD) can also be applied.

[0016] The endoscope 2 includes an insertion section 2a, an operation section 2b, and a universal cable 2c.

[0017] The insertion section 2a is the portion to be inserted into the subject, and includes, in order from the distal end to the proximal end, a distal end portion 2a1, a bending portion 2a2, and a flexible tube portion 2a3. The distal end portion 2a1 is provided with an imaging unit including an imaging optical system and an imaging element 2d (see FIG. 2), a magnetic coil 2e (see FIG. 2), the distal end portion of a light guide, and a distal opening of a treatment tool channel.

[0018] The operation section 2b is disposed on the proximal end side of the insertion section 2a, and is a section where various operations are performed by hand.

[0019] The universal cable 2c is a connection cable that extends from, for example, the operation unit 2b and connects the endoscope 2 to the light source device 3, the image processing device 4, the suction pump 6, and the water tank .

[0020] A light guide, a signal cable, a treatment tool channel that also serves as a suction channel, and an air / water supply channel are inserted inside the insertion section 2a, operation section 2b, and universal cable 2c of the endoscope 2.

[0021] A connector provided at the extending end of the universal cable 2c is connected to the light source device 3. A cable extending from the connector is connected to the image processing device 4. Therefore, the endoscope 2 is connected to the light source device 3 and the image processing device 4.

[0022] The light source device 3 includes a light emitting device such as an LED (Light Emitting Diode) light source, a laser light source, or a xenon light source. By connecting a connector to the light source device 3, it becomes possible to transmit illumination light to the light guide.

[0023] Illumination light incident on the proximal end surface of the light guide from the light source device 3 is transmitted through the light guide and is irradiated onto the subject from the distal end surface of the light guide disposed at the distal end portion 2a1 of the insertion portion 2a.

[0024] The suction channel and the air / water supply channel are connected to the suction pump 6 and the water supply tank 7, respectively, via the light source device 3. Therefore, by connecting the connector to the light source device 3, it becomes possible to perform suction through the suction channel by the suction pump 6, to supply water from the water supply tank 7 via the air / water supply channel, and to supply air via the air / water supply channel.

[0025] The suction pump 6 is used to suck liquid or the like from the subject.

[0026] The water supply tank 7 is a tank that stores liquid such as physiological saline. By sending pressurized gas from the air and water supply pump in the light source device 3 to the water supply tank 7, the liquid in the water supply tank 7 is sent to the air and water supply channel.

[0027] The tip position detection device 5 detects the shape of the insertion portion 2a by detecting magnetism generated from one or more magnetic coils 2e (see FIG. 2) provided in the insertion portion 2a using a magnetic sensor (position detection sensor). The tip position detection device 5 detects the position and posture of the tip 2a1 of the insertion portion 2a.

[0028] The image processing device 4 transmits a drive signal for driving the image sensor 2d (see FIG. 2) via a signal cable. The image signal output from the image sensor 2d is transmitted to the image processing device 4 via the signal cable.

[0029] The image processing device 4 performs image processing on the imaging signal acquired by the imaging element 2d to generate and output a displayable image signal. Position information of the tip 2a1 of the insertion section 2a acquired from the tip position detection device 5 is also input to the image processing device 4. Note that the image processing device 4 may control not only the endoscope 2 but also the entire endoscope system 1 including the light source device 3, the tip position detection device 5, the suction pump 6, the monitor 8, etc.

[0030] The monitor 8 displays images including endoscopic images based on the image signals output from the image processing device 4.

[0031] Fig. 2 is a diagram mainly showing the structural and functional configuration of the image processing device in the first embodiment, in which the light source device 3, suction pump 6, water tank 7, etc. are omitted.

[0032] The endoscope 2 is configured as an electronic endoscope, and includes an imaging element 2d and a magnetic coil 2e at the tip 2a1 of the insertion section 2a.

[0033] The imaging element 2d captures an optical image of the subject formed by the imaging optical system and generates an imaging signal. The imaging element 2d captures images, for example, frame by frame, and generates imaging signals related to the images of multiple frames in time series. The generated imaging signals are sequentially output to the image processing device 4 via a signal cable connected to the imaging element 2d.

[0034] The position and posture of the tip 2a1 of the insertion portion 2a detected by the tip position detection device 5 based on the magnetism generated by the magnetic coil 2e are output to the image processing device 4.

[0035] The image processing device 4 includes an input unit 11, an organ model generation unit 12, an organ model shape correction unit 13, a memory 14, an unobserved region determination and correction unit 15, an output unit 16, and a recording unit 17.

[0036] The input unit 11 receives an image pickup signal from the image pickup element 2d and information on the position and posture of the tip portion 2a1 of the insertion portion 2a from the tip position detection device 5.

[0037] The organ model generation unit 12 acquires endoscopic image information (hereinafter referred to as endoscopic image as appropriate) related to the imaging signal from the input unit 11. Then, the organ model generation unit 12 detects the position and orientation of the tip 2a1 of the insertion unit 2a from the endoscopic image. Furthermore, the organ model generation unit 12 acquires information on the position and orientation of the tip 2a1 of the insertion unit 2a from the tip position detection device 5 via the input unit 11 as needed. Furthermore, the organ model generation unit 12 generates a three-dimensional organ model based on the position and orientation of the tip 2a1 and the endoscopic image.

[0038] The organ model shape correcting unit 13 corrects the shape of an organ model that has been generated in the past (an existing organ model) based on the latest endoscopic image.

[0039] The memory 14 stores the corrected organ model.

[0040] The unobserved region determination / correction unit 15 determines the unobserved region in the corrected organ model and corrects the position and shape of the unobserved region according to the corrected organ model. The position and shape of the unobserved region are stored in the memory 14 as necessary.

[0041] The output unit 16 outputs information about the corrected organ model, and also outputs information about unobserved regions as needed.

[0042] The recording unit 17 stores in a non-volatile manner the endoscopic image information that has been image-processed by the image processing device 4 and output from the output unit 16. Note that the recording unit 17 may be a recording device provided outside the image processing device 4.

[0043] Furthermore, the information on the organ model output from the output unit 16 (and information on the unobserved region, if necessary) is displayed on the monitor 8 as an organ model image together with, for example, the endoscopic image.

[0044] While FIG. 2 shows the functional configuration of each piece of hardware in the image processing device 4, FIG. 3 is a block diagram showing an example of the configuration when the image processing device 4 of the first embodiment is viewed in structural units.

[0045] 3, the image processing device 4 includes a processor 4a having hardware and a memory 4b. The processor 4a includes, for example, an ASIC (Application Specific Integrated Circuit) including a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or a GPU (Graphics Processing Unit).

[0046] The memory 4b includes the memory 14 of FIG. 2 and includes, for example, a volatile storage medium such as a RAM (Random Access Memory) and a non-volatile storage medium such as a ROM (Read Only Memory) (or an EEPROM (Electrically Erasable Programmable Read-Only Memory)). The RAM temporarily stores various information such as the image to be processed, processing parameters at the time of execution, and user setting values ​​input from outside. The ROM non-volatilely stores various information such as the processing program (computer program), default values ​​of processing parameters, and user setting values ​​that should be stored even when the power of the endoscope system 1 is turned off.

[0047] The processor 4a shown in Fig. 3 reads and executes a processing program stored in the memory 4b, thereby performing various functions of the image processing device 4 shown in Fig. 1. However, all or part of the various functions of the image processing device 4 may be configured to be performed by dedicated electronic circuits.

[0048] Moreover, although an example in which the processing program is stored in the memory 4b has been described here, the processing program (or at least a part of the processing program) may be stored in a portable storage medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disc) or a Blu-ray Disc, a storage medium such as a hard disk drive or an SSD (Solid State Drive), a storage medium on the cloud, etc. In this case, the processing program may be read from an external storage medium and stored in the memory 4b, and the processor 4a may execute the processing program.

[0049] FIG. 4 is a flowchart showing the processing of the image processing device 4 of the first embodiment.

[0050] When the power of the endoscope system 1 is turned on and the endoscope 2 starts capturing images and outputting image signals, the image processing device 4 executes the processing shown in FIG. 4 every time an image signal (endoscopic image information) for, for example, one frame is input.

[0051] The image processing device 4 acquires one or more latest endoscopic images via the input unit 11 (step S1).

[0052] The organ model generation unit 12 generates an organ model of the imaging target based on one or more acquired endoscopic images (step S2). To generate a three-dimensional organ model, it is preferable to use multiple endoscopic images captured at different positions, but it is also possible to generate a three-dimensional organ model from a single endoscopic image by using AI (artificial intelligence).

[0053] The organ model shape correcting unit 13 corrects the shape of an organ model that has been generated in the past (an existing organ model) based on the latest endoscopic image (step S3).

[0054] Here, if the endoscopic image acquired in step S2 is the first image captured by the endoscope 2 after it has started capturing images, no organ model has been generated, so the organ model shape correction unit 13 does not perform any correction, and stores the organ model acquired from the organ model generation unit 12 in memory 14.

[0055] Furthermore, if the endoscopic image acquired in step S2 is the second or subsequent image since the endoscope 2 started capturing images, the organ model shape correcting unit 13 acquires a new organ model generated based on the latest endoscopic image from the organ model generating unit 12, and also acquires the latest endoscopic image as needed. Furthermore, the organ model shape correcting unit 13 acquires an existing organ model from the memory 14. Then, the organ model shape correcting unit 13 determines whether correction of the existing organ model is necessary based on at least one of the latest endoscopic image and the new organ model. If it is determined that correction is necessary, the organ model shape correcting unit 13 corrects the existing organ model based on the new organ model. The organ model shape correcting unit 13 stores the corrected organ model in the memory 14.

[0056] The output unit 16 outputs information about the organ model corrected by the organ model shape correcting unit 13 to the monitor 8 (step S4). As a result, an organ model image is displayed on the monitor 8.

[0057] 4 is executed each time the latest endoscopic image is acquired, the user can check on the monitor 8 an organ model that is generated based on the latest endoscopic image information and matches the current state of the organ.

[0058] FIG. 5 is a diagram for explaining the generation of an organ model by the organ model generating unit 12 in the first embodiment.

[0059] The organ model generation unit 12 generates a 3D organ model using, for example, visual simultaneous localization and mapping (visual SLAM). The organ model generation unit 12 may estimate the position and orientation of the tip 2a1 of the insertion unit 2a by visual SLAM processing, or may use information input from the tip position detection device 5.

[0060] When generating a three-dimensional organ model, the organ model generation unit 12 first performs initialization. During initialization, it is assumed that the internal parameters of the endoscope 2 are known through calibration. As initialization, the organ model generation unit 12 estimates the self-position and three-dimensional position of the endoscope 2, for example, by SfM (Structure from Motion). Here, SLAM assumes real-time operation and takes as input, for example, a time-sequential video image, whereas SfM assumes as input a plurality of images that do not assume real-time operation.

[0061] Assume that the endoscope 2 moves after initialization. Fig. 5 shows how the position of the tip 2a1 of the insertion section 2a changes as time progresses from t(n), to t(n+1), to t(n+2).

[0062] At this time, the organ model generating unit 12 searches for corresponding points in multiple frames of endoscopic images.

[0063] Specifically, in the example of Figure 5, corresponding points are searched for in an endoscopic image IMG(n) captured at time t(n), an endoscopic image IMG(n+1) captured at time t(n+1) later than time t(n), and an endoscopic image IMG(n+2) captured at time t(n+2) later than time t(n+1).

[0064] For example, image point IP1 corresponding to point P1 in the subject's organ OBJ is searched for in endoscopic image IMG(n) and endoscopic image IMG(n+1), but not in endoscopic image IMG(n+2), and image point IP2 corresponding to point P2 in the subject's organ OBJ is not searched for in endoscopic image IMG(n), but is searched for in endoscopic image IMG(n+1) and endoscopic image IMG(n+2).

[0065] Next, the organ model generation unit 12 estimates (tracks) the position and orientation of the endoscope 2. The problem of estimating the position and orientation of the endoscope 2 (or more generally, the camera) is known as the PnP problem, which is a problem of estimating the position and orientation of the camera (the endoscope 2 in this embodiment) from the three-dimensional coordinates of n points in the world coordinate system and the image coordinates at which the n points are observed.

[0066] First, the organ model generating unit 12 estimates the posture of the endoscope 2 based on a plurality of points whose three-dimensional positions are known and the positions of the plurality of points on the image.

[0067] Next, the organ model generation unit 12 registers (maps) the points on a 3D map. That is, by finding correspondence between common points appearing in multiple endoscopic images obtained by the endoscope 2 whose posture is now known, the three-dimensional position of the points can be determined (triangulation).

[0068] Thereafter, the organ model generating unit 12 repeats the above-mentioned tracking and mapping to obtain the three-dimensional position of any point on the endoscopic image, and generates an organ model.

[0069] FIG. 6 is a diagram showing an overall image of an organ model and an example of an organ model that is generated, corrected, and displayed in the first embodiment.

[0070] Column A of Figure 6 shows an overall view of the organ model OM. Here, an intestinal tract, specifically an organ model of the colon, is shown as an example of the organ model OM, but is not limited to this. In Column A of Figure 6, IC indicates the cecum, AN indicates the anus, FCD indicates the hepatic flexure (right colic flexure), FCS indicates the splenic flexure (left colic flexure), and TC indicates the transverse colon.

[0071] Column B of Fig. 6 shows the state of the organ model OM generated when the insertion portion 2a of the endoscope 2 is moved from the cecum IC side, through the hepatic flexure FCD, toward the splenic flexure FCS side. The triangles in each of columns B to D of Fig. 6 indicate the viewing angle when observing the subject from the tip 2a1 of the insertion portion 2a.

[0072] In the left column of Figure 6B, an organ model OM near the cecum IC is generated.

[0073] In the center column B of FIG. 6, the insertion portion 2a passes through the hepatic flexure FCD and moves toward the transverse colon TC, so organ models OM are generated from the cecum IC to the hepatic flexure FCD and part of the transverse colon TC.

[0074] The right column B of Fig. 6 shows a modified example of the center column B of Fig. 6. It is assumed that there is no unobserved region in the part of the organ model OM shown by the dotted line in the right column B of Fig. 6. In this case, the existing organ model part shown by the dotted line may or may not be stored in memory 14, and may or may not be displayed on monitor 8.

[0075] In column C of Figure 6, the portion indicated by the dashed line represents the organ model OM1 before correction, and the portion indicated by the solid line represents the organ model OM2 after correction. When the corrected organ model OM2 is generated, the organ model OM1 before correction is deleted.

[0076] As shown in section D of FIG. 6, the monitor 8 displays the corrected organ model OM2.

[0077] 7 is a diagram for explaining an example of detecting a change in an organ model based on a feature point in the first embodiment. The feature point is one of the specific targets included in the endoscopic image information.

[0078] Column A1 in Fig. 7 shows the appearance of endoscopic image IMG(n) captured at time t(n). Column A2 in Fig. 7 shows the appearance of endoscopic image IMG(n+1) captured at time t(n+1). Multiple feature points SP(n) in endoscopic image IMG(n) and multiple feature points SP(n+1) in endoscopic image IMG(n+1) correspond to the same feature points (points having the same feature amount).

[0079] 7 shows the subject's organ OBJ(n) at time t(n) and the imaging area IA(n) of the endoscope 2. Column B2 of Fig. 7 shows the subject's organ OBJ(n+1) at time t(n+1) and the imaging area IA(n+1) of the endoscope 2. Comparing columns B1 and B2, the luminal diameter of the subject's organ OBJ(n+1) at time t(n+1) is larger than the luminal diameter of the subject's organ OBJ(n) at time t(n).

[0080] Column C1 in Fig. 7 shows the organ model OM(n) at time t(n) and the organ model area OMA(n) corresponding to the imaging area IA(n). Column C2 in Fig. 7 shows the organ model OM(n+1) at time t(n+1) and the organ model area OMA(n+1) corresponding to the imaging area IA(n+1) in comparison with the organ model OM(n).

[0081] Column D of Fig. 7 shows how, in the cross section CS shown in Column C2 of Fig. 7, multiple feature points SP(n) in the organ model area OMA(n) and multiple feature points SP(n+1) in the organ model area OMA(n+1) corresponding to the multiple feature points SP(n) are detected using feature amounts, brightness values, brightness gradient values, etc. Because the lumen diameter has expanded at time t(n+1), feature point SP(n+1) is a point where feature point SP(n) has moved so as to expand in the radial direction.

[0082] Column E of Figure 7 shows how, among the corresponding points at time t(n) and time t(n+1), point OMP(n) on the existing organ model OM(n) is deleted, and a new organ model OM(n+1) is generated using point OMP(n+1) obtained from the latest endoscopic image.

[0083] According to the first embodiment, the shape of the generated organ model is corrected, so that an organ model that matches the current shape of the organ can be generated. Furthermore, since the point OMP(n) on the existing organ model OM(n) is deleted, multiple organ models are not generated for the same region, and an appropriate organ model is obtained. [Second embodiment]

[0084] 8 to 20 show a second embodiment of the present invention, and Fig. 8 is a flowchart showing the processing of the image processing device 4 of the second embodiment. In the second embodiment, parts that are the same as those in the first embodiment are given the same reference numerals and their explanations will be omitted as appropriate, and the differences will be mainly described.

[0085] When the process shown in FIG. 8 starts, the image processing device 4 performs the process of step S1 to acquire one or more latest endoscopic images, and the organ model generation unit 12 estimates the position and posture of the tip 2a1 of the insertion unit 2a from the acquired endoscopic images (step S11).

[0086] Next, based on the estimated position and posture of the tip 2a1 of the insertion section 2a, the organ model generating unit 12 generates an organ model of the imaging target (step S2A).

[0087] Then, the organ model shape correcting unit 13 identifies a changed portion of the organ model (new organ model) of the current imaging target generated in step S2A relative to a previously generated organ model (existing organ model), and estimates the amount of change of the changed portion (step S12). The amount of change is estimated, for example, based on the amount of change of corresponding points (feature points, etc.) in each cross section of the existing organ model and the new organ model. For example, when the process shown in Fig. 4 is executed every time one frame of endoscopic image information is input, the amount of change is also calculated for each frame.

[0088] Next, the organ model shape correcting unit 13 corrects the shape of the existing organ model based on the estimated amount of change in the organ model (step S3A).

[0089] Thereafter, the process of step S4 is carried out, and the information on the corrected organ model is output to the monitor 8 or the like.

[0090] FIG. 9 is a chart showing how the amount of change in an organ model at different times is estimated in the second embodiment.

[0091] The overall image of the assumed organ model is shown in column A of Fig. 6. Furthermore, when there is no unobserved region, the organ model portion shown by the dotted line in column B of Fig. 6 may or may not be stored in memory 14, and may or may not be displayed on monitor 8, as in the first embodiment.

[0092] The left column A of Fig. 9 shows the organ model area OMA(n) for which the amount of change is to be detected in the organ model OM(n) at time t(n). The right column A of Fig. 9 shows the organ model area OMA(n+1) for which the amount of change is to be detected in the organ model OM(n+1) at time t(n+1).

[0093] The organ model area OMA(n+1) at time t(n+1) shown in the right column of B in Figure 9 has a lumen diameter that is expanded, for example, by an appropriate real number multiple, compared to the organ model area OMA(n) at time t(n) shown in the left column of B in Figure 9.

[0094] FIG. 10 is a diagram showing an example of correcting the shape of an organ model based on the amount of change in the second embodiment.

[0095] The organ model OM(n) at the past time t(n) is corrected to the organ model OM(n+1) at the current time t(n+1).

[0096] At this time, the unobserved area determination / correction unit 15 determines the unobserved area UOA(n+1) in the corrected organ model OM(n+1). For example, the unobserved area determination / correction unit 15 determines whether the unobserved area UOA(n) has become an observed area, and if it has not become an observed area, determines whether the unobserved area UOA(n) has moved to an unobserved area UOA(n+1), and also determines whether a new unobserved area UOA(n+1) has occurred.

[0097] The unobserved region determination and correction unit 15 then superimposes the generated unobserved region UOA(n+1) on the corrected organ model OM(n+1) and outputs the result to the output unit 16. As a result, an organ model image in which the position or shape has been corrected or in which the new unobserved region UOA(n+1) has been superimposed on the new organ model OM(n+1) is displayed on the monitor 8, for example, together with an endoscopic image. The unobserved region determination and correction unit 15 may also store the unobserved region UOA(n+1) in the memory 14.

[0098] FIG. 11 is a flowchart showing the process of estimating the amount of change in the organ model in step S12 of FIG. 8 in the second embodiment.

[0099] The organ model shape correcting unit 13 estimates the amount of change in the organ model by, for example, detecting the amount of expansion / contraction of the lumen diameter of the new organ model relative to the existing organ model (step S21), detecting the amount of rotation around the central axis (lumen axis) of the lumen (step S22), detecting the amount of expansion / contraction of the lumen along the lumen axis (step S23), and detecting the amount of movement of the lumen within the subject (step S24). Note that, although an example of the detection order is shown in Fig. 11, the detection order is not limited to the one shown.

[0100] FIG. 12 is a diagram showing how changes in an organ model include expansion, rotation, and movement in the second embodiment.

[0101] Column B of Figure 12 shows the state when a cross section CS perpendicular to the luminal axis of the organ model area OMA is taken in the organ models OM(n) and OM(n+1) at different times t(n) and t(n+1) shown in Column A of Figure 12.

[0102] The changes from multiple feature points SP(n) to multiple feature points SP(n+1) in the organ model OM include, for example, the expansion of the lumen diameter EXP, the rotation of the lumen around the lumen axis ROT, and the movement of the lumen MOV within the subject.

[0103] Fig. 13 is a flowchart showing the process of detecting the amount of enlargement / reduction in step S21 of Fig. 11 in the second embodiment. Fig. 14 is a chart for explaining the process of detecting the amount of enlargement / reduction in the second embodiment.

[0104] When the organ model shape correction unit 13 starts the process of detecting the amount of expansion / contraction shown in Figure 13, it detects the feature point SP(n) of the existing organ model OM(n) read from the memory 14 and the feature point SP(n+1) corresponding to the feature point SP(n) in the new organ model OM(n+1) generated by the organ model generation unit 12 based on the latest endoscopic image (step S31).

[0105] Next, the organ model shape correction unit 13 detects the distance D1 between two specific feature points SP(n) on a cross section CS(n) perpendicular to the luminal axis of the existing organ model OM(n), as shown in columns A1 and B1 of Figure 14 (step S32).

[0106] Furthermore, as shown in columns A2 and B2 of Figure 14, the organ model shape correction unit 13 detects the distance D2 between two feature points SP(n+1) on a cross section CS(n+1) perpendicular to the luminal axis of the new organ model OM(n+1) that correspond to the feature points SP(n) at which the distance D1 was detected (step S33).

[0107] Then, the organ model shape correcting unit 13 determines the ratio (D2 / D1) of the distance D2 to the distance D1 as the amount of expansion / contraction of the lumen diameter (step S34), and returns to the processing in Fig. 11. When the ratio (D2 / D1) is greater than 1, the lumen diameter is expanded, and when the ratio (D2 / D1) is less than 1, the lumen diameter is reduced.

[0108] Fig. 15 is a flowchart showing the process of detecting the amount of rotation in step S22 of Fig. 11 in the second embodiment. Fig. 16 is a chart for explaining the process of detecting the amount of rotation in the second embodiment.

[0109] The organ model shape correcting unit 13 performs image estimation, for example, by SLAM processing, based on the endoscopic images captured at different times and acquired from the input unit 11 via the organ model generating unit 12, and detects a first rotation amount θ1 of the tip 2a1 of the insertion unit 2a as shown in column A of Fig. 16 (step S41). For example, when the rotation amount of multiple pieces of endoscopic image information captured at different times and detected based on a specific target (such as a feature point) is -θ1, the organ model shape correcting unit 13 detects the rotation amount of the tip 2a1 as θ1.

[0110] Next, based on the output of the tip position detection device 5 obtained from the organ model generation unit 12, the organ model shape correction unit 13 detects the second rotation amount θ2 of the tip 2a1 of the insertion unit 2a between the two times at which the first rotation amount θ1 was detected, as shown in column B of Figure 16 (step S42).

[0111] Furthermore, the organ model shape correcting unit 13 detects the difference (θ1−θ2) between the first rotation amount θ1 and the second rotation amount θ2 as the rotation amount of the organ (step S43), and returns to the processing of FIG.

[0112] Fig. 17 is a flowchart showing the process of detecting the amount of expansion / contraction in step S23 of Fig. 11 in the second embodiment. Fig. 18 is a chart for explaining the process of detecting the amount of expansion / contraction in the second embodiment.

[0113] The organ model shape correction unit 13 selects two cross sections CS1(n) and CS2(n) that are perpendicular to the luminal axis and contain feature points in the existing organ model OM(n), as shown in column A of Figure 18, and detects the distance L1 between the two cross sections CS1(n) and CS2(n) (step S51).

[0114] Next, in the new organ model OM(n+1), as shown in column B of Figure 18, the organ model shape correction unit 13 searches for two cross sections CS1(n+1) and CS2(n+1) perpendicular to the luminal axis that contain feature points corresponding to the two cross sections CS1(n) and CS2(n) where the distance L1 was detected, and detects the distance L2 between the two cross sections CS1(n+1) and CS2(n+1) (step S52).

[0115] Then, the organ model shape correcting unit 13 determines the ratio (L2 / L1) of the distance L2 to the distance L1 as the amount of expansion or contraction of the lumen diameter (step S53), and returns to the processing in Fig. 11. When the ratio (L2 / L1) is greater than 1, the lumen length is expanded, and when the ratio (L2 / L1) is less than 1, the lumen length is shortened.

[0116] FIG. 19 is a flowchart showing the process of detecting the amount of movement in step S24 of FIG. 11 in the second embodiment.

[0117] The organ model shape correcting unit 13 corrects the existing organ model OM(n) based on the amount of expansion / contraction detected in step S21, the amount of rotation detected in step S22, and the amount of extension / contraction detected in step S23 (step S61).

[0118] Next, the organ model shape correcting unit 13 detects the same feature points of the organ model before and after correction (step S62). Here, the number of feature points to be detected may be one, but preferably multiple. Therefore, an example of detecting multiple feature points will be described below.

[0119] Next, the organ model shape correcting unit 13 calculates the average distance between the same feature points in the organ model before and after correction (step S63). Note that if the number of feature points to be detected in step S62 is one, the process of step S63 may be omitted, and the distance between the same feature points in the organ model before and after correction may be regarded as the average distance.

[0120] Then, the organ model shape correcting unit 13 determines whether the calculated average distance is equal to or greater than a predetermined threshold value (step S64).

[0121] If it is determined that the distance is equal to or greater than the threshold value, the average distance calculated in step S63 is detected as the amount of movement (step S65).

[0122] If it is determined in step S64 that the average distance is less than the threshold, the amount of movement is detected as 0 (step S66). That is, to prevent erroneous detection, if the average distance is less than the threshold, it is determined that there is no movement of the organ.

[0123] After the processing in step S65 or step S66 is performed, the process returns to the processing in FIG.

[0124] FIG. 20 is a chart for explaining an example of a method for correcting the shape of an organ model in step S3A of FIG. 8 in the second embodiment.

[0125] The shape of the organ model is corrected based on the amount of change in the organ model detected in step S12.

[0126] The correction range in this case can be, for example, a fixed distance range (part of the organ model including the changed site) along the luminal axis, with the region (changed site) that is the target of change detection as the base point. Here, the fixed distance along the luminal axis in front of the changed site and the fixed distance along the luminal axis behind the changed site may be the same distance or different distances.

[0127] The correction range may also be a range (part of the organ model including the changed part) with at least one of the landmark and the position of the tip 2a1 of the insertion unit 2a as its endpoint. Here, if the organ is the large intestine, landmarks include the cecum IC and anus AN, which are the ends of the organ, and the hepatic flexure FCD and splenic flexure FCS, which are the boundaries between the fixed part and the movable part. These landmarks vary depending on the organ and can be detected by AI-based part recognition. In this way, the organ model shape correction unit 13 can set a range within the organ model that is not subject to correction based on the type of organ. The organ model shape correction unit 13 then calculates the amount of change by referring to type information of the specific target according to the type of organ.

[0128] Alternatively, the organ model shape correcting unit 13 may set the entire organ model as the correction range.

[0129] The amount of correction within the correction range is controlled according to the distance along the lumen axis, for example, by setting the amount of correction to 1 in the region where the amount of change is detected and the amount of correction to 0 at the end points of the correction range.

[0130] Column A in FIG. 20 shows an example in which the correction range CTA is set with the position of the tip 2a1 of the insertion section 2a in the central part of the transverse colon TC and the hepatic curvature FCD as its two end points.

[0131] 20 shows an example in which the correction amount is controlled according to the distance along the lumen axis, with the correction amount for the region DA (specifically, a fold) that is the target of change detection in the correction range CTA being 1, and the correction amount for both end points being 0. That is, the organ model shape correcting unit 13 reduces the correction amount for the shape of the organ model as the distance from the region DA (specific target) that is the target of change detection increases.

[0132] According to the second embodiment, it is possible to achieve substantially the same effects as the first embodiment described above, and also to present the unobserved area UOA at the correct position, as shown in FIG.

[0133] Furthermore, the amount of change in the organ model can be detected by an appropriate method depending on whether it is enlargement / reduction, rotation, expansion / contraction, or movement.

[0134] Furthermore, by controlling the amount of correction according to the distance along the lumen axis, the corrected organ model can be made to have a reasonable shape. [Third embodiment]

[0135] 21 to 39 show a third embodiment of the present invention, and Fig. 21 is a flowchart showing the processing of the image processing device 4 of the third embodiment. In the third embodiment, parts that are the same as those of the first and second embodiments are given the same reference numerals and their explanations will be omitted as appropriate, and the differences will be mainly described.

[0136] When the process shown in FIG. 21 starts, step S1 is performed to acquire one or more latest endoscopic images, and step S11 is performed to estimate the position and posture of the tip 2a1 of the insertion portion 2a from the endoscopic images.

[0137] Next, processing in step S2A is performed to generate an organ model of the imaging target. At this time, as shown in column B of Fig. 6, if there is no unobserved region, the existing organ model portion indicated by the dotted line may or may not be stored in memory 14, and may or may not be displayed on monitor 8, as in the first and second embodiments.

[0138] Next, the organ model shape correcting unit 13 estimates the amount of change in the intestinal folds (specific target) in the organ model (step S12B). If the position or shape of the organ changes, it may become impossible to obtain correspondence between feature points between the existing organ model and the new organ model. In contrast, the number of folds in a hollow organ does not change even if the position or shape of the organ changes, and the order of the folds does not change either. Therefore, in this embodiment, the folds are used to reliably estimate the amount of change in the organ model.

[0139] Furthermore, the organ model shape correcting unit 13 corrects the shape of the existing organ model based on the estimated change in the folds in the organ model (step S3B).

[0140] Thereafter, the process of step S4 is carried out, and the information on the corrected organ model is output to the monitor 8 or the like.

[0141] Fig. 22 is a flowchart showing the process of estimating the amount of change in a fold in step S12B of Fig. 21 in the third embodiment. Fig. 23 is a chart for explaining the process of detecting whether or not a fold has passed in the third embodiment.

[0142] The organ model shape correction unit 13 acquires an endoscopic image IMG(n) at a past time t(n) as shown in column A1 of Fig. 23, and an endoscopic image IMG(n+1) at the latest time t(n+1) as shown in column B1 or C1 of Fig. 23. The endoscopic image IMG(n) is an image used to generate an existing three-dimensional organ model, and the endoscopic image IMG(n+1) is an image used to generate a new three-dimensional organ model.

[0143] The organ model shape correcting unit 13 searches for common feature points SP other than folds as tracking points in the endoscopic images IMG(n) and IMG(n+1) captured at different times.

[0144] Next, the organ model shape correcting unit 13 determines whether the distal end 2a1 of the insertion unit 2a has passed through a fold CP1 located distal to the feature point SP in the endoscopic image IMG(n). Since column A in Fig. 23 relates to time t(n), passage through the fold is not determined as shown in column A2.

[0145] Assume that the endoscopic image IMG(n+1) looks like the image shown in section B1 of Fig. 23. It can be seen that the fold CP2 located proximal to the characteristic point SP is a different fold from the fold CP1 located distal to the characteristic point SP. Therefore, the organ model shape correcting unit 13 determines that the organ has passed through the fold CP1, as shown in section B2 of Fig. 23.

[0146] On the other hand, suppose the endoscopic image IMG(n+1) looks like the one shown in section C1 of Fig. 23. In this case, it can be seen that the fold CP1 located on the distal side near the feature point SP is the same as the fold CP1 shown in section A1. Therefore, the organ model shape correcting unit 13 determines that the fold CP1 has not been passed through, as shown in section C2 of Fig. 23.

[0147] In this way, the organ model shape correcting unit 13 determines whether or not a fold has been passed (step S71).

[0148] Next, the organ model shape correcting unit 13 detects identical folds in the existing organ model and the new organ model based on whether or not a fold has passed through (step S72).

[0149] FIG. 24 is a diagram for explaining how folds in an endoscopic image are associated with folds in an organ model in the third embodiment.

[0150] As described above, even if the shape of an organ changes, the number of folds does not change, and the order of the folds does not change either. Therefore, by counting the number of folds, the folds in the endoscopic image can be associated with the folds in the organ model.

[0151] In FIG. 24, column A1 shows the fold CP1 in the endoscopic image IMG(n), and column A2 shows the folds CP1 and CP2 in the endoscopic image IMG(n+1).

[0152] 24, column B1 indicates that the tip 2a1 of the insertion part 2a is located at a position in the organ model OM(n) where only the fold CP1 can be observed. Column B2 indicates that the tip 2a1 of the insertion part 2a is located at a position in the organ model OM(n+1) where the folds CP1 and CP2 can be observed.

[0153] If the identical fold is detected in step S72, then the organ model shape correcting unit 13 detects the amount of change in the identical fold (step S73).

[0154] FIG. 25 is a chart for explaining how the amount of change in the same fold is detected in the third embodiment.

[0155] Column A1 in FIG. 25 shows how three folds CP1(n), CP2(n), and CP3(n) are detected in the organ model OM(n) at time t(n).

[0156] Column A2 of Figure 25 shows that three folds CP1(n+1), CP2(n+1), and CP3(n+1) corresponding to the three folds CP1(n), CP2(n), and CP3(n) in column A1 have been detected in the organ model OM(n+1) at time t(n+1).

[0157] Column B1 of FIG. 25 shows that of the three folds CP1(n), CP2(n), and CP3(n), the fold CP3(n) closest to the tip 2a1 of the insertion section 2a is selected to detect the amount of change.

[0158] Column B2 in FIG. 25 shows how the fold CP3(n+1) corresponding to the fold CP3(n) is selected to detect the amount of change.

[0159] The organ model shape correcting unit 13 detects the amount of change by comparing the fold CP3(n) shown in section B1 of FIG. 25 with the fold CP3(n+1) shown in section B2 of FIG.

[0160] FIG. 26 is a flowchart showing the process of detecting the amount of change in the same fold in step S73 of FIG. 22 in the third embodiment.

[0161] The organ model shape correcting unit 13 detects the amount of change in the identical fold by, for example, detecting the amount of expansion or contraction of the diameter of the identical fold in the new organ model relative to the fold in the existing organ model (step S81), detecting the amount of rotation (step S82), detecting the amount of expansion or contraction between the two identical folds (step S83), and detecting the amount of movement of the fold within the subject (step S84). Note that, although an example of the detection order is shown in Fig. 26, the detection order is not limited to the one shown.

[0162] The process of detecting the amount of diameter expansion / contraction in step S81 in Fig. 26 may be performed by detecting the ratio D2 / D1 of the distances between corresponding feature points on the same fold, instead of calculating the ratio D2 / D1 of the distances between feature points on a cross section perpendicular to the lumen axis in the description with reference to Fig. 13 and Fig. 14. Alternatively, the description with reference to Fig. 13 and Fig. 14 may be applied as is.

[0163] Also, Fig. 27 is a flowchart showing another example of processing for detecting the amount of expansion or contraction of the diameter in step S81 of Fig. 26 in the third embodiment. Fig. 28 is a chart for explaining another example of processing for detecting the amount of expansion or contraction of the diameter in the third embodiment.

[0164] When the process shown in Fig. 27 starts, cross sections CS(n) and CS(n+1) perpendicular to the lumen axis are set in the existing organ model and the new organ model so that they include the same feature points on the corresponding folds. Furthermore, as shown in Fig. 28, a line segment AB of the same length Dx is set in each of the cross sections CS(n) and CS(n+1) (step S91). At this time, at least one end point of the line segment AB (for example, end point A) may be set to be the same feature point on the corresponding fold.

[0165] Next, the distance between the two points where the perpendicular bisector of the line segment AB intersects with the cross sections CS(n) and CS(n+1) is detected as diameters d(n) and d(n+1), respectively (step S92).

[0166] Then, the diameter ratio d(n+1) / d(n) is detected as the amount of expansion / contraction of the lumen diameter in the fold (step S93), and the process returns to the process of FIG.

[0167] Fig. 29 is a graph for explaining an example of a method for correcting the amount of expansion / contraction of the diameter of an organ model in the third embodiment. Fig. 30 is a diagram for explaining an example of correcting the amount of expansion / contraction of the diameter of an organ model within a correction range in the third embodiment.

[0168] As described above in relation to Figure 20, the correction of the amount of expansion / contraction of the diameter may also be performed within a correction range that includes the fold for which the amount of expansion / contraction has been detected. As described above, the correction range may be a fixed distance range before or after the fold, between two landmarks that include the fold, or between a landmark that includes the fold and the position of the distal end 2a1 of the insertion unit 2a. Also, as described above, the entire organ model may be the correction range.

[0169] FIG. 29 shows a graph in which the diameter change rate is set so that, within the correction range along the lumen axis CA (see FIG. 30), the diameter is changed by a ratio d(n+1) / d(n) at the position of the fold where the amount of diameter expansion / contraction is detected, and the change in the diameter ratio is 1 at both end points of the correction range. As a result, for example, at the midpoint between the position of the fold where the amount of diameter expansion / contraction is detected and the end points of the correction range, the diameter change rate is {1+([{d(n+1) / d(n)}-1] / 2)}. Note that the graph shown in FIG. 29 is just an example, and the diameter change rate may be configured as a curve.

[0170] By correcting the organ model radially in the radial direction centered on the lumen axis according to the amount of expansion / contraction shown in Fig. 29, the hatched correction range is corrected from organ model OM(n) to organ model OM(n+1) as shown in Fig. 30. Note that for the range not subject to correction, there is no change between organ model OM(n) and organ model OM(n+1).

[0171] 26. The process of detecting the amount of rotation in step S82 can be the same as that described with reference to Figures 15 and 16. In this case, in step S41, the first amount of rotation θ1 based on the endoscopic image may be detected with attention focused on the folds in the endoscopic image.

[0172] FIG. 31 is a graph for explaining an example of a method for correcting the rotation amount of an organ model in the third embodiment.

[0173] As with the diameter, the rotation of a hollow organ around its luminal axis can also be corrected by setting a part or the entire organ model as the correction range.

[0174] When correcting the amount of rotation, first, the lumen axis of the organ model within the correction range is estimated.

[0175] Next, in the correction range along the lumen axis, the amount of rotation is changed by (θ1-θ2) at the position of the fold where the amount of rotation was detected, as shown in the graph of Fig. 31, and the amount of rotation is changed so that the amount of rotation at both ends of the correction range becomes 0. Note that the graph shown in Fig. 31 is an example, and the change in the amount of rotation may be configured as a curve.

[0176] FIG. 32 is a chart for explaining the process of detecting the amount of expansion / contraction in step S83 of FIG. 26 in the third embodiment.

[0177] In FIG. 32, column A1 shows an endoscopic image IMG(n) captured at time t(n), and column A2 shows an endoscopic image IMG(n+1) captured at time t(n+1).

[0178] The organ model shape correcting unit 13 detects two identical folds in the endoscopic image IMG(n) and the endoscopic image IMG(n+1) using, for example, AI. In the example shown in column A of Fig. 32, a first fold CP1(n) and a second fold CP2(n) are detected in the endoscopic image IMG(n), and a first fold CP1(n+1) and a second fold CP2(n+1) are detected in the endoscopic image IMG(n+1).

[0179] If the distance between folds has changed between a past time t(n) and the current time t(n+1), the amount of expansion / contraction is detected based on the depth value difference of each fold using, for example, SLAM. Here, suppose that the distance between folds, L1, at time t(n) has been detected to have changed to distance L2 at time t(n+1).

[0180] Then, the shape of the organ model is corrected so that the distance L1 between the folds in the existing organ model OM(n) as shown in column B1 of Figure 32 becomes the distance L2 shown in column B2 of Figure 32 in the new organ model OM(n+1).

[0181] Furthermore, with regard to the expansion and contraction of the hollow organ in the luminal axis direction, the amount of expansion and contraction can be corrected within an appropriate correction range that includes the fold for which the amount of expansion and contraction is detected, as described above. As an example, the correction range may be between a landmark located in the opposite direction from the fold that the distal end 2a1 of the insertion section 2a last passed and a fold that is closest to the distal end 2a1 and has not yet passed through.

[0182] Fig. 33 is a graph for explaining an example of a method for correcting the amount of expansion / contraction of an organ model in the third embodiment. Fig. 34 is a diagram for explaining an example of correcting the amount of expansion / contraction of an organ model in the third embodiment.

[0183] The organ model shape correcting unit 13 first determines which portion of the organ model OM(n) to correct based on the moving direction of the tip 2a1 of the insertion unit 2a. For example, in Fig. 34, it is assumed that the tip 2a1 of the insertion unit 2a is moving from the splenic flexure FCS toward the hepatic flexure FCD. In this case, the organ model shape correcting unit 13 sets the hepatic flexure FCD side (cecum IC side) of the transverse colon TC of the organ model OM(n) as the correction range, as shown by hatching.

[0184] Next, the organ model shape correcting unit 13 calculates the length x along the lumen axis from the hepatic curvature FCD, which is a landmark in the existing organ model OM(n), to the fold CP2(n) for which the amount of change has been detected.

[0185] Next, the organ model shape correcting unit 13 fixes the landmark, the hepatic curvature FCD, as a fixed position and calculates the amount of expansion or contraction from the fixed position to the fold CP2(n+1) at time t(n+1), for example the shortened length y in this case, based on the change in the distance between the folds from L1 to L2. This reveals that the length along the lumen axis from the landmark to the fold CP2(n+1) has become (xy).

[0186] In this case, the expansion / contraction ratio of the organ model is (xy) / x. As shown in FIG. 33, the organ model shape correcting unit 13 corrects the shape of the organ model OM(n) so that the expansion / contraction ratio of each point on the lumen axis becomes (xy) / x, with the landmarks as fixed positions. To give a specific example, when x=10 and y=2, the expansion / contraction ratio is (10-2) / 10=0.8. Therefore, position 5 on the lumen axis, with the fixed position as the origin, becomes 5×0.8=4 after correction. Note that the graph shown in FIG. 33 is an example, and the change in expansion / contraction ratio may be represented by a curve.

[0187] By performing such a correction, an organ model OM(n+1) as shown in FIG. 34 is calculated.

[0188] FIG. 35 is a flowchart for explaining the process of detecting the amount of movement in step S84 of FIG. 26 in the third embodiment.

[0189] 35 starts, the position of the tip 2a1 of the insertion section 2a when the same folds of the existing organ model and the new organ model are photographed and detected is estimated (step S101). The estimation of the position of the tip 2a1 may be performed based on the endoscopic image as described above, or may be performed based on information input from the tip position detection device 5.

[0190] Next, it is determined whether the difference between the position of the tip 2a1 when the folds are photographed in the existing organ model and the position of the tip 2a1 when the folds are photographed in the new organ model is equal to or greater than a predetermined distance (step S102).

[0191] If the distance is equal to or greater than the predetermined distance, it is determined that the organ has moved, and the distance is detected as the amount of movement (step S103).

[0192] Also, if the distance is less than the predetermined distance in step S102, it is determined that the organ has not moved, and the amount of movement is detected as 0 (step S104). Here, the organ is determined to have moved only if the distance is equal to or greater than the predetermined distance in order to prevent erroneous determination due to calculation errors. After the processing of step S103 or step S104 is performed, the process returns to the processing of FIG. 26.

[0193] FIG. 36 is a diagram showing an example of detecting identical folds between an existing organ model and a new organ model in order to determine the movement of an organ in the third embodiment.

[0194] In Figure 36, the dotted line indicates the position of the subject's organ OBJ(n) before movement (at the time of the first imaging at time t(n)), and the solid line indicates the position of the subject's organ OBJ(n+1) after movement (at the time of the second imaging at time t(n+1)). The same fold is detected in the first and second imaging based on the position of the fold from the splenic flexure FCS, which is a landmark.

[0195] From time t(n) to time t(n+1), counting from the splenic flexure FCS, the first fold CP1(n) moves to the position of the first fold CP1(n+1), the second fold CP2(n) moves to the position of the second fold CP2(n+1), and the third fold CP3(n) moves to the position of the third fold CP3(n+1). The distal end 2a1 of the insertion section 2a passes through the first fold CP1 and the second fold CP2 and is positioned opposite the third fold CP3, and it is the third fold CP3 that is observed closest in the endoscopic image.

[0196] When viewed from the distal end 2a1 of the insertion section 2a, unobserved areas UOA(n) and UOA(n+1) exist in the vicinity of the distal side of the third folds CP3(n) and CP3(n+1). The unobserved area determination and correction unit 15 calculates the correct position of the unobserved area UOA at each of times t(n) and t(n+1), and displays it on the monitor 8 or the like.

[0197] FIG. 37 is a chart for explaining a method for correcting the shape of an organ model in accordance with the movement of the organ in the third embodiment.

[0198] When correcting the shape of the organ model OM, the organ model shape correcting unit 13 calculates the positions of the folds CP(n) and CP(n+1) based on the position of the tip 2a1 of the insertion unit 2a estimated in step S101.

[0199] Next, the organ model shape correcting unit 13 generates straight lines connecting the centers of the folds CP(n) and CP(n+1) whose movement has been detected and the central positions of the landmarks before and after the folds CP(n) and CP(n+1), as shown in column A of Fig. 37. Specifically, for example, straight lines SL1(n) and SL1(n+1) connecting the center position of the hepatic flexure FCD and the centers of the folds CP(n) and CP(n+1), respectively, and straight lines SL2(n) and SL2(n+1) connecting the center position of the splenic flexure FCS and the centers of the folds CP(n) and CP(n+1), respectively, are generated.

[0200] Next, the organ model shape correcting unit 13 calculates the distance from a predetermined point on the line SL1(n) to a predetermined point on the line SL1(n+1) as the movement amount of each point, as shown in column B of Fig. 37. One method of setting each point is to set each point at the intersection of the lines SL1(n) and SL1(n+1) with a plane perpendicular to a line connecting the center position of the hepatic flexure FCD and the center position of the splenic flexure FCS.

[0201] Fig. 38 is a graph illustrating a method for correcting the shape of an organ model in accordance with organ movement in the third embodiment. As shown in Fig. 38, the correction range is between the hepatic flexure FCD and the splenic flexure FCS. Let X denote the amount of movement at the fold CP where a change has been detected. The amount of movement monotonically increases as the position on the line moves from the hepatic flexure FCD to the fold CP, and monotonically decreases as the position on the line moves from the fold CP to the splenic flexure FCS. The graph shown in Fig. 38 is merely an example, and the change in the amount of movement may be represented by a curve.

[0202] Then, the organ model shape correction unit 13 calculates a corrected organ model OM(n+1) by correcting the distance between the hepatic flexure FCD and the splenic flexure FCS of the organ model OM(n) according to the calculated distance, as shown in column C of Figure 37.

[0203] FIG. 39 is a diagram showing a display example of an organ model and an unobserved region in the third embodiment.

[0204] After correcting the shape of the organ model OM(n) and calculating the organ model OM(n+1), the corrected organ model OM(n+1) is displayed on the monitor 8.

[0205] Column A in Figure 39 shows the corrected organ model OM(n+1) displayed on the monitor 8. The organ model OM(n+1) displays the range from the tip 2a1 of the insertion unit 2a through the hepatic flexure FCD to the cecum IC. At this time, the region for which the organ model OM(n+1) has not been generated may be displayed as an unobserved region UOA(n+1). Furthermore, the organ model OM(n+1) displays the position and field of view of the tip 2a1 of the insertion unit 2a, for example, using a triangular mark. When a triangular mark is used, one vertex of the triangular mark indicates the position of the tip 2a1, and the field of view direction and field of view range are indicated by two sides surrounding the vertex. However, other marks may of course be used.

[0206] Column B of Figure 39 shows an example in which only the organ model OM(n+1) after the position of the unobserved area UOA(n+1) in the direction of movement of the tip 2a1 of the insertion section 2a is displayed. For example, suppose the tip 2a1 moves from the cecum IC to the hepatic flexure FCD, and then moves from the hepatic flexure FCD toward the splenic flexure FCS. In this case, the organ model OM(n+1) before the position of the unobserved area UOA(n+1) does not need to be displayed, as indicated by the dotted line.

[0207] 39 shows an example in which an endoscopic image IMG(n+1) is displayed on the monitor 8, and an arrow AR(n+1) indicates the direction from the distal end 2a1 to the unobserved region. In this case, the length (or thickness, color, etc.) of the arrow AR(n+1) may be used to further indicate the distance from the distal end 2a1 to the unobserved region.

[0208] Furthermore, if the moving speed of the tip 2a1 of the insertion section 2a is greater than a predetermined threshold, the folds CP may not be clearly imaged. In this embodiment, the folds CP are used to correct the organ model OM. Therefore, if it is necessary to clearly image the folds CP, the moving speed of the tip 2a1 may be displayed on the monitor 8, and if the moving speed is greater than or equal to the threshold, an alert may be issued by display, audio, or the like.

[0209] The third embodiment achieves substantially the same effects as the first and second embodiments. Taking advantage of the fact that changes in an organ do not affect the order or number of folds, the third embodiment can detect identical folds by determining whether or not a fold has passed through the organ. Detecting the presence or absence of changes in identical folds and the amount of change allows accurate estimation of changes in the shape of an organ. Furthermore, by displaying the corrected direction and position of unobserved regions together with the corrected organ model or the latest endoscopic image, the unobserved regions are correctly displayed, preventing oversight of lesions.

[0210] In each of the above-described embodiments, the shape of the organ model may be corrected based on information obtained from the endoscope 2 or peripheral devices of the endoscope 2. Alternatively, the shape of the organ model may be corrected by combining information obtained from the endoscope 2 or peripheral devices of the endoscope 2 with endoscopic image information.

[0211] For example, when air is sent into an organ from the endoscope 2, the organ expands and changes shape. Therefore, the amount of expansion of the organ may be estimated based on the amount of air sent into the organ, and the shape of the organ model may be corrected.

[0212] Furthermore, although the present invention has been described above mainly in terms of an image processing device for an endoscopic system, the present invention is not limited to this, and the present invention may also be an image processing method that performs the same function as an image processing device, a program for causing a computer to perform processing similar to that of an image processing device, a non-transitory recording medium (non-volatile storage medium) that can be read by a computer and that records the program, etc.

[0213] Furthermore, the present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various aspects of the invention can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all of the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. In this way, it goes without saying that various modifications and applications are possible within the scope of the gist of the invention.

Claims

1. a processor having hardware; The processor: After acquiring endoscopic image information from the endoscope and generating an organ model, Continue acquiring the endoscopic image information; Identifying a changed part in the generated organ model based on the latest endoscopic image information; Estimating a change amount of the changed portion, which is at least one of an amount of expansion / contraction, an amount of rotation, an amount of extension / contraction, and an amount of movement; correcting a shape of at least a part of the organ model including the changed portion based on the estimated change amount of the changed portion; An image processing device that outputs information about the corrected organ model.

2. The image processing device according to claim 1 , wherein the processor generates a new organ model based on the latest endoscopic image information, and corrects the shape of the previously generated organ model based on the new organ model.

3. The image processing device according to claim 1 , wherein the processor corrects the entire organ model.

4. The image processing device according to claim 1 , wherein the processor sets a range in the organ model that is not to be corrected based on the type of organ.

5. The processor: calculating a change amount of a specific object from a plurality of pieces of endoscopic image information captured at different times; The image processing apparatus according to claim 1 , wherein the shape of the organ model is corrected based on the amount of change.

6. The image processing device according to claim 5 , wherein the processor calculates at least one of the amount of enlargement / reduction and the amount of expansion / contraction based on the amount of change in distance between a plurality of specific targets.

7. The processor: calculating a first rotation amount of the plurality of pieces of endoscopic image information based on the specific target; acquiring a second rotation amount of the distal end of the insertion section of the endoscope from an external position detection sensor; 6. The image processing apparatus according to claim 5, wherein the rotation amount is calculated based on the first rotation amount and the second rotation amount.

8. The processor: acquiring the endoscopic image information frame by frame from the endoscope; 6. The image processing apparatus according to claim 5, wherein the amount of change is calculated for each frame.

9. The image processing device described in Claim 5, characterized in that the processor reduces the amount of correction to the shape of the organ model as the distance from the specific object increases.

10. The organ for which the organ model is to be generated is the intestinal tract, The image processing apparatus according to claim 5 , wherein the specific object is a fold of the intestinal tract.

11. A computer comprising: After acquiring endoscopic image information from the endoscope and generating an organ model, Continue acquiring the endoscopic image information; Identifying a changed part in the generated organ model based on the latest endoscopic image information; estimating a change amount, which is at least one of an amount of enlargement / reduction, an amount of rotation, an amount of extension / contraction, and an amount of movement, of the changed portion; correcting a shape of at least a part of the organ model including the changed part based on the estimated amount of change in the changed part; A program for outputting information about the corrected organ model.

Images