Endoscope system, lumen structure calculating apparatus, manufacturing method

Abstract

An endoscope system (1) includes: an insertion section (2b) inserted into a lumen as an object; an imaging element (15) provided at a distal end (11) of the insertion section (2b) and configured to capture the object to obtain a captured image; a position and posture detection section (25) configured to detect a three-dimensional arrangement of the imaging element (15); and a processor (21) configured to calculate a three-dimensional structure of the lumen based on the captured image and the three-dimensional arrangement.

Description

Technical Field

[0001] This invention relates to an endoscope system, a lumen structure calculation device, a method for generating lumen structure information, and a computer program product. Background Technology

[0002] Endoscopes are widely used in the medical and industrial fields. For example, in the medical field, doctors insert the endoscope into the body of the patient and observe the endoscopic images displayed on a display device to examine the body, thereby enabling endoscopic examinations.

[0003] During endoscopic examinations, unobserved areas cannot guarantee a reliable result. Therefore, to identify unobserved areas during colonoscopy, techniques have been developed, such as the Automated visibility map of the internal colon surface from colonoscopy video proposed by Mohammad Ali Armin and five other authors (International Journal of Computer Assisted Radiology and Surgery Vol. 11, No. 9, pp. 1599-1610, 4 Aug, 2016), which constructs a 3D model of the intestinal structure based on computational estimation from images captured by an endoscope.

[0004] In this proposal, the 3D model structure is calculated solely from endoscopic images. To calculate the 3D model structure, the coordinates of multiple feature points on the inner wall of the intestine in the endoscopic image are determined, and the 3D model structure is constructed by connecting these multiple feature points.

[0005] However, because organs such as the large intestine are mobile, if image defects occur due to the movement of organs during the creation of a 3D model structure, the positions of feature points become unclear, making it impossible to calculate a 3D model structure that connects multiple points.

[0006] The present invention was made in view of the above-mentioned problems, and aims to provide an endoscope system, a lumen structure calculation device, a method for generating lumen structure information, and a computer program product that can calculate a 3D model structure of the lumen based on the position of each feature point even when image defects caused by organ movement occur. Summary of the Invention

[0007] means for solving problems

[0008] One aspect of the present invention provides a lumen structure calculation device comprising: an input unit that acquires multiple time-point images and a 3D configuration, the multiple time-point images being acquired by a camera unit and including the same part of a subject, the 3D configuration including at least some information about the position and orientation of the camera unit, the camera unit being disposed in an insertion unit inserted into a lumen that is the subject; and a lumen structure calculation unit that calculates the position of the same part based on the multiple time-point images and the 3D configuration, thereby calculating the 3D structure of the lumen.

[0009] An endoscope system according to one aspect of the present invention includes: an insertion unit inserted into a lumen serving as a subject; an imaging unit disposed in the insertion unit and acquiring images of the subject at multiple time points including the same part of the subject; a detection device for detecting a 3D configuration including at least a portion of information about the position and orientation of the imaging unit; and a lumen structure calculation unit for calculating the position of the same part based on the images at the multiple time points and the 3D configuration.

[0010] One method for generating lumen structure information according to the present invention comprises the following steps: acquiring multiple time-point images of a subject, the multiple time-point images of the subject being acquired by a camera unit and containing the same part of the subject, the camera unit being disposed in an insertion part inserted into the lumen of the subject; obtaining a 3D configuration, the 3D configuration including at least one of the position and orientation of the insertion part; and calculating the position of the same part based on the multiple time-point images and the 3D configuration, thereby calculating the 3D structure of the lumen. Attached Figure Description

[0011] Figure 1 This is a structural diagram of an endoscope system according to an embodiment of the present invention.

[0012] Figure 2 This is a perspective view of an endoscope according to an embodiment of the present invention.

[0013] Figure 3 This is a block diagram illustrating the structure of an endoscope system according to an embodiment of the present invention.

[0014] Figure 4 This is a flowchart illustrating an example of the calculation process for the lumen structure in the lumen structure calculation program according to an embodiment of the present invention.

[0015] Figure 5 This is a diagram illustrating the patient's state when setting the initial value of the front end of the insertion section according to an embodiment of the present invention.

[0016] Figure 6This is a diagram illustrating an example of a schematic partial model of the large intestine according to an embodiment of the present invention.

[0017] Figure 7 This is a diagram illustrating an example of a schematic partial model of the large intestine according to an embodiment of the present invention.

[0018] Figure 8 This is a diagram illustrating a method for correcting the positional offset of two partial models according to an embodiment of the present invention.

[0019] Figure 9 This is a diagram illustrating an example of an image displayed on a monitor according to an embodiment of the present invention.

[0020] Figure 10 This is a diagram illustrating an example of an image of structural information displayed in a structural information display area according to an embodiment of the present invention.

[0021] Figure 11 This is a diagram illustrating an example of an image representing a trajectory displayed on the front end of a monitor, according to an embodiment of the present invention.

[0022] Figure 12 This is a diagram illustrating an example of how the trajectory of the tip of the insertion part is displayed on a monitor screen when two doctors are performing an operation according to an embodiment of the present invention.

[0023] Figure 13 This is a flowchart of a method for calculating the position of each feature point in 3D space using bundle adjustment, according to an embodiment of the present invention.

[0024] Figure 14 This is a schematic diagram illustrating the relationship between feature points on multiple endoscopic images obtained in succession according to embodiments of the present invention and the position and orientation of the anterior end.

[0025] Figure 15 This is a schematic diagram illustrating the relationship between the position and orientation of feature points, the anterior end portion, and the orientation of multiple endoscopic images in embodiments of the present invention.

[0026] Figure 16 This is a perspective view of the front end portion of the front end portion of the insertion portion of the stereo camera according to an embodiment of the present invention.

[0027] Figure 17 This is a top view of the front end face of the front end of the insertion part having multiple lighting windows according to an embodiment of the present invention.

[0028] Figure 18 This is a perspective view of the front end portion of the front end portion of the insertion portion having a distance sensor according to an embodiment of the present invention.

[0029] Figure 19 This is a flowchart of a method for calculating the 3D position of the inner wall of a tube using a distance sensor, according to an embodiment of the present invention.

[0030] Figure 20 This is a diagram illustrating an example of a partially calculated lumen structure for explaining embodiments of the present invention.

[0031] Figure 21 This is a diagram illustrating a method for detecting the position and orientation of the anterior end using an endoscope with a shape sensor and a sensor for detecting insertion and torsion, according to an embodiment of the present invention.

[0032] Figure 22 This is a perspective view of an endoscope according to an embodiment of the present invention, in which multiple magnetic sensors are arranged in the insertion section.

[0033] Figure 23 This is a diagram illustrating an example of a monitor display when a hidden portion exists in an embodiment of the present invention.

[0034] Figure 24 This is a flowchart illustrating an example of a process for informing about unobserved areas based on the brightness values ​​of an image, according to an embodiment of the present invention.

[0035] Figure 25 This is a flowchart illustrating an example of a process for informing about unobserved areas based on distance images from a distance sensor, according to an embodiment of the present invention. Detailed Implementation

[0036] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0037] (structure)

[0038] Figure 1 This is a structural diagram of the endoscope system of this embodiment. Figure 2 It is a 3D view of an endoscope. Figure 3 This is a block diagram showing the structure of the endoscope system 1. The endoscope system 1 comprises: an endoscope 2, an image processing device 3, a light source device 4, a lumen structure detection device 5, a monitor 6 as a display device, and a magnetic field generating device 7. The endoscope system 1 is capable of observation under normal light using white illumination. A physician can use the endoscope system 1 to perform endoscopic examinations of the large intestine of a patient Pa lying supine on a bed 8. Furthermore, the lumen structure detection device 5 is a lumen structure calculation device for calculating the lumen structure of the large intestine of patient Pa.

[0039] Furthermore, the magnetic field generating device 7 is a separate device, but it can also be included in the lumen structure detection device 5.

[0040] The endoscope 2 includes an operating section 2a, a flexible insertion section 2b, and a universal cable 2c containing signal lines, etc. The endoscope 2 is a tubular insertion device into a body cavity through which the tubular insertion section 2b is inserted. A connector is provided at the front end of the universal cable 2c, through which the endoscope 2 is detachably connected to the light source device 4 and the image processing device 3. Here, the endoscope 2 is an endoscope capable of being inserted into the large intestine. Furthermore, although not shown, a light guide is inserted into the universal cable 2c, and the endoscope 2 is configured to allow illumination light from the light source device 4 to be emitted from the front end of the insertion section 2b through the light guide.

[0041] like Figure 2 As shown, the insertion part 2b has a front end portion 11, a bendable curved portion 12, and a flexible tube portion 13 extending from its front end towards its base end. The insertion part 2b is inserted into the lumen of the patient Pa, which is the subject. The base end portion of the front end portion 11 is connected to the front end portion of the curved portion 12, and the base end portion of the curved portion 12 is connected to the front end portion of the flexible tube portion 13. The front end portion 11 is the front end portion of the insertion part 2b, i.e., the front end portion of the endoscope 2, and is a relatively rigid front end portion.

[0042] The bending section 12 can be bent in the desired direction according to the operation of the bending operation components 14 (left-right bending operation knob 14a and up-down bending operation knob 14b) provided on the operation section 2a. When the bending section 12 is bent, changing the position and orientation of the front end 11, and capturing the observation area in the subject's body within the field of view, illumination light is shone on the observation area. The bending section 12 has multiple bending blocks (not shown) connected along the length axis of the insertion section 2b. Therefore, while the doctor presses the insertion section 2b into or pulls it out of the large intestine, the bending section 12 can be bent in various directions, thereby enabling a comprehensive observation of the patient Pa's large intestine.

[0043] The left-right bending operation knob 14a and the up-down bending operation knob 14b are used to pull and loosen the operating line inserted into the insertion part 2b in order to bend the bending part 12. The bending operation component 14 also has a fixing knob 14c to fix the position of the bent part 12. In addition, the operation part 2a is equipped with various operation buttons such as a release button and an air / water supply button, in addition to the bending operation component 14.

[0044] The flexible tube section 13 is flexible and bends according to external force. The flexible tube section 13 is a tubular component extending from the operating section 2a.

[0045] In addition, such as Figure 3As shown, an imaging element 15, serving as an imaging device, is provided at the front end 11 of the insertion part 2b. The observation area inside the large intestine, illuminated by the illumination light from the light source device 4, is captured by the imaging element 15. That is, the imaging element 15, located at the front end 11 of the insertion part 2b, constitutes an imaging unit that captures images of the subject at multiple time points to obtain images at multiple time points. The imaging signal obtained by the imaging element 15 is provided to the image processing device 3 via the signal line in the universal cable 2c.

[0046] Image processing device 3 is a video processor that performs prescribed image processing on the received imaging signal and generates an endoscopic image. The image signal of the generated endoscopic image is output from image processing device 3 to monitor 6, where the real-time endoscopic image is displayed. The doctor performing the examination inserts the front end 11 of insertion part 2b into the anus of patient Pa, allowing observation of the large intestine of patient Pa.

[0047] Furthermore, a magnetic sensor 16 is disposed at the front end portion 11 of the insertion portion 2b. Specifically, the magnetic sensor 16 is disposed near the imaging element 15 at the front end portion 11 and is a detection device for detecting the position and orientation of the viewpoint of the imaging element 15. The magnetic sensor 16 has two coils 16a and 16b. For example, the two central axes of the two cylindrical coils 16a and 16b are orthogonal to each other. Therefore, the magnetic sensor 16 is a 6-axis sensor that detects the position coordinates and orientation (i.e., Euler angles) of the front end portion 11. The signal line 2e of the magnetic sensor 16 extends from the endoscope 2 and is connected to the lumen structure detection device 5.

[0048] That is, as a detection device for detecting the position and posture of the viewpoint of the camera element 15, the magnetic sensor 16 can be disposed inside the front end portion 11 of the insertion portion 2b without affecting the size and performance of the endoscope 2 having the insertion portion 2.

[0049] A magnetic field generating device 7 generates a predetermined magnetic field, and a magnetic sensor 16 detects the magnetic field generated by the magnetic field generating device 7. The magnetic field generating device 7 is connected to the lumen structure detection device 5 via a signal line 2e. The detection signal of the magnetic field is provided from the endoscope 2 to the lumen structure detection device 5 via the signal line 2e. Alternatively, a magnetic field generating element can be provided at the front end 11 instead of the magnetic sensor 16, and a magnetic sensor can be provided outside the patient Pa instead of the magnetic field generating device 7, thereby detecting the position and orientation of the front end 11. Here, the position and orientation of the front end 11 are detected in real time using the magnetic sensor 16; in other words, the position and orientation of the viewpoint of the endoscopic image acquired by the imaging element 15 are detected in real time.

[0050] When the doctor presses the release button located on the operation unit 2a, the release button operation signal is input to the image processing device 3, and the endoscopic image at the time of pressing the release button is recorded in a recorder (not shown).

[0051] like Figure 3 As shown, the lumen structure detection device 5 includes: a processor 21, a storage device 22, a display interface (hereinafter referred to as display I / F) 23, an image acquisition unit 24, a position and posture detection unit 25, a drive circuit 26, and an input interface (hereinafter referred to as input I / F) 27. The processor 21, storage device 22, display I / F 23, image acquisition unit 24, position and posture detection unit 25, drive circuit 26, and input I / F 27 are interconnected via a bus 28.

[0052] The processor 21 has a central processing unit (hereinafter referred to as CPU) 21a and a memory 21b, and is a control unit that controls the processing of various parts within the cavity structure detection device 5. The memory 21b is a storage unit including ROM, RAM, etc. The ROM stores various processing programs and various data executed by the CPU 21a. The CPU 21a can read and execute various programs stored in the ROM and the storage device 22.

[0053] The storage device 22 stores the lumen structure calculation program LSP, which will be described later. The lumen structure calculation program LSP is a software program that calculates the lumen structure based on the position and orientation information of the anterior end piece 11 and the endoscopic image. The CPU 21a reads and executes the lumen structure calculation program LSP. The processor 21 constitutes a lumen structure calculation unit, which calculates the 3D structure of the lumen based on the captured image obtained by the imaging element 15 and the 3D configuration (i.e., position and orientation) of the imaging unit detected by the magnetic sensor 16.

[0054] The storage device 22 also stores calculated information about the lumen structure. The lumen structure information stored in the storage device 22 is output via the display I / F 23 and displayed on the screen of the monitor 6. Here, the lumen is the large intestine, and an image of the large intestine's lumen structure is displayed on the monitor 6.

[0055] Monitor 6 has a PinP (Picture In Picture) function, which can display real-time endoscopic images captured by the camera element 15 of endoscope 2 together with images of the lumen structure of the large intestine generated by CPU 21.

[0056] The image acquisition unit 24 is a processing unit that acquires endoscopic images obtained in the image processing device 3 at a certain period. For example, it acquires 30 endoscopic images from the image processing device 3 within 1 second at the same frame rate as the images acquired from the endoscope 2. Alternatively, while the image acquisition unit 24 acquires 30 endoscopic images within 1 second, it may also acquire endoscopic images at a longer period, such as 3 images within 1 second, which is different from the frame rate.

[0057] The position and posture detection unit 25 controls the drive circuit 26 that drives the magnetic field generating device 7, causing the magnetic field generating device 7 to generate a predetermined magnetic field. The position and posture detection unit 25 detects this magnetic field through the magnetic sensor 16, and generates the position coordinates (x, y, z) and orientation (i.e., Euler angles) of the imaging element 15 in real time based on the detection signal of the detected magnetic field. The position and pose detection unit 25 detects the 3D configuration based on the detection signal from the magnetic sensor 16. This 3D configuration includes information about at least a portion of the position and orientation of the imaging element 15. More specifically, the position and pose detection unit 25 detects information about changes in the 3D configuration over time, i.e., 3D configuration time-varying information. Therefore, the position and pose detection unit 25 acquires 3D configuration information of the insertion unit 2b at multiple time points.

[0058] The lumen structure detection device 5 is connected to an input device 27a such as a mouse and keyboard. The operation signal corresponding to the operation of the input device 27a is input to the processor 21 via the input I / F 27.

[0059] Furthermore, the light source device 4 is a light source device capable of emitting normal light for the normal light observation mode. Additionally, when the endoscope system 1 has a special light observation mode in addition to the normal light observation mode, the light source device 4 selectively emits normal light for the normal light observation mode and special light for the special light observation mode. The light source device 4 emits either the normal light or the special light as illumination light depending on the state of the switching switch (not shown) provided in the image processing device 3 for switching observation modes.

[0060] (Calculation of lumen structure)

[0061] Figure 4 This is a flowchart illustrating an example of the calculation process for the lumen structure using the Lumen Structure Calculation Program (LSP). The LSP is executed when the physician presses the designated operation button on the input device 27a.

[0062] First, when the doctor performs a prescribed operation on the input device 27a with the front end 11 of the insertion part 2b positioned at the anus, the processor 21 sets the position (coordinates) and posture data from the position and posture detection unit 25 as the reference position and reference posture of the front end 11 when calculating the lumen structure (step (hereinafter referred to as S1). Figure 5 This is a diagram illustrating the patient's Pa state when the initial value of the front end 11 is set. (Example) Figure 5 As shown, with the anterior end 11 in contact with the anus, the doctor sets the reference position and reference posture of the anus at point AP in 3D space as the initial values.

[0063] The lumen structure calculated through the following process is based on the reference position and reference posture set here.

[0064] After establishing the reference position and posture, the doctor inserts the front end 11 into the innermost part of the large intestine. Starting with the front end 11 of the insertion part 2b located in the innermost part of the large intestine, while supplying air to expand the large intestine, the doctor pulls the insertion part 2b towards the anus, and while stopping the withdrawal of the insertion part 2b midway, bends the curved part 12 in various directions to observe the inner wall of the large intestine. While observing the inner wall of the large intestine, the doctor calculates the luminal structure of the large intestine.

[0065] As described above, the image acquisition unit 24 acquires an endoscope image for each predetermined period Δt from the endoscope image provided by the image processing device 3 every 1 / 30th of a second (S2). The period Δt is, for example, 0.5 seconds. The CPU 21 acquires the position and orientation information of the front end 11 output by the position and orientation detection unit 25 when acquiring the endoscope image (S3).

[0066] The positional information of multiple feature points in 3D space from one endoscopic image acquired in S2 and one or more previously acquired endoscopic images is calculated (S4). The set of calculated positional information of multiple feature points constitutes the information of the lumen structure. As described later, the positional information of each feature point can be calculated using methods such as SLAM (Simultaneous Localization and Mapping) and SfM (Structure from Motion) based on image information, or it can be calculated using the principle of triangulation. The calculation method for the position of each feature point will be explained later.

[0067] In addition, when the first endoscopic image is obtained, since there are no previously obtained endoscopic images, the processing of S4 is not performed until a specified number of endoscopic images are obtained.

[0068] The position of each feature point in 3D space is calculated based on a set of feature points and the position and pose of the camera unit. A portion of the tubular structure is constructed based on these feature points. However, because the large intestine can expand and contract, the following conditions are considered: 1) whether the calculated feature points of the two sets correspond to each other; or 2) whether the calculated feature points correspond to feature points whose positions have already been calculated. For example, in the case of 1), the relative range of motion of the feature points of one set relative to the feature points of another set is determined by the expansion and contraction rates of the large intestine along the x, y, and z axes. Therefore, if the feature points of the two sets whose positions have been calculated are not within the range determined by the specified expansion and contraction rates, the feature points of the two sets are determined to be different. That is, the feature points of one set are determined to be different from the feature points of the other set whose positions have already been calculated. This means that a portion of the feature points of one of the two sets is not included in the feature points of the other set whose positions have already been calculated.

[0069] Furthermore, the spatial mobility of the calculated feature point is defined by the distance from the fixed portion of the large intestine and the elasticity of the large intestine. The fixed portion of the large intestine, as referred to here, is a part without an intestinal mesentery, fixed to the posterior abdominal wall, and with limited mobility, such as the portion near the anus, the ascending colon, and the descending colon. For a feature point whose location has been calculated, if the distance from the fixed portion of the large intestine (at least one of the upper end of the ascending colon and the upper end of the descending colon) is known, then if the distance from the calculated feature point to the fixed portion is not within the range obtained by multiplying that distance by the prescribed elasticity of the intestine, it is determined that the calculated feature point is different from the feature point whose location has been calculated. This means that the luminal region containing feature points from one of the two calculated sets is a different part from the luminal region containing feature points from the other of the two calculated sets.

[0070] Furthermore, for the middle portion of the S-shaped colon, the distance from the fixed portions on both sides of the intestine can be considered to determine the extent of its presence. The rectum and descending colon, excluding the S-shaped rectum, are fixed portions. In the middle portion of the S-shaped colon, two constraints can be set: the distance from the junction of the upper rectum and the S-shaped rectum, and the distance from the junction of the S-shaped colon and the descending colon, i.e., the SD junction.

[0071] Figure 6 This is an example diagram illustrating a schematic partial model of the large intestine. Based on the determination in 1) above, a 3D partial model (partial lumen structure) is generated using a set of feature points, therefore, as... Figure 6As shown, in the ascending colon of the large intestine C, although there is a positional offset between two adjacent 3D partial models PM1, PM2, and PM3, a 3D model of the large intestine containing multiple 3D partial models PM1, PM2, and PM3 is still generated. That is, the processor 21, which is the lumen structure calculation unit, calculates a part of multiple 3D structures, namely partial 3D structures, and determines the configuration of multiple partial 3D structures based on the 3D configuration when creating each partial 3D structure, thereby calculating the 3D structure of the large intestine C.

[0072] Figure 7 This is an example diagram illustrating a schematic partial model of the large intestine. If the distance from each feature point of two 3D partial models (partial luminal structures) to the upper end FPU of a fixed portion FP of the large intestine C is within the range of a distance obtained by multiplying by a specified elongation rate, then it is determined to be the same as a feature point at a calculated location. Figure 7 In the middle, the distances of PM4 and PM5 in each part of the model from the upper FPU ( Figure 7 The distance indicated by the arrow is within the range of the distance obtained by multiplying by the specified scaling factor. For example... Figure 7 As shown, the distances of each feature point of the two partial models PM4 and PM5 from the fixed part FP of the large intestine are within the range of the distance obtained by multiplying by the specified stretching rate. Therefore, they are generated as different models, and the two partial models PM4 and PM5 are separated.

[0073] It can generate two partial models, PM4 and PM5, separately, and can also correct the positional offset of the two partial models, PM4 and PM5. Figure 8 This diagram illustrates the method for correcting the positional offsets of the two partial models, PM4 and PM5. (For example...) Figure 8 As shown, Figure 7 The positional offsets of the two partial models PM4 and PM5 shown have been corrected.

[0074] Therefore, when similar feature points are extracted among multiple 3D structures, the processor 21 compares the difference in distance from the similar feature points to the fixed part with a predetermined reference value, thereby determining whether the similar feature points are common feature points among multiple 3D structures.

[0075] Then, based on common feature points, the positions of some models PM4 and PM5 are corrected to ensure that the lateral positions of the upper FPU of the fixed part FP are consistent, and the configuration is then performed. The two distances ( ) from the upper FPU of some models PM4 and PM5 are... Figure 8 The horizontal distance (indicated by the arrow) is respectively with Figure 7 In some models, the two horizontal distances of PM4 and PM5 are consistent.

[0076] As described above, the large intestine C, which is a tube, has a fixed portion that is fixed relative to the outside and a movable portion (transverse colon) that is not fixed relative to the outside. The processor 21 calculates the distance between the partial 3D structure corresponding to the movable portion and the fixed portion, and corrects the configuration of the partial 3D structure based on the distance between the partial 3D structure and the fixed portion. Then, the processor 21 calculates multiple partial 3D structures containing information on the positions of each feature point, and corrects the configuration of the partial 3D model based on the positions of the common feature points of the multiple 3D structures.

[0077] Therefore, as Figure 8 As shown, if a portion of the feature points of two sets are the same, then by performing position correction on two adjacent partial models, the two adjacent partial models can be connected, thereby merging the two partial models into one model.

[0078] Furthermore, the processor 21 can determine at least one of the orientation change location and the dwell location of the insertion part 2b based on the 3D configuration time change information, and estimate the positions of the fixed part and the movable part.

[0079] A predetermined expansion rate can be set for the entire large intestine, but the expansion rate of the portion near the anus, the ascending colon, and the descending colon can also be different from the upper limit of the expansion rate for other portions of the large intestine. That is, a predetermined upper limit (threshold) of the expansion rate in 3D space for each feature point in each region of the lumen can be used to calculate the position of each feature point. Furthermore, when determining the position of each feature point, the amount of change in the detected position of each feature point must be within the range determined by the upper limit (threshold) of the expansion rate. When the amount of change in the detected position of a feature point is below the predetermined upper limit of the expansion rate, the feature points are considered to be identical in all captured images.

[0080] As described above, when calculating the position of each feature point, an upper limit (threshold) [0] for the change in the position of each feature point can also be considered. This upper limit (threshold) [0] is based on the upper limit of the expansion rate of each region of the lumen containing each feature point, or the distance from a fixed point in the large intestine and its upper limit of the expansion rate. Then, using the prescribed upper limit of the expansion rate for each feature point set for each region of the lumen, it is determined whether common feature points appearing in multiple captured images are the same.

[0081] When feature points detected at different timings within a range determined by the scaling factor are identical through pattern matching, these feature points are considered to be the same. The positional information, which serves as structural information for feature points, is determined based on the information between the feature points. For example, in a simple case, information from any one of the feature points can be used as the positional information constituting the structural information.

[0082] In addition, the range determined by the elongation rate is based on the distance between the lumen region containing each feature point and a fixed point in the large intestine, and the upper limit of the elongation rate.

[0083] The structural information of the lumen is created by adding the calculated positional information of multiple feature points to the structural information data (S5). The structural information created in S5 consists of a set of one or more feature points in the region observed through the endoscope 2.

[0084] The structural information of the created cavity is displayed on monitor 6 (S6). Figure 9 This is a diagram showing an example of an image displayed on monitor 6. The display screen 6a of monitor 6 includes an endoscope image display area 31 and a structural information display area 32.

[0085] The structural information display area 32 is positioned next to the endoscope image display area 31, which displays the endoscope image as a real-time view. The structural information display area 32 displays the structural information SI generated in S5. The structural information SI is an image of the three-dimensional luminal structure of the large intestine observed from a single viewpoint, and consists of positional information of multiple feature points, etc. Figure 10 This is an example diagram showing an image of the structural information SI displayed in the structural information display area 32. The structural information SI is displayed as a three-dimensional view of the lumen of the large intestine. Because the structural information SI is 3D data, the user, i.e., the physician, can confirm the structure of the lumen when viewed from a desired direction of 360 degrees by giving instructions to change the viewpoint position.

[0086] like Figure 10 As shown, structural information SI is a set of information such as feature points calculated in S4. Therefore, the part not shown by structural information SI refers to the unobserved area UIA. Figure 10 In the diagram, the area indicated by the dashed line is the unobserved UIA region. Therefore, by observing the luminal structure of the large intestine displayed on monitor 6, the doctor can identify the unobserved UIA region.

[0087] In addition, the physician can operate the input device 27a to provide the prescribed commands to the lumen structure detection device 5, and display the timing movement of the anterior end 11 based on the position information of the anterior end 11 obtained in S3. Figure 11 This is an example diagram showing an image displayed on monitor 6 representing the trajectory of the front end 11. Figure 11 An image of a window 33 displayed on display screen 6a of monitor 6 is shown. Figure 11 Window 33, as shown, displays an image representing the motion of the front end 11 in the XY plane, as viewed from the Z-axis direction. Figure 11 In the diagram, based on the defined point 0 on the XY plane and the unit of movement distance in the XY direction being mm (millimeters), it can be seen that the trajectory of the front end 11 is displayed. By... Figure 11 The image is displayed on monitor 6, allowing the doctor to track the trajectory of the anterior end 11.

[0088] For example, based on the position information of the tip 11, it is possible to compare the trajectories of the tip 11 when two doctors perform the operation. For example, it is possible to compare the insertion operation of inserting the insertion part 2b into the large intestine of the same patient Pa based on the trajectory of the tip 11, which is generated based on the position information of the tip 11 when two doctors perform the operation.

[0089] Figure 12 This diagram illustrates an example of a display where the trajectory of the anterior endoscope 11, performed by two doctors, is shown on display screen 6a of monitor 6, using a colon model as the object. Display screen 6a shows two windows 33a and 33b. Window 33a displays an image representing the trajectory of the anterior endoscope 11 of doctor A, and window 33b displays an image representing the trajectory of the anterior endoscope 11 of doctor B. Doctor A is a senior endoscopist, and doctor B is a senior endoscopist.

[0090] Compared with the trajectory of the front end 11 of the backbone doctor B shown in window 33a, the trajectory of the front end 11 of the senior doctor A shown in window 33b shows the following: there is more movement near the anus (AA), in the descending colon (DC), and in the ascending colon (AC), and the insertion part 2b is not smoothly inserted or withdrawn.

[0091] Therefore, the trajectory information of the front end 11 can be used to perform comparisons such as the insertion or removal operation of the insertion part 2b.

[0092] (Calculation of the position of feature points, etc.)

[0093] There are various methods for calculating the positions of feature points, etc., in S4. Several methods are explained below.

[0094] 1. Calculating the positions of feature points on multiple consecutive images using methods such as SLAM and SfM.

[0095] The method for optimizing bundle adjustment is described, wherein the optimization of bundle adjustment is used to calculate the position of each feature point in 3D space using information on the position (in this case, the position of the front end 11) and pose of the camera element 15 that is photographing the subject.

[0096] Bundle adjustment is an error adjustment method that uses nonlinear least squares to optimize intrinsic parameters, extrinsic parameters, and world coordinate point sets based on an image. For example, using the estimated parameters, the perspective projection of the world coordinate points of multiple extracted feature points is transformed into image coordinate points, and the parameters and world coordinate point sets are determined in a way that minimizes the reprojection error.

[0097] The external parameters associated with the front end 11 are calculated using 5-point and 8-point algorithms. The positions of the feature points are calculated based on the position of the front end 11 and triangulation. The error E between the coordinates of the 3D points projected onto the image plane and the feature points based on the reprojection error is expressed by the following equation (1).

[0098]

[0099] Here, L is the number of feature points on the K images, Psj is the coordinate position of the 3D point Pi estimated on the image plane by triangulation and the parameters of the front end 11, and Pi is the coordinate position of the corresponding feature point on the image. The position coordinates of the front end 11 are calculated using the LM (Levenberg-Marquardt) method in a way that minimizes the error E of Equation (1) as a function.

[0100] The positions of the feature points in S4 above in 3D space are calculated based on the coordinate information of the feature points extracted from the image and the detected position and pose information of the front end 11. However, in this embodiment, the position and pose information of the front end 11 is detected by the position and pose detection unit 25, which is approximately accurate. Therefore, the position and pose information of the front end 11 is not included in the estimation of external parameters in bundle adjustment. Therefore, not only is the estimation accuracy of the positions of each feature point in 3D space based on bundle adjustment high, but the computation time for the positions of multiple extracted feature points is also shortened.

[0101] Figure 13 This is a flowchart of a method for calculating the position of each feature point in 3D space using bundle adjustment.

[0102] In Figure 5 When the anal position AP is set to the initial position, processor 21 sets time t to t0 and sets the count value n of the software counter to 0 (S11). Processor 21 acquires the endoscopic image (i.e., lumen image) at time t0 and information on the position and orientation of the anterior endpiece 11 (S12). The endoscopic image is acquired from the image processing device 3. Information on the position and orientation of the anterior endpiece 11 is acquired from the position and orientation detection unit 25.

[0103] The processor 21 determines the initial position, i.e., the position and orientation of the anal sphincter 11 at the anal position AP (S13). For example, the anal position AP (x, y, z) is determined to be (0, 0, 0), and the orientation (vx, vy, vz) is determined to be (0, 1, 0). S11 and S13 correspond to Figure 4 S1.

[0104] Processor 21 acquires the endoscopic image at time (t0+nΔt) and information on the position and orientation of the front end 11 (S14). S12 and S14 correspond to... Figure 4 S2.

[0105] Additionally, the position and orientation information of the front end 11 can be corrected. For example, a Kalman filter can be used to correct the path that the front end 11 has taken in the past, and based on the corrected path, the position of the front end 11 in the past can be corrected.

[0106] When n becomes k, processor 21 extracts multiple feature points from each endoscopic image, sets the position and orientation of the front end 11 at k time points (i.e., the 3D configuration of the front end 11) as known, and calculates the positions of m feature points commonly included in the obtained endoscopic images using the bundle adjustment method described above (S15). Therefore, the extraction process of multiple feature points in each endoscopic image in S15 constitutes a feature point extraction unit for extracting multiple feature points in each captured image. In S15, feature points that commonly appear in the captured images at multiple time points are extracted. The calculation process of the position of each feature point in 3D space in S15 constitutes a 3D position calculation unit, which calculates the position of the feature points in 3D space based on the positions of the extracted multiple feature points in the captured images and the 3D configuration of the insertion unit 2b. More specifically, the position of the feature points in 3D space is calculated based on the 3D configuration information of the insertion unit 2b at multiple time points and the positions of the feature points that commonly appear in the captured images at multiple time points. Then, the position of each feature point in 3D space is determined by bundle adjustment.

[0107] Figure 14 This is a schematic diagram illustrating the relationship between feature points on multiple consecutively acquired endoscopic images and the position and orientation of the anterior endpiece 11. Figure 14 In the diagram, the white triangle Pw represents the actual position and orientation of the front end 11, and the black triangle Pb represents the estimated position and orientation of the front end 11. The diagram shows the actual movement of the front end 11 along the solid line. The estimated movement of the front end 11 along the dashed line is also shown. As time passes, the position of the front end 11 changes, and its orientation also changes.

[0108] In addition, Figure 14In the diagram, the white quadrilateral pw represents the actual location of the feature point, while the black quadrilateral pb represents the estimated, or calculated, location of the feature point. Feature points are, for example, areas in endoscopic images that are characterized by their shape and color and are easily identifiable or trackable.

[0109] To obtain the 3D structure of the large intestine's lumen, the coordinates of several specific locations (referred to as feature points) on the inner wall of the large intestine are calculated. A 3D model is then generated by combining these coordinates or by connecting them. In other words, the 3D structure of the lumen is determined by the calculated positions of each feature point in 3D space.

[0110] As mentioned above, it is possible to calculate the positions of a certain number of feature points on the inner wall of the lumen using only information from endoscopic images. When using only endoscopic images, techniques such as SLAM and SfM can be employed. However, when using these techniques, in addition to the unknown coordinates of multiple feature points on the intestinal wall, the position and orientation of the endoscope tip 2 at each elapsed time are also unknown. Therefore, estimating the positions of multiple feature points on the intestinal wall requires enormous computation, and the estimation accuracy is relatively low.

[0111] exist Figure 14 In the calculation, the position and orientation information of the camera element 11 at each time point contains information corresponding to 6 axes, therefore the position and orientation information of the camera element 11 at k time points contains 6k pieces of information. The position of each feature point contains information corresponding to 3 axes, therefore the position information of m feature points contains 3m pieces of information. Therefore, (6k+3m) parameters are calculated through optimized calculations using bundle adjustment.

[0112] When calculating a large number of parameters using bundle adjustment, the accumulated detection errors can cause a shift in the sequentially generated 3D model structure. Furthermore, because the tip of the insertion section 2b of the endoscope 2 is pressed against the inner wall of the lumen, continuous endoscopic images cannot be obtained, thus preventing the calculation of the 3D model structure.

[0113] In contrast, in this embodiment, the position and orientation information of the front end 11 over time is obtained by the magnetic sensor 16 and set as known. Therefore, the number of parameters calculated in bundle adjustment can be reduced, thereby reducing the processing load of optimization calculations and achieving high speed. Furthermore, since the position of feature points can also be calculated by triangulation, the 3D model structure can be calculated even if continuous endoscopic images are not available.

[0114] In addition, when calculating the position and orientation of the front end 11 and the position of each feature point in 3D space by using bundle adjustment, there is a problem that the error of the calculated position of each feature point is accumulated, and the cavity structure gradually deviates from the actual structure. However, in this embodiment, the position and orientation of the front end 11 are detected by the magnetic sensor 16, so such accumulated error will not occur.

[0115] For example, in the above Figure 14 In the case where the position of each feature point is calculated based solely on the endoscopic image, optimization calculations for (6k+3m) parameters are typically performed. However, since the position and orientation information of the front end 11 are known, the optimization calculations only require 3m parameters, which reduces the processing load of the optimization calculations in bundle adjustment and achieves high speed.

[0116] Furthermore, even when the tip 11 of the insertion portion 2b of the endoscope 2 is pressed against the inner wall of the lumen, immersed in dirty cleaning water, or when the endoscope image is jittery, preventing the acquisition of a proper continuous endoscopic image, information on the position and orientation of the tip 11 can still be obtained. Therefore, even in situations where a continuous endoscopic image cannot be obtained due to the tip 11 being pressed against the inner wall of the lumen, the possibility of calculating 3m parameters is increased. As a result, the robustness of the calculation of the lumen structure is improved.

[0117] Return to Figure 13 The processor 21 adds the newly calculated feature point position information to the already created lumen structure information, updating the lumen structure information (S16). S16 corresponds to... Figure 4 S5.

[0118] Processor 21 corrects the position information of previously calculated feature points (S17). For the position information of previously calculated feature points among the newly calculated 3m feature points, the newly calculated position information is used to correct the previously calculated position information, for example, by averaging.

[0119] Alternatively, the processing in S17 can be omitted, or the position information of each feature point calculated in the past can be updated using the position information of the newly calculated feature points.

[0120] Based on the updated lumen structure information, processor 21 outputs image data of the lumen structure to monitor 6 so that the image of the lumen structure can be displayed in lumen structure display area 32 (S18). S18 corresponds to... Figure 4 S6.

[0121] After S18, processor 21 increments n by 1 (S19) and determines whether a command to end the examination has been input (S20). Regarding the command to end the examination, for example, after the insertion part 2b is pulled out of the large intestine, when the doctor inputs a specified command into the input device 27a (S20: Yes), the execution of the lumen structure calculation program LSP ends.

[0122] If no command to end the inspection is entered (S20: No), the process proceeds to S14. As a result, the processor 21 obtains the endoscope image after a period Δt from the last acquisition time of the endoscope image (S14) and executes the processing after S14.

[0123] In addition, in the above method, the position and orientation information of the front end 11 are detected by the 6-axis magnetic sensor 16, and the 3D position of each feature point is calculated by the optimization of bundle adjustment. However, a portion of all the 6-axis information can also be set to be known, that is, at least one of the 3D position (x, y, z) and 3-axis direction (vx, vy, vz) can be set to be known, and the 3D position of each feature point can be calculated by the optimization of bundle adjustment.

[0124] Therefore, in bundle adjustment, at least one of the position and orientation information of the front end 11 is set to be known to calculate the 3D position of each feature point, thus improving the calculation accuracy of the 3D position of each feature point and shortening the overall time of the optimization operation.

[0125] 2. Using image triangulation to calculate the location of feature points (e.g., the center point of the image) based on two images.

[0126] This paper explains the method of calculating the position of each feature point using triangulation based on two images from different viewpoints. Figure 15 This is a schematic diagram illustrating the relationship between feature points on multiple endoscopic images and the position and orientation of the anterior endpiece 11. Figure 15 In the diagram, the white triangle P represents the actual position and orientation of the front end 11, i.e., its 3D configuration. The diagram shows the actual movement of the front end 11 along the solid line. Figure 15 In the diagram, the white quadrilateral p represents the actual location of the feature point. Each feature point is, for example, a part of the endoscopic image that has distinctive shape and color and is easy to identify or track.

[0127] As shown by the solid line, based on the position and orientation information of the front end 11 obtained at times t1 and t2, the positions of feature points p1 and p2 on the image obtained at time t1 and the image obtained at time t2, respectively, the positions of feature points p1 and p2 are calculated by triangulation.

[0128] Similarly, as shown by the dashed line, based on the position and orientation information of the front end 11 obtained at times t2 and t3, the positions of feature points p3 and p4 on the image obtained at time t2 and the respective positions of feature points p3 and p4 on the image obtained at time t4, the positions of feature points p3 and p4 are calculated by triangulation.

[0129] As described above, based on the position and orientation information of the front end 11 and two endoscopic images, triangulation is used to calculate the position of each feature point. That is, based on the position and orientation information of the camera element 15 at multiple time points and the positions of the feature points that appear in the images captured by the camera element 15 at multiple time points on the captured images, triangulation is used to calculate the position of each feature point in 3D space and determine the 3D structure of the lumen.

[0130] In addition, the triangulation described above is based on two endoscopic images obtained at two different times, but it can also be based on two endoscopic images obtained at the same time.

[0131] Figure 16 This is a perspective view of the front end portion of the insertion part of a stereo camera. Two observation windows 11a and 11b and two illumination windows 11c are arranged on the front surface of the front end portion 11. A camera optical system and an image sensor are arranged behind each observation window 11a and 11b. The camera optical system and the image sensor constitute an image sensor 11d, and the two image sensor portions 11d constitute a stereo camera.

[0132] Based on the position of the stereo matching region in two endoscopic images obtained by a stereo camera located at the front end 11, the position of each point on the inner wall of the large intestine can be calculated by triangulation.

[0133] 3. Calculating the location of feature points using photometric stereo imaging.

[0134] The method for calculating the position of each feature point using photometric stereo images is explained. Figure 17 It is a top view of the front end face of the insert with multiple lighting windows. Figure 17 This is a diagram showing the front end face 11a1 of the front part 11 as viewed from the front end side of the insertion part 2b.

[0135] like Figure 17As shown, the front end face 11a1 of the front end portion 11 is provided with an observation window 41, three illumination windows 42, a clamping jaw 43, a cleaning nozzle 44, and a secondary water inlet 45. A camera unit 11d is provided on the rear side of the observation window 41. The three illumination windows 42 are arranged around the observation window 41. A light guide (not shown) is provided on the rear side of each illumination window 42. The clamping jaw 43 is an opening for a treatment device to protrude, which passes through the treatment device insertion channel provided in the insertion portion 2b. The cleaning nozzle 44 sprays water for cleaning the surface of the observation window 41. The secondary water inlet 45 is an opening for spraying secondary water.

[0136] By controlling the drive of multiple light-emitting diodes installed in the light source device 4 for illumination, it is possible to switch and selectively emit the three illumination lights emitted from the three illumination windows 42.

[0137] The state of the shadowed areas in an image of the subject's surface changes due to the switching of the illumination light. Therefore, based on this change, the distance to the shadowed areas on the subject's surface can be calculated. That is, the 3D structure of the lumen can be determined based on a photometric stereo method using an image of the shadowed areas in an image obtained from illumination by multiple selectively operated illumination units.

[0138] 4. Using a distance sensor to calculate the structure of the tubular cavity.

[0139] A method for calculating the position of each point on the inner wall of the lumen using a distance sensor provided at the front end 11 is described.

[0140] Figure 18 This is a perspective view of the front end portion of the insertion part with a distance sensor. The front end face 11a1 of the front end portion 11 is provided with an observation window 41, two illumination windows 42, a clamping jaw 43, and a distance sensor 51.

[0141] The distance sensor 51 is a TOF (Time of Flight) sensor, specifically a sensor that detects distance in an image using TOF. The distance sensor 51 measures distance by measuring the time of flight of light. TOF functionality is embedded in each pixel of the image sensor. Thus, the distance sensor 51 obtains distance information for each pixel. Specifically, the distance sensor 51 is positioned at the front end 11 of the insertion portion 2b to detect the distance from the front end 11 to the inner wall of the cavity.

[0142] Based on the distances associated with each pixel detected by the distance sensor 51, and the position and orientation of the front end 11, the positional information of each point on the inner wall of the large intestine, i.e., the 3D structure of the lumen, can be calculated. Alternatively, the distance sensor can be another type of sensor, such as LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging).

[0143] Figure 19 This is a flowchart illustrating a method for calculating the 3D position of the inner wall of a tube using a distance sensor. Figure 19 In the middle, to and Figure 13 For the same process, the same step number is used and the explanation is omitted; only the different processes are explained.

[0144] When using the distance sensor 51, by using one or more predetermined pixels of the distance sensor 51 as feature points, the 3D position of each point on the inner wall of the large intestine can be calculated based on the distance information of each pixel of the distance sensor 51 and the position and orientation information of the front end 11, regarding the position of each feature point. The lumen structure is composed of the set of 3D position information of each point on the inner wall. That is, in Figure 13 In step S21, the processor 21 calculates the 3D coordinates of the distance measurement point (feature point) based on the distance information output from the distance sensor 51 and the position and orientation information of the front end 11 obtained in step S14.

[0145] 5. Distance measurement based on other methods

[0146] Alternatively, an illumination section that emits a predetermined pattern light can be provided at the front end 11, and distance measurement from the front end 11 to the inner wall can be performed by projecting the pattern light.

[0147] As described above, according to the above-described embodiment, the structural information of the inner wall of the large intestine can be obtained using information about the position and orientation (i.e., configuration) of the front end portion 11 of the insertion portion 2b, as well as image information from the endoscope or distance information from the ranging sensor.

[0148] In the above-described embodiments, since information about the position and orientation of the front end 11 can be obtained, even if an endoscope image cannot be obtained due to the front end being too close to the tube wall or the distance information of the distance sensor 51 cannot be obtained, although the structure of the lumen portion where an endoscope image is not obtained cannot be calculated, the structure of the lumen portion where an endoscope image is obtained can be calculated.

[0149] Figure 20 This is a diagram illustrating an example of a partially calculated tubular structure. In Figure 20In the diagram, the portion represented by the double-dotted ellipse indicates a region of the large intestine that has received endoscopic images or distance information from the distance sensor 51. Regarding the portion represented by the dashed line, even if no endoscopic image or distance information from the distance sensor 51 is available, the structural information of the region represented by the solid line is still calculated. That is, in cases where there is no overlap between feature points of different sets or where it is impossible to determine this, multiple unconnected luminal structures are positioned according to their respective locations based on the positional information of each luminal structure.

[0150] Therefore, even if endoscopic images are not obtained midway, the positions of each feature point can be calculated based on multiple endoscopic images. As a result, the 3D model structure of the lumen can be partially calculated.

[0151] Furthermore, in the above embodiment, a magnetic sensor 16 is used as a position sensor for detecting the position and orientation of the front end portion 11 of the insertion portion 2b, but the position and orientation of the front end portion 11 can also be detected by different mechanisms.

[0152] Figure 21 This is a diagram illustrating a method for detecting the position and orientation of the anterior endpiece 11 using an endoscope with a shape sensor and a sensor that detects the amount of insertion and torsion.

[0153] The shape sensor 61 is disposed inside the insertion part 2b over the entire area from the base to the front end. The shape sensor 61 is, for example, a bending sensor that uses an optical fiber to detect the amount of bending based on the curvature of a specific part, i.e., an optical fiber sensor.

[0154] The insertion / torsion sensor 62 is disposed near the anus and has a cylindrical shape with a hole through which the insertion part 2b can be inserted. An encoder for detecting the axial insertion amount of the insertion part 2b and an encoder for detecting the rotational amount of the insertion part 2b about its axis are disposed on the inner circumferential surface of the hole of the insertion / torsion sensor 62. Therefore, using the shape sensor 61 and the insertion / torsion sensor 62, the position and orientation of the anterior end portion 11 can be estimated based on the insertion amount and torsion amount of the insertion part 2b, with the position of the anus as a reference.

[0155] Alternatively, the shape sensor 61 may not utilize optical fiber, but instead have a magnetic sensor 16 provided at the front end 11, and multiple magnetic sensors 16 arranged at predetermined intervals within the insertion part 2b, thereby detecting the shape of the insertion part 2b. Figure 22 This is a perspective view of an endoscope equipped with multiple magnetic sensors 16 within the insertion section 2b. According to... Figure 22 The position information of the multiple magnetic sensors 16 shown can be used to calculate the shape of the insertion part 2b.

[0156] In addition, such as Figure 2 As shown by the dashed line, the shape sensor 61 described above can also be installed inside the insertion portion 2b of the endoscope 2. If the shape sensor 61 is used, the overall shape of the insertion portion 2b can also be determined.

[0157] In the above-described embodiment, the lumen structure is calculated while the doctor observes the endoscopic image. However, if hidden portions exist, such as the dorsal side of a fold, the lumen structure in that region is not calculated. Hidden portions are areas not captured in the endoscopic image. Since lumen structures cannot be generated from areas not included in the endoscopic image, the presence of hidden portions can be detected from the endoscopic image and the doctor can be informed.

[0158] Figure 23 This diagram illustrates a display example of a monitor 6 when a hidden portion is present. The endoscope image display area 31 of the monitor 6's display screen 6a shows an endoscopic image of the large intestine where a shadow has formed on the inner side of a portion of the inner wall. The dark shadow area SA, not illuminated by the lighting, has a progressively lower brightness compared to other areas. Therefore, when the brightness difference between adjacent pixels or adjacent pixel areas is greater than a predetermined brightness value, it can be determined that a hidden portion exists.

[0159] In such a situation, such as Figure 23 Similar to pop-up window 34, the monitor 6 displays a specified message, thereby informing the user and allowing the user to observe the hidden portion. The hidden portion also includes recesses such as diverticula. When the user sees the specified message and observes the hidden portion, information about the luminal structure of the observed area is calculated.

[0160] Furthermore, in the example above, hidden parts are detected based on the presence or absence of shadow areas not illuminated by the lighting light. However, the same presence or absence of hidden parts can also be detected using distance image information from the distance sensor 51, etc. When the distance difference between adjacent pixels or adjacent pixel regions is a predetermined distance value or higher, it can be determined that a hidden part exists.

[0161] Figure 24 This is a flowchart illustrating an example of the process for informing about unobserved areas based on image brightness values. Figure 4 When obtaining endoscopic images from a new viewpoint in S2, and... Figure 4 The processing is performed in parallel. Figure 24 The processor 21 obtains the brightness information of the endoscopic image from the new viewpoint acquired in S2 (S31). That is, it obtains the brightness information of each pixel in the image.

[0162] The processor 21 determines (S32) the following situations: a) whether there is a situation in the specified pixel area of ​​the image where the difference in brightness value between two adjacent pixels is greater than or equal to a specified value; or b) whether there is a dark striped area in the specified pixel area of ​​the image.

[0163] Processor 21 determines which of a) and b) is true (S33). If either a) or b) is true (S33: Yes), processor 21 considers that a hidden part may exist within the field of view and displays a specified message on monitor 6 (S34). The specified message is, for example, Figure 23 A message like pop-up window 33 appears. If neither a) nor b) is true (S33: No), no processing is performed. The processing in S34 constitutes a notification unit that provides a specified notification when the difference in brightness values ​​between two adjacent pixels or two pixel regions in the captured image is greater than or equal to a specified value.

[0164] Figure 25 This is a flowchart illustrating an example of a process for informing about unobserved areas based on a distance image from distance sensor 51. Figure 4 When obtaining endoscopic images from a new viewpoint in S2, and... Figure 4 The processing is performed in parallel. Figure 25 The processor 21 obtains the distance image information of the distance sensor 51 from the new viewpoint acquired in S2 (S41). That is, it obtains the distance information of each pixel in the image.

[0165] The processor 21 determines and confirms the following situations (S42): c) whether there is a situation in the specified pixel area of ​​the image where the difference in distance between two adjacent pixels is greater than a specified value; or d) whether there is a part in the specified pixel area of ​​the image where the distance changes are discontinuous.

[0166] Processor 21 determines which of c) and d) is true (S43). If either c) or d) is true (S43: Yes), processor 21 considers that a hidden part may exist within the field of view and displays a specified message on monitor 6 (44). The specified message is, for example, Figure 23 A message like pop-up window 34. If neither c) nor d) is true (S43: No), no processing is performed. The processing in S43 constitutes a notification unit that provides a specified notification when the difference between the distance values ​​of two adjacent pixels or two pixel regions in the image is greater than or equal to a specified value.

[0167] As described above, the embodiments described above provide an endoscope system, a lumen structure calculation device, a method for generating lumen structure information, and a computer program product that can calculate a 3D model structure of the lumen based on the position of each feature point even if image defects occur due to organ movement or other reasons.

[0168] This invention is not limited to the above-described embodiments. Various changes and modifications can be made without altering the spirit of this invention.

Claims

1. A cavity structure calculation device, characterized in that, have: The input unit acquires multiple time-point images and a 3D configuration. The multiple time-point images are acquired by the camera unit and include the same part of the subject. The 3D configuration includes at least some information about the position and orientation of the camera unit. The camera unit is disposed in an insertion unit inserted into a cavity that serves as the subject. as well as The lumen structure calculation unit calculates the position of the same location based on the captured images at the multiple time points and the 3D configuration, thereby calculating the 3D structure of the lumen. The lumen structure calculation unit calculates multiple partial 3D structures and determines the configuration of the multiple partial 3D structures based on the 3D configuration when fabricating each partial 3D structure, thereby calculating the 3D structure, where the partial 3D structure is a part of the overall 3D structure. The lumen structure calculation unit includes a feature point extraction unit, which extracts multiple feature points from each captured image. The lumen structure calculation unit calculates multiple partial 3D structures containing the position information of each feature point, and corrects the configuration of the partial 3D structures based on the position of the common feature points in the multiple partial 3D structures. The cavity has a fixed part that is fixed relative to the outside and a movable part that is not fixed relative to the outside. When similar feature points are extracted among multiple 3D partial structures, the lumen structure calculation unit compares the difference in distance from the similar feature points to the fixing part with a predetermined benchmark value to determine whether the similar feature points are common feature points among multiple 3D partial structures.

2. The cavity structure calculation device according to claim 1, characterized in that, The same location is a feature point. The cavity structure calculation unit has: The feature point extraction unit extracts multiple feature points from each captured image; and The 3D position calculation unit calculates the position of each feature point in 3D space based on the positions of the plurality of feature points on the captured image and the 3D configuration of the insertion unit. The cavity structure calculation unit calculates the 3D structure of the cavity based on the calculated positions of each feature point in the 3D space.

3. The cavity structure calculation device according to claim 2, characterized in that, The feature point extraction unit extracts the feature points that commonly appear in the captured images at the multiple time points. The input unit obtains the 3D configuration information of the insertion unit at the multiple time points. The 3D position calculation unit calculates the position of the feature point in the 3D space based on the 3D configuration information of the insertion unit at the multiple time points and the position of the feature point that appears in the captured image at the multiple time points.

4. The cavity structure calculation device according to claim 1, characterized in that, The cavity has a fixed part that is fixed relative to the outside and a movable part that is not fixed relative to the outside. The cavity structure calculation unit calculates the distance between the partial 3D structure corresponding to the movable part and the fixed part, and corrects the configuration of the partial 3D structure based on the distance between the partial 3D structure and the fixed part.

5. The cavity structure calculation device according to claim 4, characterized in that, The input unit acquires 3D configuration time change information, which is information about the changes in the 3D configuration over time. Based on the 3D configuration time change information, the cavity structure calculation unit determines at least one of the orientation change location and the dwell location of the insertion part, thereby estimating the positions of the fixed part and the movable part.

6. The cavity structure calculation device according to claim 2, characterized in that, The 3D position calculation unit determines the position of each feature point in the 3D space by adjusting the error.

7. The cavity structure calculation device according to claim 6, characterized in that, The 3D position calculation unit sets the configuration obtained by the input unit as known, and determines the position of each feature point in the 3D space by error adjustment based on bundle adjustment.

8. The cavity structure calculation device according to claim 2, characterized in that, The 3D position calculation unit calculates the position of the feature point in the 3D space based on triangulation, according to the 3D configuration information of the insertion unit at the multiple time points and the position of the feature point that appears in the captured image at the multiple time points.

9. The cavity structure calculation device according to claim 1, characterized in that, The lumen structure calculation unit determines the 3D structure of the lumen based on the image of the shadow area in the captured images at multiple time points using a photometric stereo method: the captured images at multiple time points are obtained by switching the emitting illumination part among multiple illumination parts provided at the front end of the insertion part, and include the same part of the subject.

10. The cavity structure calculation device according to claim 2, characterized in that, The 3D position calculation unit uses a specified upper limit value of the expansion rate of each feature point set for each region of the cavity to determine whether the common feature points appearing in multiple captured images are the same.

11. The cavity structure calculation device according to claim 10, characterized in that, When the change in the position of the detected feature point is below the upper limit of the specified scaling factor, the 3D position calculation unit determines that the feature points appearing in each of the captured images are the same.

12. The cavity structure calculation device according to claim 1, characterized in that, The cavity structure calculation device has a notification unit that provides a specified notification when the difference in brightness values ​​between two adjacent pixels or two pixel regions in the captured image is greater than or equal to a specified value.

13. An endoscope system, characterized in that, have: An insertion part is inserted into the lumen of the object being photographed; A camera unit is provided in the insertion part and acquires images of the subject at multiple time points, including the same part of the subject; A detection device for detecting a 3D configuration containing at least a portion of information about the position and orientation of the camera unit; as well as The lumen structure calculation unit calculates the position of the same location based on the captured images at the multiple time points and the 3D configuration. The lumen structure calculation unit calculates multiple partial 3D structures and determines the configuration of the multiple partial 3D structures based on the 3D configuration when fabricating each partial 3D structure, thereby calculating the 3D structure, where the partial 3D structure is a part of the overall 3D structure. The lumen structure calculation unit includes a feature point extraction unit, which extracts multiple feature points from each captured image. The lumen structure calculation unit calculates multiple partial 3D structures containing the position information of each feature point, and corrects the configuration of the partial 3D structures based on the position of the common feature points in the multiple partial 3D structures. The cavity has a fixed part that is fixed relative to the outside and a movable part that is not fixed relative to the outside. When similar feature points are extracted among multiple 3D partial structures, the lumen structure calculation unit compares the difference in distance from the similar feature points to the fixing part with a predetermined benchmark value to determine whether the similar feature points are common feature points among multiple 3D partial structures.

14. A method for generating information about a lumen structure, characterized in that, The following steps are required: The camera captures images of a subject at multiple time points, the images being captured by a camera unit that includes the same part of the subject, the camera unit being disposed in an insertion part inserted into a cavity that serves as the subject; Obtain a 3D configuration, which includes at least one piece of information regarding the position and orientation of the insertion portion; as well as The location of the same part is calculated based on the captured images at multiple time points and the 3D configuration, thereby calculating the 3D structure of the lumen. In the step of calculating the 3D structure of the cavity, Multiple partial 3D structures are calculated, and the configuration of the multiple partial 3D structures is determined based on the 3D configuration when each partial 3D structure is fabricated, thereby calculating the 3D structure, wherein the partial 3D structure is a part of the 3D structure. Extract multiple feature points from each captured image. Calculate multiple partial 3D structures containing the position information of each feature point, and correct the configuration of the partial 3D structures based on the positions of the common feature points in the multiple partial 3D structures. The cavity has a fixed part that is fixed relative to the outside and a movable part that is not fixed relative to the outside. When similar feature points are extracted among multiple 3D partial structures, the difference in distance from the similar feature points to the fixed part is compared with a predetermined benchmark value to determine whether the similar feature points are common feature points among multiple 3D partial structures.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the following methods: The camera captures images of a subject at multiple time points, the images being captured by a camera unit that includes the same part of the subject, the camera unit being disposed in an insertion part inserted into a cavity that serves as the subject; Obtain a 3D configuration, which includes at least one piece of information regarding the position and orientation of the insertion portion; as well as The location of the same part is calculated based on the captured images at multiple time points and the 3D configuration, thereby calculating the 3D structure of the lumen. In the step of calculating the 3D structure of the cavity, Multiple partial 3D structures are calculated, and the configuration of the multiple partial 3D structures is determined based on the 3D configuration when each partial 3D structure is fabricated, thereby calculating the 3D structure, wherein the partial 3D structure is a part of the 3D structure. Extract multiple feature points from each captured image. Calculate multiple partial 3D structures containing the position information of each feature point, and correct the configuration of the partial 3D structures based on the positions of the common feature points in the multiple partial 3D structures. The cavity has a fixed part that is fixed relative to the outside and a movable part that is not fixed relative to the outside. When similar feature points are extracted among multiple 3D partial structures, the difference in distance from the similar feature points to the fixed part is compared with a predetermined benchmark value to determine whether the similar feature points are common feature points among multiple 3D partial structures.

Images