Image processing device, image processing method, and program

Abstract

Provided are an image processing device, an image processing method, and a program capable of acquiring accurate depth information from an intraluminal image by acquiring correct scope information. In an image processing device (14) provided with a processor, the processor performs: image acquisition processing of acquiring intraluminal images captured in time series by a scope of an endoscope; scope information acquisition processing of acquiring scope information regarding a change in the scope; landmark recognition processing of recognizing a landmark in the intraluminal image; scope information correction processing of correcting the scope information using information about the landmark recognized through the landmark recognition processing; and depth information acquisition processing of acquiring depth information of the intraluminal image using the intraluminal images and the scope information corrected through the scope information correction processing.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program, and more particularly to an image processing device, an image processing method, and a program for acquiring depth information of an intraluminal image. [Background technology]

[0002] In observations using an endoscopic system (endoscopic device), a technique is known that displays the position of the endoscope of the endoscopic system, the shape of the lumen, and the location of a lesion in a corresponding manner. This technique can effectively assist the user in comprehensively observing the lumen (e.g., the large intestine) that is the object of observation. In order to geometrically grasp the current position of the endoscope, the shape of the lumen, and the location of a lesion, it is necessary to accurately estimate the depth from the camera provided at the tip of the scope to the target object.

[0003] Patent Document 1 proposes a technology for acquiring distance information (depth information) based on brightness information of an endoscopic image and constructing a three-dimensional image. Patent Document 1 also describes a technology for acquiring the amount of change in the axial direction and the amount of change in the circumferential direction of an endoscope using a motion detection sensor, and correcting the unfolded image based on the acquired amount of change. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2007 / 139187 Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology described above, in order to obtain accurate depth information, it is necessary to accurately obtain scope information related to changes in the endoscope scope (e.g., the insertion length of the scope into the lumen, the bending angle and amount of rotation of the scope within the lumen).

[0006] However, because the lumen observed by the endoscope system is a non-rigid body, the scope information of the actual measurement values ​​acquired by the sensor or the like described in Patent Document 1 may not be accurate. That is, the scope information acquired by the sensor or the like may differ from the actual relative amount of change of the scope within the lumen. In this way, if the scope information cannot be obtained accurately, the depth information obtained using the scope information will also be inaccurate.

[0007] The present invention has been made in consideration of the above circumstances, and its purpose is to provide an image processing device, an image processing method, and a program that can acquire accurate scope information and obtain highly accurate depth information from intraluminal images. [Means for solving the problem]

[0008] An image processing device, which is one aspect of the present invention for achieving the above-mentioned object, is an image processing device equipped with a processor, which performs image acquisition processing to acquire time-series intraluminal images taken with an endoscopic scope, scope information acquisition processing to acquire scope information related to changes in the scope, landmark recognition processing to recognize landmarks in the intraluminal images, scope information correction processing to correct the scope information using information related to the landmarks recognized in the landmark recognition processing, and depth information acquisition processing to acquire depth information of the intraluminal images using the intraluminal images and the scope information corrected in the scope information correction processing.

[0009] According to this aspect, landmarks in an intraluminal image are recognized, and the scope information is corrected using information about the recognized landmarks. This makes it possible to obtain accurate depth information of the intraluminal image based on accurate scope information.

[0010] Preferably, the scope information acquisition process acquires the amount of change in the insertion length of the scope at time T+α, and the amount of change in the curvature and rotation of the scope, using the position of the scope at time T as a reference.

[0011] Preferably, the scope information acquisition process acquires information relating to the insertion length of the scope and the amount of curvature and rotation of the scope from the operation of the operation unit of the scope.

[0012] Preferably, the landmark recognition process recognizes changes over time in corresponding points of the landmarks, and the scope information correction process corrects the scope information using the changes over time in the corresponding points.

[0013] Preferably, the landmark recognition process outputs the recognition reliability of the recognized landmark, and the scope information correction process determines whether or not to correct the scope information based on the recognition reliability, and performs the correction based on the result of the determination.

[0014] According to this aspect, the recognition reliability of the landmark is output, and whether or not to correct the scope information is determined based on the recognition reliability, thereby enabling accurate correction to be performed and obtaining accurate distance information.

[0015] Preferably, the scope information correction process outputs a correction value obtained from information relating to the landmark, determines whether or not to perform correction based on the correction value, and performs correction based on the result of the determination.

[0016] According to this aspect, a correction value is output from information about landmarks, and whether or not to perform correction is determined based on the output correction value, thereby enabling accurate correction to be performed and obtaining accurate distance information.

[0017] Preferably, the processor performs display control processing to display geometric information relating to the lumen on the display unit based on the depth information acquired by the depth information acquisition processing.

[0018] According to this aspect, geometric information relating to the lumen is displayed on the display unit based on the acquired depth information, so that accurate geometric information relating to the scope can be provided to the user.

[0019] Preferably, the geometric information is at least one of the shape of a lumen, the position of a lesion, the position of a scope, and the position of a treatment tool.

[0020] Another aspect of the image processing method of the present invention is an image processing method using an image processing device equipped with a processor, in which the processor performs an image acquisition step of acquiring time-series intraluminal images taken with an endoscopic scope, a scope information acquisition step of acquiring scope information related to changes in the scope, a landmark recognition step of recognizing landmarks in the intraluminal images, a scope information correction step of correcting the scope information using information related to the landmarks recognized in the landmark recognition step, and a depth information acquisition step of acquiring depth information of the intraluminal images using the intraluminal images and the scope information corrected in the scope information correction step.

[0021] Another aspect of the present invention is a program that causes an image processing method to be executed using an image processing device equipped with a processor, and causes the processor to execute an image acquisition step of acquiring time-series intraluminal images taken with an endoscopic scope, a scope information acquisition step of acquiring scope information related to changes in the scope, a landmark recognition step of recognizing landmarks in the intraluminal images, a scope information correction step of correcting the scope information using information related to the landmarks recognized in the landmark recognition step, and a depth information acquisition step of acquiring depth information of the intraluminal images using the intraluminal images and the scope information corrected in the scope information correction step. [Effects of the Invention]

[0022] According to the present invention, landmarks in an intraluminal image are recognized, and scope information is corrected using information about the recognized landmarks, so that accurate depth information of an intraluminal image can be obtained based on accurate scope information. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram showing a schematic diagram of an endoscopic image and a depth image obtained from the endoscopic image. [Figure 2] FIG. 2 shows geometric information about the large intestine and corresponding endoscopic images. [Figure 3] FIG. 3 is a diagram for explaining how depth information is obtained from an endoscopic image and scope information. [Figure 4] FIG. 4 is a schematic diagram showing the overall configuration of an endoscope system including an image processing device. [Figure 5] FIG. 5 is a block diagram illustrating an embodiment of an image processing device. [Figure 6] FIG. 6 is a diagram for explaining acquisition of scope information. [Figure 7] FIG. 7 is a diagram for explaining acquisition of scope information. [Figure 8] FIG. 8 is a diagram illustrating an example of a landmark. [Figure 9] FIG. 9 is a diagram illustrating information about landmarks. [Figure 10] FIG. 10 is a diagram illustrating the acquisition of a depth image from corrected scope information and an intraluminal image. [Figure 11] FIG. 11 is a diagram showing an example of geometric information F of a lumen displayed on the display unit. [Figure 12] FIG. 12 is a diagram illustrating the flow of obtaining depth information. [Figure 13] FIG. 13 is a diagram illustrating the flow of obtaining depth information. [Figure 14] FIG. 14 is a flow diagram illustrating an image processing method. [Figure 15] FIG. 15 is a flow diagram illustrating an image processing method. DETAILED DESCRIPTION OF THE INVENTION

[0024] Preferred embodiments of an image processing apparatus, an image processing method, and a program according to the present invention will be described below with reference to the accompanying drawings.

[0025] [Acquisition of depth information from endoscopic images] First, the acquisition of depth information from an endoscopic image will be described.

[0026] FIG. 1 is a diagram showing a schematic diagram of an endoscopic image and a depth image obtained from the endoscopic image.

[0027] FIG. 1 shows an intraluminal image P, which is an example of an endoscopic image acquired by the endoscope system 9 (FIG. 4). The intraluminal image P is an image obtained by capturing, for example, the inside of a large intestine. In the intraluminal image P, folds 101 in the large intestine are shown, and the tubular shape continues in the direction of the arrow. The depth image I is an image having depth information corresponding to the intraluminal image P. The depth image I has information about the depth (distance) from a camera (e.g., the image sensor 28 (FIG. 4)). In the depth image I, the depth information is shown in a heat map format. Note that the depth image I is shown in a simplified format, specifically, seven regions having different depth information from each other are shown. Note that the depth image I may actually display depth information in a heat map format for finer regions, or, for example, different depth information may be displayed for each pixel. In addition, in this example, a case where the large intestine is observed by the endoscope system 9 will be described as an example of an intraluminal image, but examples of intraluminal images are not limited to this. The intraluminal image may also be an image of another luminal organ.

[0028] Typically, the depth information described above is acquired from the depth image I using images from multiple viewpoints whose relative positional relationships are known, such as a stereo camera. However, since the endoscope system 9 is equipped with a monocular camera, when acquiring depth information, it is necessary to acquire the depth information based on the endoscopic image acquired by the monocular camera.

[0029] For example, the literature (Daniel Freedman et al., “Detecting Deficient Coverage in Colonoscopies”, CVPR2020, https: / / arxiv.org / pdf / 2001.08589.pdf) describes a technology that uses a recognizer configured with a CNN (Convolutional Neural Network) to acquire depth images with depth information from endoscopic images acquired with a monocular camera.

[0030] When obtaining depth information using only endoscopic images captured by the monocular camera, the relative depth is estimated by estimating the amount of movement of the endoscope 10 (see FIG. 4) between adjacent frames. In this case, since organs are non-rigid bodies, the shape of the lumen may deform from frame to frame, which may result in errors in the depth information.

[0031] FIG. 2 shows geometric information about the large intestine and corresponding endoscopic images.

[0032] The geometric information 500 relating to the large intestine indicates the current position of the insertion section 20 of the endoscope 10. Also, an endoscopic image acquired at the position of the endoscope 10 indicated in the geometric information 500 of the large intestine is shown.

[0033] As shown in FIG. 2(A), an intraluminal image P1 is acquired at the position of the insertion section 20 indicated in the geometric information 500. As shown in FIG. 2(B), an intraluminal image P2 is acquired when the endoscope 10 is translated, bent, and rotated from the position indicated in FIG. 2(A). Between the intraluminal image P1 and the intraluminal image P2, the viewpoint and the shape of the lumen change due to the movement of the endoscope 10 and the influence of the observation target (large intestine), which is a non-rigid body. In such a case, if depth information is acquired using only the intraluminal image captured by the monocular camera described above, errors may occur in the depth information.

[0034] As described above, when obtaining depth information from an intraluminal image, since the object being observed is a non-rigid body, obtaining depth information from only the intraluminal image may result in reduced accuracy.

[0035] [Acquisition of depth information from endoscopic images and scope information] In order to suppress the above-described deterioration of the depth information, it is conceivable to acquire an actual measurement value of the scope information and acquire the depth information from the scope information and an endoscopic image.

[0036] FIG. 3 is a diagram for explaining how depth information is obtained from an endoscopic image and scope information.

[0037] As shown in FIG. 3, scope information S and an intraluminal image P are input to a depth information acquisition unit 45 (see FIG. 5), which outputs a depth image I having depth information. Here, the scope information S is information indicating changes in the endoscope 10 and is an actual measurement value. For example, the scope information S is the insertion length of the endoscope 10, and the bending angle and rotation amount of the endoscope 10. The depth information acquisition unit 45 is a trained model configured using CNN, and has been trained to output a depth image I in response to input of the scope information S and the intraluminal image P. In this way, by inputting the actual measurement value of the scope information S to the depth information acquisition unit 45, it is possible to calibrate the amount of movement change of the endoscope 10 (more specifically, the position of the image sensor 28), and more accurate depth information can be acquired.

[0038] [Scope information correction] Next, the correction of the above-mentioned scope information S of the actual measurement value will be described.

[0039] The scope information S is an actual measurement value, and basically, there is no error in the amount of change of the endoscope 10 inside the actual lumen. However, because the observation object is a non-rigid organ, the scope information S obtained as an actual measurement value may not necessarily match the relative amount of change of the scope inside the lumen. In such cases, the depth image I output from the depth information acquisition unit 45 based on the scope information S and the intraluminal image P may have low accuracy.

[0040] Here, by correcting the scope information S, which is an actual measurement value, using information about landmarks in the endoscopic image, it is possible to obtain information that is closer to the relative amount of change in the endoscope 10 within the lumen. Therefore, the present invention proposes a method of correcting the scope information S, which is an actual measurement value, using landmarks in the endoscopic image, and acquiring more accurate depth information based on the corrected scope information T.

[0041] First Embodiment [Overall configuration of endoscope system including image processing device] FIG. 4 is a schematic diagram showing the overall configuration of an endoscope system including an image processing device.

[0042] As shown in Figure 4, the endoscope system 9 includes an endoscope scope 10, which is an electronic endoscope, a light source device 11, an endoscope processor device 12, a display device 13, an image processing device 14, an operation unit 15, and a display unit 16.

[0043] The endoscope 10 is a device for capturing time-series endoscopic images including subject images, and is, for example, a scope for use in the lower or upper gastrointestinal tract. The endoscope 10 includes an insertion section 20 that is inserted into a subject (e.g., the stomach) and has a distal end and a proximal end, a handheld operation section 21 that is connected to the proximal end of the insertion section 20 and that is held by a physician as an operator to perform various operations, and a universal cord 22 that is connected to the handheld operation section 21. The endoscope 10 also has a rotation scale 24. The user can obtain the amount of circumferential rotation of the endoscope 10 by reading the rotation scale 24. Here, the circumferential direction refers to the circumferential direction of a circle centered on the axis of the endoscope 10.

[0044] The insertion section 20 is formed in an elongated shape with a small diameter as a whole. The insertion section 20 is configured by sequentially arranging a flexible soft section 25 having flexibility from its base end side to its tip end side, a bending section 26 that can be bent by operating the handheld operation section 21, and a tip section 27 that incorporates an imaging optical system (objective lens) and an imaging element 28 (not shown). The insertion section 20 is provided with a length scale 34 that indicates the insertion length (push amount) of the insertion section 20. The user can obtain the insertion length of the insertion section 20 by reading the length scale 34.

[0045] The imaging element 28 is a CMOS (complementary metal oxide semiconductor) type or CCD (charge coupled device) type imaging element. Image light of the observation site is incident on the imaging surface of the imaging element 28 via an observation window (not shown) opened in the distal end surface of the distal end portion 27 and an objective lens (not shown) arranged behind the observation window. The imaging element 28 captures (converts into an electrical signal) the image light of the observation site incident on its imaging surface and outputs an imaging signal. In other words, the imaging element 28 sequentially captures endoscopic images. The endoscopic images are acquired as video 38 and still images 39, which will be described later.

[0046] The handheld operation unit 21 is provided with various operation members that are operated by a doctor (user). Specifically, the handheld operation unit 21 is provided with two types of bending operation knobs 29 used to bend the bending section 26, an air / water supply button 30 for air / water supply operation, and a suction button 31 for suction operation. The handheld operation unit 21 also is provided with a still image capturing instruction unit 32 for issuing an instruction to capture a still image 39 of the observation site, and a treatment tool introduction port 33 for inserting a treatment tool (not shown) into a treatment tool insertion passage (not shown) that passes through the insertion section 20.

[0047] The universal cord 22 is a connection cord for connecting the endoscope 10 to the light source device 11. The universal cord 22 contains a light guide 35, a signal cable 36, and a fluid tube (not shown) that are inserted through the insertion section 20. The end of the universal cord 22 is provided with a connector 37a that is connected to the light source device 11, and a connector 37b that branches off from the connector 37a and is connected to the endoscope processor device 12.

[0048] By connecting the connector 37a to the light source device 11, the light guide 35 and a fluid tube (not shown) are inserted into the light source device 11. As a result, the necessary illumination light, water, and gas are supplied from the light source device 11 to the endoscope 10 via the light guide 35 and the fluid tube (not shown). As a result, illumination light is irradiated toward the observation site from an illumination window (not shown) on the distal end surface of the tip portion 27. Furthermore, in response to the operation of pressing the air / water supply button 30, gas or water is sprayed from an air / water supply nozzle (not shown) on the distal end surface of the tip portion 27 toward an observation window (not shown) on the distal end surface.

[0049] Connecting the connector 37b to the endoscope processor device 12 electrically connects the signal cable 36 to the endoscope processor device 12. As a result, an image signal of the observation site is output from the imaging element 28 of the endoscope 10 to the endoscope processor device 12 via the signal cable 36, and a control signal is output from the endoscope processor device 12 to the endoscope 10.

[0050] The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37a. The illumination light is selected from various wavelength bands depending on the purpose of observation, such as white light (light in a white wavelength band or light in multiple wavelength bands), light in one or multiple specific wavelength bands, or a combination of these.

[0051] The endoscope processor device 12 controls the operation of the endoscope 10 via the connector 37b and the signal cable 36. The endoscope processor device 12 also generates an image (also called a "video 38") consisting of time-series frame images 38a including an image of a subject, based on an imaging signal acquired from the imaging element 28 of the endoscope 10 via the connector 37b and the signal cable 36. Furthermore, when the still image capture instruction unit 32 is operated on the handheld operation unit 21 of the endoscope 10, the endoscope processor device 12 converts one frame image 38a in the video 38 into a still image 39 corresponding to the timing of the capture instruction, in parallel with the generation of the video 38.

[0052] The moving image 38 and still image 39 are endoscopic images captured inside the subject, i.e., inside the living body. Furthermore, if the moving image 38 and still image 39 are images obtained using light (special light) in the above-mentioned specific wavelength band, both are special light images. The endoscope processor device 12 then outputs the generated moving image 38 and still image 39 to the display device 13 and the image processing device 14.

[0053] The endoscope processor device 12 may generate (acquire) a special light image having information of the specific wavelength band based on the normal light image obtained using the white light. In this case, the endoscope processor device 12 functions as a special light image acquisition unit. The endoscope processor device 12 acquires the signal of the specific wavelength band by performing calculations based on color information of red, green, and blue (RGB (Red, Green, Blue)) or cyan, magenta, and yellow (CMY (Cyan, Magenta, Yellow)) contained in the normal light image.

[0054] The endoscope processor device 12 may also generate a feature image, such as a publicly known oxygen saturation image, based on at least one of the normal light image obtained using the above-mentioned white light and the special light image obtained using light of the above-mentioned specific wavelength band (special light). In this case, the endoscope processor device 12 functions as a feature image generator. Note that the above-mentioned in-vivo image, normal light image, special light image, and video 38 or still image 39 including the feature image are all endoscopic images that are images of the results of capturing or measuring the human body for the purpose of image-based diagnosis or examination.

[0055] The display device 13 is connected to the endoscope processor device 12 and functions as a display unit 16 that displays moving images 38 and still images 39 input from the endoscope processor device 12. The doctor (user) performs operations such as moving forward and backward of the insertion unit 20 while checking the moving images 38 displayed on the display device 13. If a lesion or the like is found in the observation area, the doctor operates the still image capture instruction unit 32 to capture a still image of the observation area, and also performs treatment such as diagnosis and biopsy. Note that the moving images 38 and still images 39 are also displayed on the display unit 16, which is connected to the image processing device 14, which will be described later. When the moving images 38 and still images 39 are displayed on the display unit 16, a notification display, which will be described later, is also displayed at the same time. Therefore, it is preferable that the user perform diagnosis or the like while looking at the display on the display unit 16.

[0056] [Image processing device] 5 is a block diagram showing an embodiment of the image processing device 14. The image processing device 14 sequentially acquires endoscopic images in time series and displays the endoscopic images and geometric information related to the lumen on the display unit 16. The image processing device 14 is composed of, for example, a computer. The operation unit 15 includes a keyboard, a mouse, etc. connected to the computer by wire or wirelessly, as well as buttons provided on the handheld operation unit 21 of the endoscope 10, and the display unit 16 can be any of various monitors such as an LCD monitor connectable to the computer.

[0057] The image processing device 14 is composed of an image acquisition unit 40, a CPU (Central Processing Unit) 41, a scope information acquisition unit 42, a landmark recognition unit 43, a scope information correction unit 44, a depth information acquisition unit 45, a display control unit 46, an audio control unit 47, and a memory 48. The processing of each unit is realized by one or more processors. Here, the processor may be composed of the CPU 41, or may be composed of one or more CPUs not shown.

[0058] The CPU 41 operates based on various programs including an operation system and an endoscopic image processing program stored in the memory 48, and controls the image acquisition unit 40, the scope information acquisition unit 42, the landmark recognition unit 43, the scope information correction unit 44, the depth information acquisition unit 45, the display control unit 46, and the audio control unit 47, and also functions as part of each of these units.

[0059] The image acquisition unit 40 performs image acquisition processing and sequentially acquires time-series endoscopic images. The image acquisition unit 40 acquires time-series endoscopic images including a subject image from the endoscope processor device 12 (FIG. 4) using an image input / output interface (not shown) that is connected to the endoscope processor device 12 by wire or wirelessly. In this example, a moving image 38 captured by the endoscope scope 10 is acquired. Furthermore, if the above-mentioned still image 39 is captured by the endoscope scope 10 while capturing the moving image 38, the image acquisition unit 40 acquires the moving image 38 and the still image 39 from the endoscope processor device 12. Note that this example will be described using an intraluminal image P (FIG. 1) captured of the large intestine as an example of an endoscopic image.

[0060] The memory 48 includes a flash memory, a ROM (Read-only Memory), a RAM (Random Access Memory), a hard disk drive, etc. The flash memory, ROM, and hard disk drive are non-volatile memories that store an operating system, various programs such as an endoscopic image processing program, and captured still images 39. The RAM is a volatile memory that is capable of high-speed reading and writing of data and functions as an area for temporarily storing the various programs stored in the non-volatile memory and as a working area for the CPU 41.

[0061] The scope information acquisition unit 42 performs a scope information acquisition process to acquire scope information related to changes in the endoscope 10. The scope information is information indicating the operation of the insertion section 20 of the endoscope 10. Specifically, the scope information includes an insertion length indicating the length to which the insertion section 20 of the endoscope 10 is pushed into a lumen, a bending angle indicating the bending of the bending section 26, and a rotation amount indicating the circumferential rotation of the endoscope 10. The scope information S can be acquired by actual measurement, and the scope information acquisition unit 42 can acquire the scope information S using various methods. For example, when the scope information acquisition unit 42 acquires the insertion length, the insertion length may be acquired by capturing an image of the length scale 34 on the insertion section 20 with a camera, or the insertion length may be acquired by a sensor (not shown) provided together with the length scale 34. For example, when the scope information acquisition unit 42 acquires the bending angle, the bending angle may be acquired based on the amount of rotation of the bending operation knob 29, or the bending angle may be acquired by a sensor (not shown) provided in the bending section 26. For example, when the scope information acquisition unit 42 acquires the amount of rotation, it may photograph the rotation scale 24 provided on the endoscope 10 with a camera and acquire the amount of rotation read from the camera, or it may acquire the amount of rotation in the circumferential direction of the endoscope 10 using a gyro sensor (not shown) built into the handheld operating unit 21.

[0062] 6 and 7 are diagrams for explaining acquisition of scope information S. Fig. 6 is a diagram for explaining acquisition of the insertion length in the scope information S, and Fig. 7 is a diagram for explaining acquisition of the bending angle and rotation amount in the scope information S.

[0063] 6, at time T, the insertion section 20 is inserted into the lumen by a length a. At time T+α, the insertion section 20 is inserted into the lumen by a length a+b. The scope information acquisition unit 42 acquires, as scope information S, the amount of change in the insertion length at time T+α, based on the insertion length of the insertion section 20 at time T. That is, the scope information acquisition unit 42 acquires, as scope information S, length b as the amount of change in the insertion length.

[0064] 7, at time T, the tip of the insertion section 20 has a bending angle of 0° and a circumferential rotation amount of 0. At time T+α, the tip of the insertion section 20 has a bending angle c and a circumferential rotation amount d. The scope information acquisition unit 42 acquires, as scope information S, the amount of change in the bending angle and the amount of rotation at time T+α, based on the bending angle and the amount of rotation at time T. In this case, the scope information acquisition unit 42 acquires, as scope information S, c as the amount of change in the bending angle and d as the amount of change in the amount of rotation in the circumferential direction.

[0065] The landmark recognition unit 43 (Figure 5) performs landmark recognition processing to recognize landmarks in endoscopic images. Here, a landmark is a point that serves as a landmark in an endoscopic image, and by tracking the landmark over time, the movement (amount of change) of the endoscope 10 can be grasped. Specific examples of landmarks include folds in the large intestine or duodenum, lesions such as polyps, and the start, end, and midpoint of an organ (in the case of the large intestine, the splenic flexure, hepatic flexure, ileocecal junction, etc.). The landmark recognition unit 43 can recognize landmarks in endoscopic images using various techniques. For example, the landmark recognition unit 43 is configured with a recognizer (trained model) that is configured with CNN and has undergone machine learning, and recognizes landmarks from the input endoscopic image.

[0066] FIG. 8 is a diagram illustrating an example of a landmark recognized by the landmark recognition unit 43.

[0067] The landmark recognition unit 43 recognizes landmarks L, which are lesions, in the intraluminal image P. When the landmark recognition unit 43 is configured as a recognizer, it may output a score regarding the recognition of the landmark L. This score is used as the recognition reliability described in the second embodiment.

[0068] The scope information correcting unit 44 (FIG. 5) performs a scope information correction process and corrects the scope information S using information about the landmarks recognized by the landmark recognizing unit 43. Here, the information about the landmarks specifically refers to information about changes over time in the positions of the landmarks recognized in each of the time-series consecutive intraluminal images P. The scope information correcting unit 44 can correct the scope information S using the information about the landmarks in various ways. For example, the scope information correcting unit 44 acquires corrected scope information T by replacing the amount of change in the endoscope 10 obtained based on the information about the landmarks with the scope information acquired by the scope information acquiring unit 42.

[0069] FIG. 9 is a diagram illustrating information about landmarks acquired by the scope information corrector 44. As shown in FIG.

[0070] As will be described below, the scope information corrector 44 tracks the landmarks in time series, uses the depth information corresponding to the landmarks as information about the landmarks, and acquires the amount of change of the endoscope 10 inside the lumen.

[0071] First, the landmark recognition unit 43 recognizes a landmark L in the intraluminal image P1 at time T. The landmark recognition unit 43 also recognizes the landmark L (a point corresponding to the landmark L) in the depth image I1 corresponding to the intraluminal image P1. Then, the scope information correction unit 44 acquires depth information of the landmark L at time T.

[0072] At time T+α, the landmark recognition unit 43 also recognizes the landmark L recognized at time T in the intraluminal image P2. The landmark recognition unit 43 also recognizes the landmark L (the point corresponding to the landmark L) in the depth image I2 corresponding to the intraluminal image P2. The scope information correction unit 44 then acquires depth information of the landmark L at time T+α. Thereafter, the scope information correction unit 44 acquires a change X in the insertion length of the endoscope 10, a change Y in the bending angle of the scope, and a change Z in the amount of rotation in the circumferential direction of the scope based on the temporal change in the depth information of the landmark L (the point corresponding to the landmark L) from time T to time T+α. Note that, in this example, the change amounts X, Y, and Z are acquired based on the temporal change in the position of the landmark L at times T and T+α, but the change amounts X, Y, and Z may also be acquired based on the temporal change in the position of the landmark L at three or more times.

[0073] Then, the scope information correcting unit 44 corrects the scope information by the amount of change in the scope acquired based on the landmark L. For example, the scope information correcting unit 44 replaces the scope information S acquired by the scope information acquiring unit 42 with the amount of change in the endoscope 10 acquired based on the landmark L. Specifically, the scope information correcting unit 44 corrects the amount of change b in the insertion length acquired by the scope information acquiring unit 42 to the amount of change X in the insertion length based on the landmark. The scope information correcting unit 44 also corrects the amount of change c in the bending angle acquired by the scope information acquiring unit 42 to the amount of change Y in the bending angle based on the landmark. The scope information correcting unit 44 also corrects the amount of change d in the circumferential direction acquired by the scope information acquiring unit 42 to the amount of change Z in the circumferential direction based on the landmark.

[0074] In the above example, the amount of change in the endoscope 10 is obtained using landmarks in the depth image, but this is not limited to this. For example, the movement of the landmark in the intraluminal image P and the amount of change in the endoscope 10 may be estimated by machine learning or the like. A recognizer is prepared that has previously performed machine learning on the movement (amount of change) of the landmark in the intraluminal image P in response to multiple patterns of change in the endoscope 10 (translation, rotation, and curvature). Then, this recognizer calculates and estimates the amount of translation of the endoscope 10 from the amount of change in the landmark in the intraluminal image P from time T to T+α.

[0075] The depth information acquisition unit 45 (FIG. 5) performs depth information acquisition processing to acquire depth information of the endoscopic image based on the endoscopic image and the scope information T corrected by the scope information correction unit 44. The depth information acquisition unit 45 is a trained model configured by CNN and subjected to machine learning. The depth information acquisition unit 45 receives the corrected scope information T (or scope information S) and time-series intraluminal images as input, and outputs a depth image having depth information.

[0076] FIG. 10 is a diagram illustrating how a depth image I is obtained from the corrected scope information T and the intraluminal image P. In FIG.

[0077] 10, the depth information acquisition unit 45 receives the scope information T corrected by the scope information correction unit 44 and the intraluminal image P. The scope information T corrected by the scope information correction unit 44 more accurately indicates the amount of change in the scope within the lumen. Therefore, the depth information acquisition unit 45 can output a depth image I having more accurate depth information.

[0078] The display control unit 46 (FIG. 5) generates image data for display based on the endoscopic image (video 38) acquired by the image acquisition unit 40, and outputs the image data to the display unit 16. The display control unit 46 also generates geometric information related to the lumen and outputs the information to the display unit 16. The audio control unit 47 (FIG. 5) controls the audio output from the speaker 17. For example, the audio control unit 47 controls the speaker 17 to output an alarm sound to the user.

[0079] FIG. 11 is a diagram showing an example of an intraluminal image P and geometric information F of the lumen displayed on the display unit 16. As shown in FIG.

[0080] The main area of ​​the display unit 16 displays an intraluminal image P captured by the endoscope system 9. The sub-area of ​​the display unit 16 displays geometric information F of the lumen. The geometric information F of the lumen is generated by the display control unit 46 and output to the display unit 16. The geometric information F is generated based on highly accurate depth information acquired by the depth information acquisition unit 45, based on the corrected scope information T. The geometric information F indicates the shape of the lumen to be observed (shape of the large intestine) and the current position of the endoscope 10. The geometric information F may also indicate the position of a lesion, the position of a treatment tool, etc. In this way, by acquiring highly accurate depth information by the depth information acquisition unit 45, the geometric information F using that depth information can accurately indicate position information, etc.

[0081] [Image processing method and program] Next, an image processing method using the image processing device 14 and a program for causing the image processing device 14 to execute the image processing method will be described.

[0082] 12 and 13 are diagrams illustrating the flow of acquiring depth information. Fig. 12 is a diagram showing the flow of data in the functional block diagram of the image processing device 14. Fig. 13 is a flowchart showing an image processing method using the image processing device 14.

[0083] First, the image acquiring unit 40 acquires an intraluminal image P (image acquiring step: step S101). Here, the intraluminal image P is a frame image 38a that constitutes the video 38. Furthermore, the scope information acquiring unit 42 acquires scope information S (scope information acquiring step: step S102). Next, the landmark recognizing unit 43 recognizes landmarks L in the intraluminal image P (landmark recognizing step: step S103). Thereafter, the scope information correcting unit 44 corrects the scope information S (scope information correcting step: step S104). For example, the scope information correcting unit 44 acquires scope information T, which is the amount of change in the endoscope 10 acquired based on the landmarks L. Then, the depth information acquiring unit 45 acquires a depth image I having depth information of the intraluminal image using the intraluminal image P and the scope information T (depth information acquiring step: step S105). Thereafter, the display unit 16 displays geometric information relating to the lumen on the basis of the depth information (display control processing step: step S106).

[0084] As described above, in this embodiment, landmarks in an intraluminal image are recognized, and information about the recognized landmarks is used to correct the scope information S. This makes it possible to obtain accurate depth information of the intraluminal image based on the corrected, accurate scope information T.

[0085] In the above-described embodiment, an example has been described in which scope information T is acquired by correcting the scope information S acquired by the scope information acquisition unit 42 using information related to landmarks. However, there are cases in which the scope information S acquired by the scope information acquisition unit 42 does not need to be corrected. In other words, when accurate scope information T can be obtained by performing correction by the scope information correction unit 44, it is preferable to correct the scope information S. Such an embodiment will be described below.

[0086] <Second embodiment> Next, a second embodiment will be described. In this embodiment, the scope information corrector 44 corrects the scope information S in accordance with the recognition reliability of the landmark.

[0087] FIG. 14 is a flowchart showing an image processing method according to this embodiment.

[0088] First, the image acquisition unit 40 acquires an intraluminal image P (step S201). Further, the scope information acquisition unit 42 acquires scope information S (step S202). Next, the landmark recognition unit 43 recognizes landmarks L in the intraluminal image P (step S203).

[0089] Next, the landmark recognition unit 43 acquires the recognition reliability of the recognized landmark (step S204). Here, the recognition reliability of the landmark can be acquired by various methods. For example, the landmark recognition unit 43 is configured with a recognizer (trained model) that has undergone machine learning, and the score when the landmark is recognized can be used as the recognition reliability of the landmark.

[0090] Thereafter, the scope information correcting unit 44 determines whether the recognition reliability of the landmark is equal to or greater than a threshold (step S205). If the recognition reliability of the landmark is less than the threshold, the scope information correcting unit 44 does not correct the scope information S. In this case, the depth information acquiring unit 45 acquires depth information using the uncorrected scope information S of the actual measurement value (step S207). On the other hand, if the recognition reliability of the landmark is equal to or greater than the threshold, the scope information correcting unit 44 corrects the scope information S (step S206). Then, the depth information acquiring unit 45 acquires depth information based on the corrected scope information T (step S207). Thereafter, the display unit 16 displays geometric information related to the lumen on the display unit 16 based on the depth information (step S208).

[0091] As described above, if the landmark recognition unit 43 accurately recognizes a landmark, the scope information correction unit 44 can accurately obtain the amount of change in the endoscope 10 based on that landmark. On the other hand, if the landmark recognition unit 43 cannot accurately recognize a landmark, it may be difficult for the scope information correction unit 44 to accurately obtain the amount of change in the endoscope 10 based on that landmark. Therefore, in this embodiment, the scope information S is corrected according to the recognition reliability of the landmark, making it possible to obtain highly accurate depth information.

[0092] <Third embodiment> Next, a third embodiment will be described. In this embodiment, scope information is corrected according to a correction value obtained by the scope information corrector 44.

[0093] FIG. 15 is a flowchart showing an image processing method according to this embodiment.

[0094] First, the image acquisition unit 40 acquires an intraluminal image P (step S301). Further, the scope information acquisition unit 42 acquires scope information S (step S302). Next, the landmark recognition unit 43 recognizes landmarks L in the intraluminal image P (step S303). Next, the scope information correction unit 44 acquires a correction value (step S304).

[0095] The scope information corrector 44 outputs a correction value obtained from the information about the landmark. For example, as described in the first embodiment, when the information about the landmark is depth information corresponding to the landmark L, the scope information corrector 44 acquires the amount of change of the endoscope 10 obtained based on the information about the landmark, and acquires the difference between the scope information T corrected using the amount of change and the scope information S before correction as a correction value. The scope information corrector 44 then determines whether the correction value is equal to or greater than a threshold (step S305). When the correction value is less than the threshold, the scope information corrector 44 does not correct the scope information S, and the depth information acquirer 45 acquires the depth information (step S307). On the other hand, when the correction value is equal to or greater than the threshold, the scope information corrector 44 corrects the scope information S (step S306), and the depth information acquirer 45 acquires the depth information (step S307). Thereafter, the display unit 16 displays geometric information about the lumen on the display unit 16 based on the depth information (step S308).

[0096] As described above, when the correction value is equal to or greater than the threshold value, there is a large difference between the scope information S, which is an actual measurement value, and the scope information T, which has been corrected with the amount of change in the endoscope 10 obtained based on the landmarks L. Therefore, the scope information correction unit 44 corrects the scope information S. On the other hand, when the correction value is less than the threshold value, there is a small difference between the scope information S, which is an actual measurement value, and the scope information T, which has been corrected with the amount of change in the endoscope 10 obtained based on the landmarks L. Therefore, even if the scope information S is used as is, there is little effect on the accuracy of the depth information, and so the scope information correction unit 44 does not correct the scope information S. As a result, in this embodiment, the scope information is corrected according to the correction value, making it possible to efficiently acquire highly accurate depth information.

[0097] [others] In the above embodiment, the hardware structure of the processing units (e.g., image acquisition unit 40, scope information acquisition unit 42, landmark recognition unit 43, scope information correction unit 44, depth information acquisition unit 45, display control unit 46, and audio control unit 47) that perform various processes is made up of various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for performing specific processes.

[0098] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0099] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0100] The above-described configurations and functions can be realized by any hardware, software, or a combination of both. For example, the present invention can be applied to a program that causes a computer to execute the above-described processing steps (processing procedures), a computer-readable recording medium (non-transitory recording medium) on which such a program is recorded, or a computer on which such a program can be installed.

[0101] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0102] 9: Endoscope system 10: Endoscope 11:Light source device 12: Endoscope processor device 13:Display device 14: Image processing device 15:Operation section 16:Display section 17: Speaker 20: Insertion section 21: Handheld control unit 22: Universal Code 24: Rotation scale 25: Soft part 26: Curved section 27:Tip 28: Image sensor 29: Curvature control knob 30: Air and water supply button 31: Suction button 32: Still image shooting instruction section 33: Treatment tool introduction port 34: Length scale 35: Light guide 36: Signal cable 37a: Connector 37b: Connector 40: Image acquisition unit 41: CPU 42: Scope information acquisition section 43: Landmark recognition unit 44: Scope information correction section 45: Depth information acquisition section 46: Display control section 47: Audio control section 48: Memory

Claims

1. An image processing device including a processor and a memory storing a program, The processor executes the program, an image acquisition process for acquiring time-series intraluminal images taken by an endoscope; a scope information acquisition process for acquiring scope information relating to the change in the scope; a landmark recognition process for recognizing landmarks in the intraluminal image; a scope information correction process for correcting the scope information using information about the landmark recognized in the landmark recognition process; a depth information acquisition process for acquiring depth information of the intraluminal image using the intraluminal image and the scope information corrected by the scope information correction process; An image processing device that performs the above.

2. The image processing device according to claim 1 , wherein the scope information acquisition process acquires the amount of change in the insertion length of the scope at time T+α, and the amount of change in curvature and rotation of the scope, based on the position of the scope at time T.

3. The image processing device according to claim 1 , wherein the scope information acquisition process acquires information about the insertion length of the scope and the curvature and rotation of the scope from the operation of an operation unit of the scope.

4. the landmark recognition processing recognizes a change over time in the corresponding points of the landmarks; The image processing device according to claim 1 , wherein the scope information correction process corrects the scope information using a change over time of the corresponding points.

5. The landmark recognition processing outputs a recognition reliability of the recognized landmark, The image processing device according to claim 1 , wherein the scope information correction process determines whether or not to correct the scope information based on the recognition reliability, and performs the correction based on a result of the determination.

6. 6. The image processing device according to claim 1, wherein the scope information correction process outputs a correction value obtained from information related to the landmark, determines whether or not to perform correction based on the correction value, and performs the correction based on a result of the determination.

7. The processor: The image processing device according to claim 1 , further comprising: a display control process for displaying geometric information relating to a lumen on a display unit based on the depth information acquired by the depth information acquisition process.

8. The image processing device according to claim 7 , wherein the geometric information is at least one of the shape of the lumen, the position of the lesion, the position of the scope, and the position of a treatment tool.

9. An image processing method using an image processing device equipped with a processor, by the processor an image acquisition step of acquiring time-series intraluminal images taken with an endoscopic scope; a scope information acquisition step of acquiring scope information relating to a change in the scope; a landmark recognition step of recognizing landmarks in the intraluminal image; a scope information correcting step of correcting the scope information using information about the landmark recognized in the landmark recognizing step; a depth information acquiring step of acquiring depth information of the intraluminal image using the intraluminal image and the scope information corrected in the scope information correcting step; An image processing method in which

10. A program for causing an image processing method to be executed using an image processing device having a processor, the processor, an image acquisition step of acquiring time-series intraluminal images taken with an endoscopic scope; a scope information acquisition step of acquiring scope information relating to a change in the scope; a landmark recognition step of recognizing landmarks in the intraluminal image; a scope information correcting step of correcting the scope information using information about the landmark recognized in the landmark recognizing step; a depth information acquiring step of acquiring depth information of the intraluminal image using the intraluminal image and the scope information corrected in the scope information correcting step; A program that executes the following.

11. A non-transitory computer-readable recording medium on which the program according to claim 10 is recorded.

Images