Image processing device, endoscope system, image processing method, and recording medium

Abstract

This image processing device includes a processor, wherein the processor acquires an image of the inside of a digestive tract captured by an endoscope inserted into the digestive tract, determines whether or not the depth direction of the digestive tract is visible in the acquired endoscopic image, outputs endoscope operation information, which relates to the operation of the endoscope, in association with the endoscopic image when having determined that the depth direction of the digestive tract is visible in the endoscopic image, and does not output the endoscope operation information when having determined that the depth direction of the digestive tract is not visible in the endoscopic image.

Description

Image Processing Apparatus, Endoscope System, Image Processing Method, and Recording Medium

[0001] The present invention relates to an image processing apparatus, an endoscope system, an image processing method, and a recording medium.

[0002] In endoscopy, due to imaging omissions caused by structural dead angles or biases in the observation direction, etc., there is a possibility that a part of the surface of the inner wall of the digestive tract may not be observed. Therefore, it is known to detect an unobserved area in the digestive tract that has not been observed and present the detected unobserved area on a monitor (see, for example, Patent Document 1). In this document, a three-dimensional organ model of the digestive tract is generated from an image of the inner wall of the digestive tract obtained by an endoscope, and missing parts where the organ model cannot be generated are specified as unobserved areas.

[0003] U.S. Patent No. 10,682,108

[0004] However, in the image processing apparatus of Patent Document 1, although processing is performed to exclude images that are inappropriate as targets for the organ model, such as images with blur or defocus, images with moisture or feces remaining, and images with a strong proximity of the endoscope to the inner wall of the digestive tract, etc., the timing at which it is easy for the operator to identify the unobserved area and the determination processing of that timing, etc., are not mentioned.

[0005] An object of the present invention is to provide an image processing apparatus, an endoscope system, an image processing method, and a recording medium that can reliably inform an operator of operation information of an endoscope such as an unobserved area without interfering with the observation.

[0006] One aspect of the present invention is an image processing apparatus including a processor, the processor acquiring an image obtained by an endoscope inserted into the digestive tract, determining whether the depth direction of the digestive tract is reflected in the acquired image, and when it is determined that the depth direction is reflected in the image, outputting endoscope operation information related to the operation of the endoscope in association with the image, and when it is determined that the depth direction is not reflected in the image, not outputting the endoscope operation information.

[0007] Another aspect of the present invention is an endoscope system comprising the above-mentioned image processing device, an endoscope connected to the image processing device, and a display device connected to the image processing device.

[0008] Another aspect of the present invention is an image processing method that includes acquiring an image obtained by an endoscope inserted into the digestive tract, determining whether the depth direction of the digestive tract is visible in the acquired image, outputting endoscopic operation information relating to the operation of the endoscope in association with the image if it is determined that the depth direction is visible in the image, and not outputting the endoscopic operation information if it is determined that the depth direction is not visible in the image.

[0009] Furthermore, another aspect of the present invention is a non-temporary recording medium that stores an image processing program that causes a computer to perform the following actions: acquire an image obtained by an endoscope inserted into the digestive tract; determine whether or not the depth direction of the digestive tract is visible in the acquired image; if it is determined that the depth direction is visible in the image, output endoscope operation information relating to the operation of the endoscope in association with the image; and if it is determined that the depth direction is not visible in the image, not output the endoscope operation information.

[0010] This is an overall configuration diagram showing an endoscope system according to the first embodiment of the present invention. This is a functional block diagram showing an image processing device provided in the endoscope system of Figure 1. This is a flowchart explaining the image processing method according to the first embodiment of the present invention. This is a flowchart explaining the process performed in step SA2 of Figure 3. This is a diagram showing an example of an endoscope image in which the depth direction of the digestive tract is determined to be visible by the image processing method of Figure 3. This is a diagram showing an example of a depth map generated based on the endoscope image of Figure 5. This is a diagram showing an example of an endoscope image in which the depth direction of the digestive tract is determined not to be visible by the image processing method of Figure 3. This is a diagram showing an example of a depth map generated based on the endoscope image of Figure 7. This is a diagram showing an example of an endoscope image in which an arrow indicating an unobserved area is superimposed on the endoscope image of Figure 5. This is a flowchart explaining an image processing method according to a modified example of the first embodiment of the present invention. This is a diagram showing an example of a depth map based on an endoscope image in which the depth direction of the digestive tract is determined to be visible by the image processing method of Figure 10. This is a diagram showing an example of a depth map based on an endoscope image in which the depth direction of the digestive tract is determined not to be visible by the image processing method of Figure 10. This is a flowchart explaining the image processing method according to the second embodiment of the present invention. Figure 13 shows an example of an endoscopic image in which the number of folds is determined to be above a predetermined fold threshold by the image processing method of Figure 13. Figure 13 shows an example of an endoscopic image in which the number of folds is determined to be below a predetermined fold threshold by the image processing method of Figure 13. Figure 20 shows an example of a three-dimensional organ model generated by the image processing method according to the third embodiment of the present invention. This is a flowchart illustrating the image processing method according to the third embodiment of the present invention. Figure 17 shows an example of an endoscopic image in which the depth direction of the digestive tract is determined to be visible by the image processing method of Figure 17. Figure 17 shows an example of an endoscopic image in which the depth direction of the digestive tract is determined to be not visible by the image processing method of Figure 17. Figure 17 illustrates the predetermined first and second thresholds used in the image processing method of Figure 17 using a three-dimensional organ model. Figure 20 illustrates the predetermined first and second thresholds at an angle viewed along the central axis of the digestive tract. Figure 20 illustrates a trained model used in the image processing method according to the fourth embodiment of the present invention.This figure shows an example of an endoscopic image classified into the first image group, in which the depth direction of the digestive tract is visible and that depth direction is located near the center of the endoscopic image. This figure shows an example of an endoscopic image classified into the second image group, in which the depth direction of the digestive tract is visible, but that depth direction is not located near the center of the endoscopic image. This figure shows an example of an endoscopic image classified into the third image group, in which the depth direction of the digestive tract is not visible and the mucosal surface of the inner wall of the digestive tract is visible. This figure shows an example of an endoscopic image classified into the fourth image group, in which the depth direction of the digestive tract is not visible and the mucosal surface of the inner wall of the digestive tract is not visible. This is a flowchart illustrating the image processing method according to the fifth embodiment of the present invention.

[0011] [First Embodiment] An endoscope system, image processing device, image processing method, and storage medium according to the first embodiment of the present invention will be described below with reference to the drawings. The endoscope system 1 according to this embodiment comprises an endoscope 2, an image processing device 3, and a display device 4, as shown in Figure 1. The display device 4 is, for example, a monitor.

[0012] The endoscope 2 includes an insertion section 2a that is inserted into the patient's body. The insertion section 2a includes a light source that generates illumination light, a light guide fiber that guides the illumination light emitted from the light source from the base end to near the tip of the insertion section 2a, an illumination optical system that irradiates the illumination light guided by the light guide fiber from the tip of the insertion section 2a toward the subject (digestive tract) inside the body, an objective lens that collects the reflected light from the subject that has been irradiated with illumination light, and an image sensor (all not shown) that acquires an endoscopic image (image) G0 of the digestive tract by capturing the reflected light collected by the objective lens. The endoscope 2 outputs the endoscopic image G0 acquired by the image sensor to the image processing device 3.

[0013] The endoscope 2 is supported, for example, by a robotic arm (not shown). The robotic arm is, for example, an electrically operated holder of a general-purpose 6-axis articulated robot that holds the endoscope 2 in a movable position. The robotic arm is equipped with a motor (drive unit) for each joint that operates each joint.

[0014] The endoscope 2 and the robotic arm are controlled by a control device (not shown). The control device is implemented, for example, by a dedicated or general-purpose computer. Specifically, the control device consists of a controller such as a CPU (Central Processing Unit) and a storage device such as RAM (Random Access Memory) and HDD (Hard Disk Drive).

[0015] As shown in Figure 2, the image processing device 3 comprises an input unit 5, a processor 6 which is a central processing unit, a memory 7, a storage unit 8, and an output unit 9. The input unit 5 has a known input interface and is connected to the endoscope 2. Sequential frames of the endoscopic image G0 output from the endoscope 2 are input to the image processing device 3 in chronological order through the input unit 5. The output unit 9 has a known output interface and is connected to the display device 4. The endoscopic image processed by the image processing device 3 is output chronologically from the output unit 9 to the display device 4 and then displayed on the display device 4.

[0016] Memory 7 consists of a volatile memory device called RAM and is used as a workspace for the processor 6. The storage unit 8 consists of a computer-readable non-temporary recording medium such as ROM, flash memory, or a hard disk drive. The storage unit 8 stores an image processing program 8a that causes the processor 6 to execute an image processing method.

[0017] Next, the image processing method executed by the processor 6 will be described. The processor 6 executes the image processing method shown below in response to a start trigger. The start trigger is input to the image processing device 3, for example, by an operator such as a physician operating an input device 10 such as a switch provided on the endoscope 2.

[0018] As shown in Figure 3, the processor 6 first acquires information about the endoscope 2 supported by the robotic arm (step SA1). The endoscope information includes, for example, the model number and serial number. Next, the processor 6 determines whether or not the depth direction of the digestive tract is visible in the endoscopic image G0 acquired by the endoscope 2 inserted into the patient's body, for example, the digestive tract (step SA2).

[0019] In step SA2, as shown in Figure 4, the processor 6 first acquires an endoscopic image G0 of the inside of the digestive tract obtained by the endoscope 2 (step SB1). Next, the processor 6 generates a depth map representing the depth of each part of the digestive tract by estimating the depth of each part based on the acquired endoscopic image G0 (step SB2). Next, the processor 6 calculates the depth of the deepest region in the depth map (step SB3). Next, the processor 6 determines whether the calculated deepest region is deeper than a predetermined depth threshold (step SB4).

[0020] If the processor 6 determines that the depth of the deepest region is greater than or equal to the depth threshold, it determines that the endoscopic image G0 shows the depth direction of the digestive tract (step SB5). For example, in the case of the endoscopic image G0 shown in Figure 5 and the depth map shown in Figure 6, where the depth direction of the digestive tract is shown near the center, it is determined that the depth direction of the digestive tract is shown because the depth of the region near the center of the image is greater than a predetermined depth threshold.

[0021] On the other hand, if the processor 6 determines that the depth of the deepest region is shallower than the depth threshold, it determines that the depth direction of the digestive tract is not shown in the endoscopic image G0 (step SB6). For example, in the case of the endoscopic image G0 shown in Figure 7 and the depth map shown in Figure 8, where the mucosal surface of the inner wall of the digestive tract is shown overall, it is determined that the depth direction of the digestive tract is not shown because there are no regions deeper than a predetermined depth threshold.

[0022] If the endoscopic image G0 is composed of, for example, 8-bit data, and the depth of the region closest to the image sensor (i.e., the shallowest region) is set to 0, and the depth of the region furthest from the image sensor (i.e., the deepest region) is set to 255, then the predetermined depth threshold is, for example, 150. By pre-setting an appropriate depth threshold according to the observation area of ​​the digestive tract, it is possible to easily determine whether or not the image shows the depth direction of the digestive tract. Note that the endoscopic image G0 may also be composed of 16 bits. Depth values ​​may be normalized from 0 to 1, or output as inverse depth (for example, smaller values ​​indicate greater distance, and larger values ​​indicate closer). In these cases, the depth threshold may also be adjusted according to the scale and the relationship between the magnitudes of the depths. Furthermore, the depth threshold may be adjusted depending on the training data or model used.

[0023] If step SB5 determines that the depth direction is visible in the endoscopic image G0, the process returns to the flowchart in Figure 3, and the processor 6 outputs endoscopic operation information relating to the operation of the endoscope 2, associated with the endoscopic image G0 (step SA3). On the other hand, if step SB6 determines that the depth direction is not visible in the endoscopic image G0, the processor 6 does not output endoscopic operation information. Endoscopic operation information is, for example, information indicating unobserved areas within the digestive tract that have not been observed.

[0024] The generation and output of information indicating unobserved areas is carried out as follows: First, the processor 6 performs a three-dimensional reconstruction of the digestive tract by processing the endoscopic image G0 acquired by the endoscope 2 while observing the inside of the digestive tract with the endoscope 2. As a result, the processor 6 generates an organ model consisting of multiple point clouds in three dimensions of the digestive tract in real time as the observation progresses.

[0025] The three-dimensional organ model is generated, for example, by Visual Simultanate Localization and Mapping (SLAM). The tip position and orientation of the insertion section 2a of the endoscope 2 may be estimated by processing with Visual SLAM, or information input from a tip position detection device (not shown) may be used.

[0026] Next, the processor 6 identifies unobserved regions of the generated three-dimensional organ model that have not been observed by the endoscope 2. The identification of unobserved regions is performed by designating the constructed regions of the organ model as observed regions and the unconstructed regions of the organ model as unobserved regions. For example, unobserved regions that are outside the field of view of the endoscope 2, such as the back of the folds of the inner wall of the digestive tract, become unconstructed regions of the organ model, that is, regions where the organ model is missing (missing parts). Therefore, the unobserved regions are identified by linking the missing regions to the organ model.

[0027] Next, the processor 6 generates an arrow pointing to the identified unobserved area as information indicating the unobserved area. Then, the processor 6 generates an endoscopic image G1 by superimposing the generated arrow A onto the endoscopic image G0 in a manner that points to the unobserved area, as shown in Figure 9, for example. The generated endoscopic image G1 is output from the processor 6 to the display device 4 and displayed by the display device 4. In this way, information indicating the unobserved area is communicated to the operator.

[0028] Next, the processor 6 checks whether a termination instruction has been input by the input device 10 (step SA4). If a termination instruction has been input, the processor 6 terminates the process; otherwise, it repeats the process from step SA2.

[0029] Thus, only when an endoscopic image G0 showing the depth of the digestive tract is acquired by the endoscope 2, an arrow A indicating an unobserved area is superimposed on the endoscopic image G0 displayed by the display device 4.

[0030] When an endoscopic image G0 is acquired that does not show the depth direction of the digestive tract, for example, when the image sensor of endoscope 2 is oriented in a direction intersecting the depth direction of the digestive tract or facing the inner wall of the digestive tract, it is highly likely that the operator is adjusting the orientation of endoscope 2 or concentrating on observing the lesion. On the other hand, when an endoscopic image G0 that shows the depth direction of the digestive tract is acquired, that is, when the image sensor of endoscope 2 is oriented in the depth direction of the digestive tract, it is highly likely that the operator's operation of endoscope 2 is stable or that the position of endoscope 2 is stable.

[0031] Therefore, according to this embodiment, when the depth direction of the digestive tract is visible on the endoscopic image G0, an arrow A indicating an unobserved area is superimposed on the endoscopic image G0, thereby more reliably informing the operator of the unobserved area without interfering with the observation. Furthermore, the operator can more reliably identify the unobserved area and quickly repeat the observation of the unobserved area.

[0032] In this embodiment, arrow A is used as an example of information indicating an unobserved area, but the information indicating an unobserved area only needs to be something that allows the operator to visually recognize the unobserved area on the endoscopic image G0. For example, text indicating that it is an unobserved area may be displayed on the endoscopic image G0 along with arrow A. Alternatively, instead of arrow A, the unobserved area may be enclosed in a frame such as a circle on the endoscopic image G0, or the color of the unobserved area may be made different from the color of the observed area.

[0033] Furthermore, in this embodiment, unobserved regions were identified by not constructing organ models for regions corresponding to unobserved areas at the same time as generating the organ models. Alternatively, for example, the organ models for regions corresponding to unobserved areas may be omitted after the entire organ model has been generated.

[0034] Furthermore, in this embodiment, the processor 6 determines whether the depth direction of the digestive tract is reflected in the endoscopic image G0 based on whether the depth of the deepest region in the depth map is greater than or equal to a predetermined depth threshold. Alternatively, the processor 6 may determine whether the depth of the deepest region is greater than or equal to a predetermined depth threshold, and whether the distance of the deepest region from the vicinity of the center of the endoscopic image G0 is less than or equal to a predetermined distance threshold.

[0035] In this case, for example as shown in Figure 10, if in step SB4 the depth of the deepest region is determined to be greater than or equal to a predetermined depth threshold, the processor 6 calculates the distance of the deepest region from the vicinity of the center of the endoscopic image G0 (step SB4'). Next, the processor 6 determines whether the calculated distance is less than or equal to a predetermined distance threshold (step SB4'').

[0036] Then, the processor 6 determines that the depth of the digestive tract is reflected in the endoscopic image G0 if the depth of the deepest region is greater than or equal to the depth threshold and the distance from the center of the endoscopic image G0 is less than or equal to the distance threshold (step SB5). For example, in the case of the depth map shown in Figure 11, where the depth of the digestive tract is reflected near the center, the depth of the region near the center of the image is greater than or equal to a predetermined depth threshold, and the position of that region is less than or equal to a predetermined distance threshold from the center of the image, so it is determined that the depth of the digestive tract is reflected.

[0037] On the other hand, if the processor 6 determines that the distance from the center of the endoscopic image G0 is longer than a distance threshold, it determines that the depth direction of the digestive tract is not visible in the endoscopic image G0 (step SB6). For example, in the case of the depth map shown in Figure 12, where the depth direction of the digestive tract is visible in the corner of the image away from the center, although the depth of the area in the corner of the image is greater than a predetermined depth threshold, the position of that area is further from the center of the image than a predetermined distance threshold, so it is determined that the depth direction of the digestive tract is not visible.

[0038] According to this modification example, the endoscopic image G0 determined to show the depth direction of the digestive tract is limited to the endoscopic image G0 in which the depth direction is shown near the center of the image. In the endoscopic image G0 where the depth direction of the digestive tract is shown near the center, it is easy to observe the inner wall of the digestive tract in the depth direction, and it is also easy to grasp the positions of the folds of the surrounding inner wall centered on the depth direction. Also, at the timing when the tip of the endoscope 2 is arranged near the center inside the digestive tract, it is easy to operate the tip of the endoscope 2 without interfering with the inner wall of the digestive tract. Therefore, it is possible to easily execute the operation of the endoscope 2 according to the information indicating the unobserved area.

[0039] In this embodiment, it is determined whether or not the depth direction of the digestive tract is shown in the endoscopic image G0 as 0 or 1. Instead of this, for example, it may be possible to perform scoring based on the depth value and select the endoscopic image G0 with the highest score in a certain section of the observation range as the image showing the depth direction of the digestive tract.

[0040] 〔Second Embodiment〕 Next, an endoscopic system, an image processing apparatus, an image processing method, and a recording medium according to the second embodiment of the present invention will be described below with reference to the drawings. The image processing apparatus 3 and the image processing method according to this embodiment are different from the first embodiment in that it is determined whether or not the depth direction of the digestive tract is shown in the endoscopic image G0 based on the number of folds of the inner wall of the digestive tract in the endoscopic image G0. In the description of this embodiment, the same reference numerals are given to the parts having the same configuration as those of the first embodiment described above, and the description thereof is omitted.

[0041] As shown in FIG. 13, the image processing method according to this embodiment is such that the processor 6 detects the folds of the inner wall of the digestive tract in the endoscopic image G0 acquired in step SB1 (step SC2). Next, the processor 6 counts the number of detected folds by means of labeling processing or the like (step SC3). Next, the processor 6 determines whether or not the counted number of folds is equal to or greater than a predetermined fold number threshold (step SC4).

[0042] When the processor 6 determines that the number of folds in the endoscopic image G0 is greater than or equal to the fold number threshold, it determines that the depth direction of the digestive tract is reflected in the endoscopic image G0 (step SA5). For example, in the case of the endoscopic image G0 shown in FIG. 14 where the depth direction of the digestive tract is reflected near the center, there are a predetermined number or more of folds extending in the circumferential direction around the center at intervals in a direction away from the vicinity of the center of the image, so it is determined that the depth direction of the digestive tract is reflected. In FIG. 14, the folds are shown in white on the endoscopic image G0. The same applies to FIG. 15.

[0043] On the other hand, when the processor 6 determines that the number of folds in the endoscopic image G0 is less than the fold number threshold, it determines that the depth direction of the digestive tract is not reflected in the endoscopic image G0 (step SB6). For example, in the case of the endoscopic image G0 shown in FIG. 15 where only one fold is reflected, since there are less than a predetermined number of folds, it is determined that the depth direction of the digestive tract is not reflected. The fold number threshold is, for example, four. It is also possible to count not only a complete circular fold but also a substantially arc-shaped fold with a part of the circumferential direction cut off as one fold.

[0044] On the inner wall of the digestive tract such as the large intestine, folds exist at substantially regular intervals in the depth direction. Therefore, according to the present embodiment, by presetting an appropriate fold number threshold according to the observation region of the digestive tract, it is possible to easily determine whether the depth direction of the digestive tract is reflected in the image. Since the interval between folds varies for each subject or for each part of the large intestine, the fold number threshold is set to an appropriate value according to the conditions.

[0045] 〔Third Embodiment〕Next, an endoscopic system, an image processing apparatus, an image processing method, and a recording medium according to the third embodiment of the present invention will be described below with reference to the drawings. The image processing apparatus 3 and the image processing method according to the present embodiment are different from the first embodiment in that it is determined whether the depth direction of the digestive tract is reflected in the endoscopic image G0 based on a three-dimensional organ model M of the digestive tract as shown in FIG. 16. In the description of the present embodiment, the same reference numerals are given to the parts having the same configuration as those in the first embodiment described above, and the description thereof is omitted.

[0046] In this embodiment, as shown in Figure 17, the image processing method involves the processor 6 processing the endoscopic image G0 acquired in step SB1 to generate a three-dimensional organ model M of the digestive tract (step SD2). The above visual SLAM calculates a matrix containing multiple point clouds in three dimensions, as well as the XYZ position and orientation of the image sensor of the endoscope 2 in each endoscopic image G0.

[0047] Next, the processor 6 estimates the centerline C extending in the depth direction of the digestive tract and the orientation of the image sensor of the endoscope 2 based on the generated organ model M (step SD3). For example, an approximate centerline C is estimated by dividing the organ model M into fixed sections along its longitudinal direction and connecting the centroids of each section.

[0048] Next, the processor 6 determines the perpendicular distance d (for example, d) from the position of the image sensor of the endoscope 2 to the center line C of the digestive tract in the organ model M. １ d ２ ), and the direction vector c of the image sensor (for example, c １ , c ２ ) and the angle θ (for example, θ) made by a straight line parallel to the center line C of the digestive tract １ , θ ２ The processor 6 then calculates the perpendicular distance d and the angle θ, respectively (step SD4). The processor 6 then determines whether the calculated perpendicular distance d is less than or equal to a predetermined first threshold, and whether the calculated angle θ is less than or equal to a predetermined second threshold (step SD5).

[0049] The processor 6 determines that the depth direction of the digestive tract is being captured if the perpendicular distance d is less than or equal to the first threshold and the angle θ is less than or equal to the second threshold (step SB5). For example, in the case of the endoscopic image G0 shown in Figure 18, where the depth direction of the digestive tract is captured near the center, the processor determines that the depth direction of the digestive tract is being captured because the perpendicular distance d is less than or equal to the first threshold and the angle θ is less than or equal to the second threshold.

[0050] On the other hand, processor 6 determines that the depth direction of the digestive tract is not visible in endoscopic image G0 if the perpendicular distance d is longer than the first threshold and / or the angle θ is greater than the second threshold (step SB6). For example, in the case of endoscopic image G0 shown in Figure 19, where the mucosal surface of the inner wall of the digestive tract is shown in its entirety, it is determined that the depth direction of the digestive tract is not visible because the perpendicular distance d is greater than the first threshold and the angle θ is greater than the second threshold.

[0051] The first threshold is, for example, r / 2, where r is the distance between any point P1 on the center line C of the organ model M shown in Figure 20 and the nearest three-dimensional point P2 from that point P1, as shown in Figure 21. The second threshold is, for example, 15°.

[0052] In this embodiment, the processor 6 generates a three-dimensional organ model M in real time while observing the inside of the digestive tract with the endoscope 2, and estimates the centerline C of the digestive tract. The processor 6 then sequentially calculates the perpendicular distance d and angle θ of the image sensor of the endoscope 2 to the centerline C, and when it is determined that the depth direction of the digestive tract is visible in the endoscopic image G0 based on the calculated perpendicular distance d and angle θ, information indicating the unobserved area is displayed in the endoscopic image G0.

[0053] The closer the position of the image sensor of the endoscope 2 is to the centerline C of the digestive tract, and the smaller the inclination of the image sensor with respect to the centerline C of the digestive tract, the closer the depth direction of the digestive tract will be displayed to the center of the endoscopic image G0. Therefore, according to this embodiment, by pre-setting appropriate first and second thresholds according to the observation area of ​​the digestive tract, it is possible to easily determine whether or not the endoscopic image shows the depth direction of the digestive tract.

[0054] In this embodiment, the current endoscopic image G0 is processed in real time, and information indicating unobserved areas is displayed on the endoscopic image G0. Alternatively, when the insertion portion 2a of the endoscope 2 has advanced a certain distance within the digestive tract, the centerline C of the digestive tract up to that point may be estimated, and endoscopic images G0 that are determined to show the depth direction of the digestive tract up to that point may be extracted. The current endoscopic image G0 acquired by the endoscope 2 is displayed in real time on a main monitor (not shown) of the display device 4, while endoscopic images G0 that were extracted retrospectively and determined to show the depth direction of the digestive tract may be displayed on a sub-monitor (not shown) of the display device 4 along with information indicating unobserved areas at the time of extraction.

[0055] [Fourth Embodiment] Next, an endoscope system, image processing device, image processing method, and recording medium according to the fourth embodiment of the present invention will be described below with reference to the drawings. The image processing device 3 and image processing method according to this embodiment differ from the first embodiment in that the processor 6 switches whether or not to output endoscope operation information based on the determination result of the learning model. In the description of this embodiment, parts that have the same configuration as the first embodiment described above are denoted by the same reference numerals and their description is omitted.

[0056] The image processing device 3 stores a trained model in its memory unit 8 that uses machine learning, such as deep learning, to determine whether or not the depth direction of the digestive tract is visible in the endoscopic image G0. The processor 6 inputs the endoscopic image G0 acquired by the endoscope 2 into the trained model.

[0057] The trained model can include, for example, a neural network which is a classification model such as Resnet or EfficientNet. The neural network has, for example, an input layer 11 into which an endoscopic image G0 is input, an intermediate layer 12 which performs calculations based on the output from the input layer 11, and an output layer 13 which outputs a determination result of whether or not the depth direction of the digestive tract is visible in the endoscopic image G0, based on the output from the intermediate layer 12. In Figure 22, a network with two intermediate layers 12 is shown as an example, but the intermediate layers 12 may be one layer or three or more layers. Also, the number of nodes (neurons) included in each layer is not limited to the example in Figure 22 and various modifications are possible.

[0058] Furthermore, the trained model uses machine learning to label and classify the input endoscopic images G0 into the following four groups. For example, an image like the endoscopic image G0 shown in Figure 23, which shows the depth direction of the digestive tract and where that depth direction is located near the center of the endoscopic image G0, is classified into the first image group. Also, an image like the endoscopic image G0 shown in Figure 24, which shows the depth direction of the digestive tract but where that depth direction is located in a corner far from the center of the endoscopic image G0, is classified into the second image group. Also, an image like the endoscopic image G0 shown in Figure 25, which does not show the depth direction of the digestive tract but where the mucosal structure of the inner wall of the digestive tract is clearly visible without being distorted, is classified into the third image group. Also, an image like the endoscopic image G0 shown in Figure 26, which does not show the depth direction of the digestive tract and does not show the mucosal surface of the inner wall of the digestive tract, is classified into the fourth image group. Furthermore, even if the inner wall of the digestive tract is visible, images that are excessively close to the mucosa or that are not clearly visible due to blurring or other reasons are also classified into the fourth group. Whether or not the image is excessively close to the mucosa may be determined using the actual distance from the tip of the insertion section 2a of the endoscope 2 to the inner wall of the digestive tract and a predetermined proximity threshold. Whether or not the mucosa is clearly visible may be determined using the amount of blurring or distortion in the image and a predetermined clarity threshold. For example, the softmax function or the like outputs category probability values ​​such that the sum of the four image groups from the first to the fourth image group is 1, and the endoscopic image G0 is classified into the class with the highest probability for that probability value. Annotation is performed subjectively by an observer, such as a developer or a physician.

[0059] The trained model then determines that endoscopic images G0 classified into the first image group show the depth direction of the digestive tract, while endoscopic images G0 classified into any of the second to fourth image groups do not show the depth direction of the digestive tract.

[0060] The processor 6 outputs endoscopic operation information associated with the endoscopic image G0 classified into the first image group, based on the judgment results of the trained model. On the other hand, if the endoscopic image G0 is classified into any of the second to fourth image groups, the processor 6 does not output endoscopic operation information.

[0061] Endoscopic images G0 classified into the second image group are used, for example, for initializing the self-position and point cloud of the endoscope 2 when generating a three-dimensional organ model M, and for processing when determining unobserved areas. Endoscopic images G0 classified into the third image group are used for generating the organ model M after initialization. Endoscopic images G0 classified into the fourth image group are unsuitable images, such as those that are blurry or out of focus, have residual water or stool, or the endoscope is too close to the inner wall of the digestive tract, and are therefore not used for generating the organ model M.

[0062] According to this embodiment, by pre-training the depth direction of the digestive tract as seen in the image using a trained model, it is possible to easily determine whether or not the depth direction of the digestive tract is visible in the endoscopic image G0.

[0063] [Fifth Embodiment] Next, an endoscope system, image processing device, image processing method, and recording medium according to the fifth embodiment of the present invention will be described below with reference to the drawings. The image processing device 3 and image processing method according to this embodiment differ from the first embodiment in that the endoscope operation information is a control signal that advances the insertion portion 2a of the endoscope 2 in the insertion direction. In the description of this embodiment, parts that have the same configuration as the first embodiment described above are denoted by the same reference numerals and their description is omitted.

[0064] After acquiring information from the endoscope 2 (step SA1), as shown in Figure 27, for example, the processor 6 determines whether the depth direction of the digestive tract is visible in the endoscopic image G0 acquired by the endoscope 2 (step SA2) in a method similar to that of any of the first to fourth embodiments described above.

[0065] If the processor 6 determines in step SA2 that the depth direction is visible in the endoscopic image G0, it outputs a control signal to the robot arm motor to advance the insertion section 2a in the insertion direction (step SE3). As a result, the insertion section of the endoscope 2 is automatically inserted into the digestive tract in the depth direction by the operation of the robot arm. Then, the process proceeds to step SA4.

[0066] On the other hand, if step SA2 determines that the depth direction is not visible in the endoscopic image G0, the processor 6 does not output a control signal to advance the insertion section 2a in the insertion direction. In this case, the processor 6 outputs an adjustment signal to the robot arm motor to adjust the tilt of the insertion section 2a, that is, the angle around an axis perpendicular to the longitudinal axis of the insertion section 2a, so that, for example, the depth direction of the digestive tract is visible in the endoscopic image G0 (step SE4). As a result, the tilt of the insertion section of the endoscope 2 is adjusted by the operation of the robot arm. Then, the process returns to step SA2.

[0067] According to this embodiment, a control signal is output to advance the insertion section 2a in the insertion direction only when an endoscopic image showing the depth direction of the digestive tract is acquired. Therefore, when the endoscope 2 is facing the depth direction of the digestive tract, the insertion section 2a can be automatically inserted in the depth direction of the digestive tract.

[0068] In this embodiment, the processor 6 may perform the passage of the endoscope 2 through the bends in the digestive tract by using, for example, the structural analysis type shown below. First, structural information within the digestive tract is extracted based on each endoscopic image G0 acquired by the endoscope 2. Next, the edge of the bend in the digestive tract is moved to near the center of the endoscopic image G0. Next, the possible directions in which the depth direction of the digestive tract exists are stored. Next, as the tip of the insertion part 2a of the endoscope 2 is inserted into the bend in the digestive tract, the angle of the insertion part 2a of the endoscope 2 is adjusted to face the direction in which the stored depth direction of the digestive tract may exist.

[0069] The present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, some components of all the components shown in the embodiments may be deleted. Moreover, components from different embodiments may be appropriately combined.

[0070] 1 Endoscopy system 2 Endoscope 3 Image processing device 4 Display device 6 Processor 8 Storage unit (recording medium) 8a Image processing program C Centerline G0 Endoscopic image (image) G1 Endoscopic image (image) M Organ model

Claims

1. An image processing device comprising a processor, the processor acquires an image obtained by an endoscope inserted into the digestive tract, determines whether the depth direction of the digestive tract is visible in the acquired image, outputs endoscope operation information relating to the operation of the endoscope in association with the image if it is determined that the depth direction is visible in the image, and does not output the endoscope operation information if it is determined that the depth direction is not visible in the image.

2. The image processing apparatus according to claim 1, wherein the processor further determines, based on the acquired image, whether the tip of the endoscope is located near the center of the cross-section of the digestive tract perpendicular to the depth direction, and outputs the endoscope operation information associated with the image if the depth direction is visible in the image and the tip is located near the center, and does not output the endoscope operation information if the processor determines that the tip is not located near the center.

3. The image processing apparatus according to claim 1 or 2, wherein the endoscopic operation information is information indicating an unobserved area within the gastrointestinal tract that has not been observed, and the processor notifies the operator of the information indicating the unobserved area.

4. The image processing apparatus according to claim 3, wherein the processor generates a three-dimensional organ model of the digestive tract by processing the image acquired by the endoscope, and identifies the missing portion for which the organ model could not be generated as the unobserved region.

5. The image processing apparatus according to claim 1 or 2, wherein the endoscope operation information is a control signal that advances the insertion portion of the endoscope in the insertion direction, and the processor outputs the control signal to the drive unit of the endoscope.

6. The image processing apparatus according to any one of claims 1 to 5, wherein the processor determines whether the depth of the deepest region in the acquired image is deeper than a predetermined depth threshold, determines that the depth direction is reflected in the image where the depth is determined to be greater than or equal to the depth threshold, and determines that the depth direction is not reflected in the image where the depth is determined to be shallower than the depth threshold.

7. The image processing apparatus according to claim 6, wherein the processor further determines whether the distance from the center of the image of the deepest region is less than or equal to a predetermined distance threshold, determines that the depth direction is reflected in the image if the depth is greater than or equal to the depth threshold and the distance is less than or equal to the distance threshold, and determines that the depth direction is not reflected in the image if the distance is longer than the distance threshold.

8. The image processing apparatus according to any one of claims 1 to 5, wherein the processor determines whether the number of folds on the inner wall of the digestive tract in the acquired image is equal to or greater than a predetermined fold threshold, determines that the depth direction is reflected in the image in which the number of folds is determined to be equal to or greater than the fold threshold, and determines that the depth direction is not reflected in the image in which the number of folds is determined to be less than the fold threshold.

9. The image processing apparatus according to any one of claims 1 to 5, wherein the processor determines whether the perpendicular distance from the position of the image sensor of the endoscope to the center line extending in the depth direction is less than or equal to a predetermined first threshold in the acquired image, determines whether the angle formed by the direction vector of the image sensor and a straight line parallel to the center line is less than or equal to a predetermined second threshold, determines that the depth direction is reflected in the image in which the perpendicular distance is less than or equal to the first threshold and the angle is less than or equal to the second threshold, and determines that the depth direction is not reflected in the image in which the perpendicular distance is longer than the first threshold and / or the angle is greater than the second threshold.

10. An image processing apparatus according to any one of claims 1 to 4, comprising a storage unit that stores a trained model for determining whether or not the depth direction is captured in the image by machine learning, wherein the processor inputs the acquired image to the trained model and switches whether or not to output the endoscope operation information based on the determination result of the trained model.

11. The image processing apparatus according to claim 10, wherein the trained model can classify the input images into one of the following groups by machine learning: a first image group in which the depth direction is visible and the depth direction is located near the center of the image; a second image group in which the depth direction is visible and the depth direction is not located near the center of the image; a third image group in which the depth direction is not visible and the surface of the mucosal structure of the inner wall of the digestive tract is visible to the extent that it is not distorted; and a fourth image group in which the depth direction is not visible and the mucosal surface of the inner wall of the digestive tract is not visible, or even if the inner wall of the digestive tract is visible, the endoscope is closer to the inner wall than a predetermined proximity threshold, or the amount of image blur or distortion is greater than a predetermined clarity threshold, and the image classified into the first image group is determined to be visible, and the image classified into any of the second to fourth image groups is determined to be not visible.

12. An endoscope system comprising: an image processing device according to any one of claims 1 to 11; an endoscope connected to the image processing device; and a display device connected to the image processing device.

13. An image processing method comprising: acquiring an image obtained by an endoscope inserted into the digestive tract; determining whether the depth direction of the digestive tract is visible in the acquired image; if it is determined that the depth direction is visible in the image, outputting endoscope operation information relating to the operation of the endoscope in association with the image; and if it is determined that the depth direction is not visible in the image, not outputting the endoscope operation information.

14. A non-temporary recording medium storing an image processing program that causes a computer to perform the following actions: acquire an image obtained by an endoscope inserted into the digestive tract; determine whether or not the depth direction of the digestive tract is visible in the acquired image; if it is determined that the depth direction is visible in the image, output endoscope operation information relating to the operation of the endoscope in association with the image; and if it is determined that the depth direction is not visible in the image, not output the endoscope operation information.

Images