Endoscope image processing device and endoscope image processing method

Abstract

In this invention, an image acquisition unit 261 acquires an endoscopic image captured by an endoscope. A region division unit 262 divides the endoscopic image into multiple regions. A depth information generation unit 263 generates depth information indicating the depth of the endoscopic image. A recognition unit 264 specifies the direction in which the endoscope can travel, on the basis of region information indicating the result of the division into the regions made by the region division unit 262 and the depth information of the endoscopic image generated by the depth information generation unit 263.

Description

[Technical field]

[0001] The present disclosure relates to a technique for processing images captured by an endoscope. [Background technology]

[0002] In endoscopic observation, a long and thin flexible insertion section is inserted into a subject to capture images of the inside of the subject. In recent years, research has been conducted into automating the operation of the insertion section, and Patent Document 1 discloses a technique for controlling the bending angle of a bending section in an electronic endoscope device that is provided with a bending section that can be bent up, down, left, and right, so that the tip of the insertion section faces the center of the lumen being photographed.

[0003] In recent years, a method for estimating depth information from an image has been proposed as a deep learning technology (Non-Patent Document 1), and research is also being conducted on generating depth information from endoscopic images (Non-Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent No. 3645223 [Non-patent literature]

[0005] [Non-Patent Document 1] Lei He, Guanghui Wang and Zhanyi Hu,“Learning Depth from Single Images with Deep Neural Network Embedding Focal Length”, 27 Mar 2018<URL:https: / / arxiv.org / pdf / 1803.10039.pdf> [Non-Patent Document 2] Faisal Mahmood, Richard Chen, Nicholas J. Durr, “Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training”, 29 Nov 2017<URL:https: / / arxiv.org / pdf / 1711.06606.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, the darkest part in the video output signal is determined to be the center of the lumen, and the bending angle of the bending section is controlled so that the tip of the endoscope faces the center of the lumen. Therefore, a dark part that is shaded by a structure (e.g., a fold) that protrudes from the tip of the endoscope may be determined to be the center of the lumen, and the tip of the endoscope may be directed to the dark part. Even if the center of the lumen is accurately identified, there are cases in which it is not desirable to advance the tip of the endoscope toward the center of the lumen due to the situation around the tip of the endoscope.

[0007] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide a technique for generating appropriate information regarding the operation or manipulation of an endoscope based on an endoscopic image. [Means for solving the problem]

[0008] In order to solve the above problems, an endoscopic image processing device according to one aspect of the present disclosure includes an image acquisition unit that acquires an endoscopic image captured by an endoscope, a region division unit that divides the endoscopic image acquired by the image acquisition unit into a plurality of regions, a depth information generation unit that generates depth information indicating the depth of the endoscopic image acquired by the image acquisition unit, and a recognition unit that identifies a direction in which the endoscope can proceed based on the region information indicating the results of the region division by the region division unit and the depth information of the endoscopic image generated by the depth information generation unit.

[0009] An endoscopic image processing device according to another aspect of the present disclosure includes an image acquisition unit that acquires an endoscopic image captured by an endoscope, an operation content selection unit that selects one or more operation contents from a predetermined number of operation contents based on the endoscopic image acquired by the image acquisition unit, a region division unit that divides the endoscopic image acquired by the image acquisition unit into a number of regions, a recognition unit that recognizes the situation around the tip of the endoscope based on region information indicating the results of the region division by the region division unit, and an operation content determination unit that determines the operation content to be performed based on the operation content selected by the operation content selection unit and the situation recognized by the recognition unit.

[0010] A method for processing endoscopic images according to yet another aspect of the present disclosure includes acquiring an endoscopic image captured by an endoscope, dividing the acquired endoscopic image into a plurality of regions, generating depth information indicating the depth of the acquired endoscopic image, and determining a direction in which the endoscope can proceed based on the region information indicating the results of the region division and the depth information of the endoscopic image.

[0011] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc. are also valid aspects of the present disclosure. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating a configuration of an endoscope system according to an embodiment. [Diagram 2] 1 is a diagram showing functional blocks of an endoscope system according to an embodiment. [Diagram 3] FIG. 13 is a diagram showing an example of an endoscopic image. [Figure 4] FIG. 2 is a diagram illustrating functional blocks of a control unit according to the first embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of an endoscopic image. [Figure 6] FIG. 13 is a diagram showing an example of an area division result. [Figure 7] FIG. 11 is a diagram showing an example of a depth information estimation result. [Figure 8]11 is a diagram showing an example of a superimposed image obtained by superimposing an area segmentation result image and a depth estimation result image. FIG. [Figure 9] FIG. 13 is a diagram showing an example of a recognition result in a superimposed image. [Figure 10] FIG. 13 is a diagram showing another example of an endoscopic image. [Figure 11] FIG. 13 is a diagram showing an example of an area division result. [Figure 12] FIG. 11 is a diagram showing an example of a depth information estimation result. [Figure 13] FIG. 13 is a diagram showing another example of an endoscopic image. [Figure 14] FIG. 13 is a diagram showing an example of an area division result. [Figure 15] FIG. 11 is a diagram showing an example of a depth information estimation result. [Figure 16] FIG. 13 is a diagram showing another example of an endoscopic image. [Figure 17] FIG. 13 is a diagram showing an example of an area division result. [Figure 18] FIG. 11 is a diagram showing an example of a depth information estimation result. [Figure 19] FIG. 11 is a diagram illustrating functional blocks of a control unit according to a second embodiment. [Figure 20] FIG. 11 is a diagram illustrating an example of teacher data. [Figure 21] FIG. 13 is a diagram showing another example of teacher data. [Figure 22] FIG. 13 is a diagram showing another example of teacher data. [Figure 23] FIG. 13 is a diagram showing another example of teacher data. [Figure 24] FIG. 13 is a diagram showing another example of teacher data. [Diagram 25] FIG. 13 is a diagram showing another example of teacher data. [Figure 26] FIG. 13 is a diagram showing another example of teacher data. [Figure 27] FIG. 13 is a diagram showing another example of teacher data. [Figure 28] FIG. 13 is a diagram showing another example of teacher data. [Figure 29] FIG. 13 is a diagram showing another example of teacher data. [Diagram 30] FIG. 13 is a diagram showing another example of teacher data. [Diagram 31] FIG. 1A shows an example of an endoscopic image, and FIG. 1B shows an example of an image resulting from region segmentation. [Diagram 32] FIG. 13 is a diagram showing a state in which an image resulting from region segmentation is segmented into a plurality of partial regions. [Diagram 33] FIG. 1A shows an example of an endoscopic image, and FIG. 1B shows an example of an image resulting from region segmentation. [Diagram 34] FIG. 13 is a diagram showing a state in which an image resulting from region segmentation is segmented into a plurality of partial regions. [Diagram 35] FIG. 1A shows an example of an endoscopic image, and FIG. 1B shows an example of an image resulting from region segmentation. [Diagram 36] FIG. 13 is a diagram showing a state in which an image resulting from region segmentation is segmented into a plurality of partial regions. [Figure 37] FIG. 13 shows the two end points of a fold edge at a bend and its midpoint. [Figure 38] FIG. 13 is a diagram showing another example of an endoscopic image. [Figure 39] FIG. 13 is a diagram showing an example of an area division result. [Diagram 40] FIG. 11 is a diagram showing an example of a depth information estimation result. [Diagram 41] FIG. 13 is a diagram showing an example of recognition of a depth estimation result image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. 1 shows the configuration of an endoscope system 1 according to an embodiment. The endoscope system 1 is installed in an endoscope examination room, and includes an endoscope control device 2, an endoscope 10, an input device 50, and a display device 60. The endoscope control device 2 has a processing device 20, an insertion shape detection device 30, and an external force information acquisition device 40, and has a function of automatically operating the endoscope 10 inserted into the body of a subject. The automatic operation of the endoscope 10 is performed by the processing device 20, which includes one or more processors 22 and a storage medium 24.

[0014] The input device 50 is an input interface operated by a user, and is configured to output instructions according to user operations to the processing device 20. The input device 50 may include an operating device such as a mouse, a keyboard, a touch panel, etc. The display device 60 is a device that displays an endoscopic image, etc. output from the processing device 20 on a screen, and may be a liquid crystal display or an organic EL display.

[0015] The endoscope 10 has an imaging section including a solid-state imaging element (e.g., a CCD image sensor or a CMOS image sensor). The solid-state imaging element converts incident light into an electrical signal and outputs it to the processing device 20. The processing device 20 has a signal processing section that performs signal processing such as A / D conversion and noise removal on the imaging signal photoelectrically converted by the solid-state imaging element, and generates an endoscopic image. Note that the signal processing section may be provided on the endoscope 10 side, and the endoscope 10 may generate the endoscopic image. The processing device 20 causes the video captured by the endoscope 10 to be displayed on the display device 60 in real time.

[0016] The endoscope 10 includes an insertion section 11 to be inserted into a subject, an operation section 16 provided on the base end side of the insertion section 11, and a universal cord 17 extending from the operation section 16. The endoscope 10 is detachably connected to the processing device 20 by a scope connector (not shown) provided at the end of the universal cord 17.

[0017] The elongated insertion section 11 has, in this order from the distal end to the proximal end, a hard tip section 12, a bending section 13 formed to be freely bendable, and a long, flexible flexible tube section 14. Inside the tip section 12, the bending section 13, and the flexible tube section 14, multiple source coils 18 are arranged at predetermined intervals along the longitudinal direction of the insertion section 11, and the source coils 18 generate a magnetic field in response to a coil drive signal supplied from a processing device 20.

[0018] When a user such as a doctor operates the release switch of the operation unit 16 while the endoscope 10 is inserted into the subject, the processing device 20 captures an endoscopic image and transmits it to an image server (not shown) for recording. The release switch may be provided in the input device 50. A light guide (not shown) is provided inside the endoscope 10 for transmitting illumination light supplied from the processing device 20 to illuminate the inside of the subject, and the tip portion 12 is provided with an illumination window for emitting the illumination light transmitted by the light guide to the subject, and an imaging unit for capturing images of the subject at a predetermined period and outputting an imaging signal to the processing device 20.

[0019] In the embodiment of the endoscopic system 1, the processing device 20 automatically operates the endoscope 10 and automatically controls the operation of the endoscope 10 inside the subject, but it is also possible for a user to hold the operating unit 16 and manually operate the endoscope 10.

[0020] The operation section 16 may include an operation member for the user to operate the endoscope 10. The operation section 16 includes at least an angle knob for bending the bending section 13 in eight directions intersecting with the longitudinal axis of the insertion section 11. An example of the basic operation of the endoscope 10 will be described below. "Advance operation (push operation)" for advancing the insertion part 11 "Retraction operation (pulling operation)" for retracting the insertion part 11 "Angle operation" for bending the curved portion 13 "Twisting operation" for rotating the insertion part 11 around the insertion axis "Air supply operation" for blowing gas forward from the tip 12 "Water supply operation" for spraying liquid forward of the tip 12 "Suction operation" for sucking up tissue fragments and other objects present in the vicinity of the tip portion 12 A "search operation" for searching the center of the lumen by bending the bending portion 13 in multiple directions and directing the tip portion 12 in multiple directions.

[0021] In the embodiment, the up-down direction of the tip portion 12 is set as a direction perpendicular to the insertion axis of the insertion portion 11 and as a direction corresponding to the vertical direction of the solid-state image sensor provided in the imaging section. The left-right direction of the tip portion 12 is set as a direction perpendicular to the insertion axis of the insertion portion 11 and as a direction corresponding to the horizontal direction of the solid-state image sensor provided in the imaging section. Therefore, in the embodiment, the up-down direction of the tip portion 12 coincides with the up-down direction of the endoscopic image output from the signal processing section 220, and the left-right direction of the tip portion 12 coincides with the left-right direction of the endoscopic image.

[0022] The processing device 20 is detachably connected to each of the components of the insertion shape detection device 30, the external force information acquisition device 40, the input device 50, and the display device 60. The processing device 20 receives instructions from the user inputted from the input device 50, and carries out processing corresponding to the instructions. The processing device 20 also acquires an imaging signal periodically outputted from the endoscope 10, and causes the display device 60 to display an endoscopic image.

[0023] The insertion shape detection device 30 has a function of detecting a magnetic field generated by each of the multiple source coils 18 provided in the insertion portion 11, and acquiring the position of each of the multiple source coils 18 based on the strength of the detected magnetic field. The insertion shape detection device 30 generates insertion shape information indicating the acquired positions of the multiple source coils 18, and outputs it to the processing device 20 and the external force information acquisition device 40.

[0024] The external force information acquisition device 40 stores data on the curvature (or curvature radius) and curvature angle at a plurality of predetermined positions of the insertion section 11 in a state where no external force is applied, and data on the curvature (or curvature radius) and curvature angle at the plurality of predetermined positions acquired in a state where a predetermined external force is applied to any position of the insertion section 11 from all possible directions. The external force information acquisition device 40 identifies the positions of the plurality of source coils 18 provided in the insertion section 11 based on the insertion shape information output from the insertion shape detection device 30, and acquires the curvature (or curvature radius) and curvature angle at each position of the plurality of source coils 18. The external force information acquisition device 40 may acquire external force information indicating the magnitude and direction of the external force at each position of the plurality of source coils 18 from the acquired curvature (or curvature radius) and curvature angle and various data stored in advance. The external force information acquisition device 40 outputs the acquired external force information to the processing device 20.

[0025] 2 shows functional blocks of an endoscope system 1 according to an embodiment. The endoscope system 1 includes an endoscope 10, a processing device 20, an insertion shape detection device 30, an external force information acquisition device 40, an input device 50, and a display device 60.

[0026] The endoscope 10 includes a source coil 18, an imaging section 110, an advance / retract mechanism 141, a bending mechanism 142, an AWS mechanism 143, and a rotation mechanism 144. The advance / retract mechanism 141, the bending mechanism 142, the AWS mechanism 143, and the rotation mechanism 144 configure an operating mechanism in the endoscope 10.

[0027] The imaging section 110 has an observation window into which return light from a subject illuminated by illumination light is incident, and a solid-state imaging element (for example, a CCD image sensor or a CMOS image sensor) that captures the return light and outputs an imaging signal.

[0028] The advancing / retracting mechanism 141 has a mechanism for realizing the operation of advancing and retracting the insertion portion 11. For example, the advancing / retracting mechanism 141 may be configured to have a pair of rollers arranged at positions facing each other across the insertion portion 11, and a motor that rotates the pair of rollers. The advancing / retracting mechanism 141 drives the motor in response to an advancing / retracting control signal output from the processing device 20 to rotate the pair of rollers, thereby performing either the operation of advancing the insertion portion 11 or the operation of retracting the insertion portion 11.

[0029] The bending mechanism 142 has a mechanism for realizing an operation of bending the bending portion 13. For example, the bending mechanism 142 may be configured to have a plurality of bending pieces provided in the bending portion 13, a plurality of wires connected to the plurality of bending pieces, and a motor for pulling the plurality of wires. The bending mechanism 142 can bend the bending portion 13 in any of eight directions intersecting with the longitudinal axis of the insertion portion 11 by driving the motor in response to a bending control signal output from the processing device 20 and changing the pulling amount of the plurality of wires.

[0030] The AWS (Air feeding, Water feeding, and Suction) mechanism 143 has a mechanism for realizing an air feeding operation, a water feeding operation, and a suction operation. For example, the AWS mechanism 143 may be configured to have two pipelines, an air / water feeding pipeline and a suction pipeline, provided inside the insertion section 11, the operation section 16, and the universal cord 17, and a solenoid valve that opens one of the two pipelines while closing the other.

[0031] When the AWS mechanism 143 operates the solenoid valve to open the air / water supply pipe in response to the AWS control signal output from the processing device 20, it causes a fluid containing at least one of water and air supplied from the processing device 20 to flow through the air / water supply pipe and discharges the fluid from a discharge port formed in the tip 12. When the AWS mechanism 143 operates the solenoid valve to open the suction pipe in response to the AWS control signal output from the processing device 20, it applies a suction force generated in the processing device 20 to the suction pipe, and sucks in an object present near the suction port formed in the tip 12 with the suction force.

[0032] The rotation mechanism 144 has a mechanism for realizing an operation of rotating the insertion portion 11 around the insertion axis of the insertion portion 11 as a rotation axis. For example, the rotation mechanism 144 may be configured to have a support member that rotatably supports the insertion portion 11 on the base end side of the flexible tube section 14, and a motor for rotating the support member. The rotation mechanism 144 drives the motor in response to a rotation control signal output from the processing device 20 to rotate the support member, thereby rotating the insertion portion 11 around the insertion axis.

[0033] The insertion shape detection device 30 includes a receiving antenna 310 and an insertion shape information acquisition unit 320. The receiving antenna 310 is configured to have a plurality of coils that three-dimensionally detect the magnetic fields generated by the plurality of source coils 18. When the receiving antenna 310 detects the magnetic fields generated by the plurality of source coils 18, the receiving antenna 310 outputs a magnetic field detection signal according to the intensity of the detected magnetic field to the insertion shape information acquisition unit 320.

[0034] The insertion shape information acquisition section 320 acquires the positions of each of the multiple source coils 18 based on the magnetic field detection signal output from the receiving antenna 310. Specifically, the insertion shape information acquisition section 320 acquires multiple three-dimensional coordinate values ​​in a virtual spatial coordinate system having a predetermined position of the subject (such as the anus) as the origin or reference point, as the positions of the multiple source coils 18. The insertion shape information acquisition section 320 generates insertion shape information including the three-dimensional coordinate values ​​of the multiple source coils 18, and outputs the information to the control section 260 and the external force information acquisition device 40.

[0035] The external force information acquisition device 40 acquires the curvature (or curvature radius) and bending angle at each position of the multiple source coils 18 based on the insertion shape information output from the insertion shape detection device 30. The external force information acquisition device 40 may acquire external force information indicating the magnitude and direction of the external force at each position of the multiple source coils 18 from the acquired curvature (or curvature radius) and bending angle and various data stored in advance. The external force information acquisition device 40 outputs the acquired external force information to the control unit 260.

[0036] The processing device 20 includes a light source unit 210, a signal processing unit 220, a coil drive signal generating unit 230, a driving unit 240, a display processing unit 250, and a control unit 260. In the embodiment, the processing device 20 serves as an image processing device that processes endoscopic images. Specifically, the processing device 20 generates information related to the operation or manipulation of the endoscope 10 based on the endoscopic images, and automatically controls the operation of the endoscope 10.

[0037] The light source unit 210 generates illumination light for illuminating the inside of the subject and supplies the illumination light to the endoscope 10. The light source unit 210 may have one or more LEDs or one or more lamps as a light source. The light source unit 210 may change the amount of illumination light in response to an operation control signal supplied from the control unit 260.

[0038] The signal processing unit 220 has a signal processing circuit, performs predetermined processing on the imaging signal output from the endoscope 10 to generate an endoscopic image, and outputs the generated endoscopic image to the display processing unit 250 and the control unit 260.

[0039] The coil drive signal generation unit 230 generates a coil drive signal for driving the source coil 18. The coil drive signal generation unit 230 has a drive circuit, generates a coil drive signal based on an operation control signal supplied from the control unit 260, and supplies the coil drive signal to the source coil 18.

[0040] The drive unit 240 generates a control signal corresponding to a basic operation of the endoscope 10 based on an operation control signal supplied from the control unit 260, and drives the operation mechanism of the endoscope 10. Specifically, the drive unit 240 controls at least one of the advance / retract operation by the advance / retract mechanism 141, the bending operation by the bending mechanism 142, the AWS operation by the AWS mechanism 143, and the rotation operation by the rotation mechanism 144. The drive unit 240 includes an advance / retract drive unit 241, a bending drive unit 242, an AWS drive unit 243, and a rotation drive unit 244.

[0041] The advance / retract drive unit 241 generates and outputs an advance / retract control signal for controlling the operation of the advance / retract mechanism 141, based on the operation control signal supplied from the control unit 260. Specifically, the advance / retract drive unit 241 generates and outputs an advance / retract control signal for controlling the rotation of a motor provided in the advance / retract mechanism 141, based on the operation control signal supplied from the control unit 260.

[0042] The curvature drive unit 242 generates and outputs a curvature control signal for controlling the operation of the curvature mechanism 142 based on the operation control signal supplied from the control unit 260. Specifically, the curvature drive unit 242 generates and outputs a curvature control signal for controlling the rotation of a motor provided in the curvature mechanism 142 based on the operation control signal supplied from the control unit 260.

[0043] The AWS driving unit 243 generates and outputs an AWS control signal for controlling the operation of the AWS mechanism 143, based on the operation control signal supplied from the control unit 260. Specifically, the AWS driving unit 243 generates and outputs an AWS control signal for controlling the operation state of the solenoid valve provided in the AWS mechanism 143, based on the operation control signal supplied from the control unit 260.

[0044] The rotation drive unit 244 generates and outputs a rotation control signal for controlling the operation of the rotation mechanism 144, based on the operation control signal supplied from the control unit 260. Specifically, the rotation drive unit 244 generates and outputs a rotation control signal for controlling the rotation of a motor provided in the rotation mechanism 144, based on the operation control signal supplied from the control unit 260.

[0045] The display processing unit 250 generates a display image including the endoscopic image output from the signal processing unit 220, and causes the generated display image to be displayed on the display device 60. Note that the display processing unit 250 may cause the display device 60 to display a resultant image obtained by processing the endoscopic image by the control unit 260.

[0046] The control unit 260 has a function of generating an operation control signal for causing the endoscope 10 to perform an operation in accordance with instructions from the operation unit 16 and the input device 50, and outputting the signal to the drive unit 240, when the manual insertion mode of the endoscope 10 is set to on. The control unit 260 also has a function of automatically controlling the operation of the endoscope 10 based on an endoscopic image generated by the signal processing unit 220, when the automatic insertion mode of the endoscope 10 is set to on. Before describing the automatic operation control in the embodiment, manual operation of the endoscope by a doctor will be described below.

[0047] In manual insertion mode, the doctor operates the endoscope based on various judgments. The doctor looks at the endoscopic image and makes instantaneous judgments to, for example, avoid obstacles near the tip of the endoscope, avoid touching the tip of the endoscope to the mucosal surface, avoid putting stress on the intestinal tract, and determine the current route by anticipating the route further ahead.

[0048] Fig. 3(a) shows an example of an endoscopic image. The endoscopic image 70a is an image of an intestine (rubber intestine) made of rubber captured by an endoscope. When the doctor looks at the endoscopic image 70a and confirms that the lumen (i.e., the center of the lumen, in other words, the direction of the lumen) is in the center of the image, he or she determines that it is OK to advance the tip of the endoscope, and advances the tip of the endoscope forward.

[0049] FIG. 3(b) shows another example of an endoscopic image. Endoscopic image 70b is also an image of a rubber intestine. The doctor looks at endoscopic image 70b and confirms that the lumen center is at the top of the image, and determines that if the endoscope tip is advanced in this state, it will come into contact with the folds in the center of the image. The doctor then operates the angle knob to bend bending portion 13 upward so that the lumen center is photographed in the center of the image. When the lumen center is photographed in the center of the image, the state becomes the same as that of endoscopic image 70a shown in FIG. 2(a), so the doctor determines that it is OK to advance the endoscope tip, and advances the endoscope tip forward.

[0050] The above judgments and operations can be easily performed by a doctor, but to realize this with a device, it is necessary to identify the center of the lumen from the endoscopic image and recognize and understand the situation around the tip of the endoscope. Therefore, in the embodiment, a technology is proposed for appropriately determining the operation of the endoscope 10 based on the endoscopic image.

[0051] <Example 1> 4 shows functional blocks of the control unit 260 in Example 1. The control unit 260 includes an image acquisition unit 261, an area division unit 262, a depth information generation unit 263, a recognition unit 264, an operation determination unit 265, an operation control unit 266, and a drive determination unit 267.

[0052] The control unit 260 shown in Fig. 4 can be realized in hardware by one or more processors 22, memory, auxiliary storage device, and other LSIs, and in software by a program loaded into memory, but here, functional blocks realized by cooperation of these are depicted. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination of both. For example, a program for executing at least some of the functions of the control unit 260 may be stored in the storage medium 24, and the processor 22 may load the program from the storage medium 24 into the memory to realize each function of the control unit 260.

[0053] The image acquiring unit 261 acquires an endoscopic image captured by the endoscope 10 inserted into the subject from the signal processing unit 220. The imaging unit 110 of the endoscope 10 supplies an imaging signal to the signal processing unit 220 at a predetermined cycle (for example, 30 frames / second), and the signal processing unit 220 generates an endoscopic image from the imaging signal and supplies it to the image acquiring unit 261. Therefore, the image acquiring unit 261 acquires an endoscopic image at a predetermined cycle. The image acquiring unit 261 supplies the acquired endoscopic image to the region dividing unit 262 and the depth information generating unit 263.

[0054] (Area segmentation processing of endoscopic images) The region division unit 262 has a function of dividing the endoscopic image acquired by the image acquisition unit 261 into a plurality of regions. Specifically, the region division unit 262 executes semantic segmentation for labeling each pixel in the endoscopic image, and divides the endoscopic image into regions corresponding to a plurality of predetermined structures. The region division unit 262 defines a region having a structure of a type (class) to be divided, and generates a region division result in which pixels of various structures are labeled. Semantic segmentation is realized using a fully convolutional neural network (FCN) or a bilateral segmentation network (BiSeNet), and the region division unit 262 in the first embodiment may execute semantic segmentation using an FCN.

[0055] As the types (classes) of the divided areas, label values ​​from 0 to 255 may be prepared. In the first embodiment, label values ​​are assigned to the following structures. Label value 0: Background pixel Label value 1: Normal lumen Label value 2: Fold edge (contour) Label value 3: Lumen of bend Label value 4: Bend fold edge Set.

[0056] In semantic segmentation, a label value of 0 generally means a "region not to be extracted," but the label value of 0 defined in Example 1 means a mucosal surface. A "normal lumen" assigned a label value of 1 means a structure in an endoscopic image through which an endoscope can advance, and is defined as a structure indicating the direction of advancement of the tip of the endoscope. Specifically, a structure defined as a "normal lumen" represents the extension direction of the lumen. In addition to these classes, classes may be set for structures such as residues, polyps, and blood vessels that appear in a colonoscopic examination, and label values ​​may be assigned to each of these classes.

[0057] (Depth information generation process for endoscopic images) The depth information generating unit 263 has a function of generating information indicating the depth of the endoscopic image acquired by the image acquiring unit 261. Various methods for estimating the depth of pixels or blocks included in an image have been proposed in the past. Non-Patent Document 2 uses three-dimensional information by CT colonography as teacher data for distance information, and the depth information generating unit 263 may generate information indicating the depth of each pixel of the endoscopic image by using the technology disclosed in Non-Patent Document 2.

[0058] The depth information generating unit 263 may generate a learning model for depth estimation processing based on easily created teacher data. For example, the creator of the teacher data may create the teacher data by visually specifying each level of label value 0 to 4 according to the positional relationship in the depth direction for each region of the image. In this case, a relative positional relationship in the depth direction based on human senses can be obtained. Although it is not easy to obtain distance information as an absolute numerical value from a normal endoscopic image, it is easy for a person skilled in viewing an endoscopic image to intuitively determine whether it is a close or distant view, and in fact, a doctor performs an insertion operation using the sensory distance information obtained from the image. Therefore, the reliability of the teacher data created in this way is high, and it is possible to generate a learning model capable of estimating accurate depth.

[0059] In the depth estimation method by the depth information generating unit 263, a class is set according to a distance range from the endoscope tip portion 12. In the first embodiment, a label value is assigned to each distance range. Label value 0: Depth < 1st distance Label value 1: First distance ≦ depth < second distance Label value 2: 2nd distance ≦ depth < 3rd distance Label value 3: 3rd distance ≦ depth < 4th distance Label value 4: 4th distance ≦ depth A label value of 0 indicates that the area is closest to the tip 12, and a label value of 4 indicates that the area is farthest from the tip 12.

[0060] Figure 5 shows an example of an endoscopic image. The endoscopic image has a size of 720 x 480, and each RGB pixel is expressed in 8 bits. In this endoscopic image, the lumen is photographed linearly in the depth direction, and multiple folds surround the lumen. Of these, the fold photographed in the lower right is located close to the tip of the endoscope.

[0061] When the image acquisition unit 261 acquires an endoscopic image captured by the endoscope 10 from the signal processing unit 220, it supplies the image to the region division unit 262 and the depth information generation unit 263. The region division unit 262 executes semantic segmentation to divide the endoscopic image into a plurality of regions. At the same time, the depth information generation unit 263 executes depth estimation processing to generate depth information indicating the depth of the endoscopic image.

[0062] 6 shows an example of the region division result by the region division unit 262. The region division unit 262 divides an endoscopic image into a plurality of regions, and derives region information indicating the region division result. Here, the region information may be derived as a label value of each pixel related to a structure. In the first embodiment, the region division unit 262 generates a region division result image using the derived label value. As will be described later, the region division result image may be displayed on the display device 60 and presented to the user.

[0063] The region dividing unit 262 may set the (R, G, B) pixel values ​​corresponding to the label values ​​of the divided regions as follows: In order to distinguish from the label values ​​related to the depth information, the label values ​​0 to 4 of the divided regions will be expressed as label values ​​a0 to a4 hereinafter. Label value a0 (background pixel) → (0,0,0) Label value a1 (normal lumen) → (128,0,0) Label value a2 (fold edge) → (0,0,128) Label value a3 (lumen of bent part) → (192,0,0) Label value a4 (fold edge of bend) → (128,128,128)

[0064] By setting pixel values ​​in this way, the region dividing unit 262 generates a region division result image in which the mucosal surface (label value e0) that occupies the majority is painted black and the extracted structural portion is painted in color. The region dividing unit 262 supplies the region division result image to the recognition unit 264 as region information indicating the result of the region division. In the example shown in FIG. 6, the region of the concentric fold edge and the region of the normal lumen are displayed so that the user can visually recognize them. In yet another example, the region dividing unit 262 may supply the label value of each pixel to the recognition unit 264 as region information indicating the result of the region division.

[0065] 7 shows an example of a depth information estimation result by the depth information generating unit 263. The depth information generating unit 263 executes a depth estimation process on an endoscopic image to generate depth information indicating the depth of the endoscopic image. Here, the depth information may be derived as a label value of each pixel related to the depth (distance from the tip of the endoscope). In the first embodiment, the depth information generating unit 263 generates a depth estimation result image using the derived label value. As will be described later, the depth estimation result image may be displayed on the display device 60 together with the region division result image and presented to the user.

[0066] The depth information generating unit 263 may set the (R, G, B) pixel values ​​corresponding to the label values ​​expressing the depth levels as follows: In order to distinguish from the label values ​​related to the divided regions, the label values ​​0 to 4 of the depth information will be expressed as label values ​​d0 to d4 below. Label value d0 (less than the first distance) → (40,0,0) Label value d1 (greater than or equal to the first distance, less than the second distance) → (80,0,0) Label value d2 (greater than or equal to the second distance, less than the third distance) → (120,0,0) Label value d3 (greater than or equal to the third distance, less than the fourth distance) → (160,0,0) Label value d4 (fourth distance or greater) → (200,0,0)

[0067] By setting pixel values ​​in this manner, the depth information generating unit 263 generates a depth estimation result image in which the deeper the region, the brighter the red color is. The depth information generating unit 263 supplies the depth estimation result image to the recognition unit 264 as depth information of the endoscopic image. In another example, the depth information generating unit 263 may supply the label value of each pixel to the recognition unit 264 as depth information of the endoscopic image.

[0068] The recognition unit 264 recognizes the situation around the tip of the endoscope by receiving region information of the endoscopic image from the region division unit 262 and depth information of the endoscopic image from the depth information generation unit 263. Specifically, the recognition unit 264 uses the region information and depth information to recognize structures such as the lumen direction and folds included in the endoscopic image, together with their positional relationship in the depth direction.

[0069] FIG. 8 shows an example of a superimposed image in which the region segmentation result image and the depth estimation result image are superimposed. When the region segmentation result image and the depth estimation result image are generated with high accuracy, the fold edge extracted in the region segmentation result image and the boundary line shown in the depth estimation result image basically match. In addition, the normal lumen region extracted in the region segmentation result image and the deepest region shown in the depth estimation result image also basically match. From the superimposed image, the recognition unit 264 can recognize how deep in the depth direction each of the regions corresponding to the various structures segmented by the region segmentation unit 262 is located.

[0070] Next, recognition of various structures involving positional relationships in the depth direction will be described. Fig. 9 shows an example of a recognition result in the superimposed image shown in Fig. 8. In the superimposed image shown in Fig. 9, label values ​​d0 to d4 are added to the regions as information related to the depth direction. Also, codes e1 to e5 are assigned to the regions (label value a2) extracted as fold edges by the region dividing unit 262.

[0071] 6, the fold edges e2, e3, and e4 are continuous, and therefore, based only on the information of the image resulting from the region segmentation, one code is assigned to each of the fold edges e2, e3, and e4. However, the recognition unit 264 can recognize that the fold edges e2, e3, and e4 are different from each other by referring to the depth information of the endoscopic image and specifying the label values ​​d0 to d4 related to the depth of each pixel. Note that, when a fold edge exists on the boundary between different label values ​​related to the depth, it is preferable that the recognition unit 264 applies the shallower label value.

[0072] A method for recognizing the state of an arbitrary pixel p(x,y) will be described. Note that 0≦x<720, 0≦y<480. Assume that the label value pa(x,y) for the pixel p(x,y) regarding the region division is “a2”, and the label value pd(x,y) regarding the depth is “d0”. Here, the label value a2 indicates that it is a fold edge, and the label value d0 indicates that it is a region estimated to be closest to the tip of the endoscope in the depth direction. The recognition unit 264 recognizes that such a pixel p is a pixel that constitutes the fold edge e1 or e3. Furthermore, since the recognition unit 264 recognizes that the region d0 includes the fold edges e1 and e3, it recognizes that the region d0 is a region in which a mucosal surface including a fold is photographed. The label values ​​pd(x, y) relating to the depth of the fold edges e2, e4, and e5 are “d1”, “d2”, and “d3”, respectively, and therefore the recognition unit 264 recognizes the regions d1, d2, and d3 as regions where the mucosal surface including folds of different depths is photographed.

[0073] The recognition unit 264 identifies a pixel p(x,y) having a label value d4 indicating the deepest region, and refers to the label value pa(x,y) related to the region division assigned to the pixel p(x,y). At this time, if the region having the label value d4 and the region having the label value a1 substantially match, the recognition unit 264 recognizes that the region having the label value d4 and the label value a1 is the deepest region and indicates a structure indicating the direction of travel. In addition, since the fold edge e3 that is the boundary of the deepest region with the label value d4 belongs to the shallowest region with the label value d0, the recognition unit 264 recognizes that there is a fold in the lower right of the image that may be an obstacle to the forward movement of the tip of the endoscope. In other words, the recognition unit 264 recognizes that the direction shown in the lower right of the image is a direction in which the endoscope should not travel.

[0074] As described above, the recognition unit 264 recognizes various structures included in the endoscopic image together with their positional relationship in the depth direction based on the region information indicating the result of region division by the region division unit 262 and the depth information of the endoscopic image generated by the depth information generation unit 263. In this way, the recognition unit 264 identifies directions in which the endoscope can proceed and directions in which it must not proceed, and identifies the positions of structures that may become obstacles when the endoscope proceeds. The recognition unit 264 supplies information indicating these recognition results to the operation determination unit 265.

[0075] The operation determination unit 265 generates information regarding the traveling direction of the endoscope 10 based on the recognition result in the recognition unit 264. Specifically, the operation determination unit 265 may generate information regarding the traveling direction of the endoscope 10 from directions in which the endoscope tip portion can travel and directions in which it must not travel. In this example, the operation determination unit 265 may generate information regarding the traveling direction of the endoscope 10 so as to move the tip portion 12 upwardly while avoiding folds that exist on the near side of the deepest normal lumen region.

[0076] 10 shows another example of an endoscopic image. In this endoscopic image, a curved portion of the large intestine is captured. When the image acquisition unit 261 acquires the endoscopic image captured by the endoscope 10 from the signal processing unit 220, it supplies the image to the region division unit 262 and the depth information generation unit 263. The region division unit 262 divides the endoscopic image into a plurality of regions to generate a region division result image. At the same time, the depth information generation unit 263 executes a depth estimation process to generate depth information indicating the depth of the endoscopic image, and generates a depth estimation result image based on the depth information.

[0077] FIG. 11 shows an example of the result of region division by the region division unit 262. The region division unit 262 divides an endoscopic image into a plurality of regions and derives region information indicating the result of the region division. The region information is derived as a label value pa(x, y) of each pixel related to the structure, and the region division unit 262 generates a region division result image using the derived label value. This region division result image includes a region of the fold edge of the bent portion extending vertically near the center, and a region of the lumen of the bent portion extracted along the upper part of the fold edge of the bent portion. The region division unit 262 supplies the region division result image to the recognition unit 264 as region information indicating the result of the region division.

[0078] 12 shows an example of a depth information estimation result by the depth information generating unit 263. The depth information generating unit 263 executes a depth estimation process on the endoscopic image to generate depth information indicating the depth of the endoscopic image. The depth information is derived as a label value pd(x, y) of each pixel related to the depth, and the depth information generating unit 263 generates a depth estimation result image using the derived label value. The depth information generating unit 263 supplies the depth estimation result image to the recognition unit 264 as the depth information of the endoscopic image.

[0079] The recognition unit 264 receives area information of the endoscope image from the area division unit 262, and receives depth information of the endoscope image from the depth information generation unit 263, and recognizes the situation around the tip of the endoscope. Specifically, the recognition unit 264 uses the area information and depth information to recognize structures such as the lumen direction and folds included in the endoscope image, along with their positional relationship in the depth direction. In this way, the recognition unit 264 identifies the location of structures that may be an obstacle when the endoscope advances, and identifies the directions in which the endoscope can advance and the directions in which it must not advance. In this example, the recognition unit 264 recognizes that the left side of the image is the direction in which the endoscope can advance and the right side of the image is the direction in which it must not advance, and recognizes that the lumen continues to the right. The recognition unit 264 supplies the recognition result to the operation determination unit 265, and the operation determination unit 265 generates information regarding the traveling direction of the endoscope based on the directions in which the endoscope can advance and the directions in which it must not advance. Specifically, the operation determination unit 265 determines the traveling direction of the endoscope tip so that the endoscope tip is directed to the left and advanced in the direction in which it is directed.

[0080] 13 shows another example of an endoscopic image. In this endoscopic image, a large fold is captured on the right side. When the image acquisition unit 261 acquires the endoscopic image from the signal processing unit 220, it supplies the image to the region division unit 262 and the depth information generation unit 263.

[0081] 14 shows an example of the region division result by the region division unit 262. The region division unit 262 divides the endoscopic image into a plurality of regions, derives region information indicating the region division result, and generates a region division result image. This region division result image includes a region of a fold edge and a region of a normal lumen. The region division unit 262 supplies the region division result image to the recognition unit 264 as region information indicating the region division result.

[0082] 15 shows an example of a depth information estimation result by the depth information generating unit 263. The depth information generating unit 263 executes a depth estimation process on the endoscopic image to generate depth information indicating the depth of the endoscopic image, and generates a depth estimation result image. The depth information generating unit 263 supplies the depth estimation result image to the recognition unit 264 as the depth information of the endoscopic image.

[0083] The recognition unit 264 receives area information of the endoscope image from the area division unit 262, and depth information of the endoscope image from the depth information generation unit 263, and recognizes the situation around the tip of the endoscope. In this example, the recognition unit 264 recognizes that there is a large fold on the right side of the image, which will be an obstacle to the forward movement of the endoscope, and recognizes that the left side of the image is a direction in which the endoscope can proceed and the right side of the image is a direction in which the endoscope must not proceed. The recognition unit 264 supplies the recognition result to the operation determination unit 265, and the operation determination unit 265 generates information regarding the traveling direction of the endoscope based on the directions in which the endoscope can proceed and the directions in which the endoscope must not proceed. Specifically, the operation determination unit 265 determines the traveling direction of the tip of the endoscope so that the tip of the endoscope is directed left and advanced in the direction in which it is directed.

[0084] 16 shows another example of an endoscopic image. In this endoscopic image, a large fold is captured from the top to the left. When the image acquisition unit 261 acquires the endoscopic image from the signal processing unit 220, it supplies the image to the region division unit 262 and the depth information generation unit 263.

[0085] 17 shows an example of the region division result by the region division unit 262. The region division unit 262 divides the endoscopic image into a plurality of regions, derives region information indicating the region division result, and generates a region division result image. This region division result image includes a region of a fold edge and a region of a normal lumen. The region division unit 262 supplies the region division result image to the recognition unit 264 as region information indicating the region division result.

[0086] 18 shows an example of a depth information estimation result by the depth information generating unit 263. The depth information generating unit 263 executes a depth estimation process on the endoscopic image to generate depth information indicating the depth of the endoscopic image, and generates a depth estimation result image. The depth information generating unit 263 supplies the depth estimation result image to the recognition unit 264 as the depth information of the endoscopic image.

[0087] The recognition unit 264 receives area information of the endoscope image from the area division unit 262, receives depth information of the endoscope image from the depth information generation unit 263, and recognizes the situation around the endoscope tip. When comparing the area division result image shown in FIG. 17 with the depth estimation result image shown in FIG. 18, the area extracted as the normal lumen (label value a1) in the area division result image spans the areas of label values ​​d3 and d4 in the depth estimation result image. Therefore, the recognition unit 264 specifies the overlapping area of ​​the area of ​​label value a1 in the area division result image and the area of ​​label value d4 in the depth estimation result image as an area where the endoscope can proceed. The recognition unit 264 supplies the recognition result to the operation determination unit 265, and the operation determination unit 265 generates information regarding the traveling direction of the endoscope. Specifically, the operation determination unit 265 determines the traveling direction of the endoscope tip so that the endoscope tip is directed to the right and advanced in the direction of direction.

[0088] In the first embodiment, each time the image acquisition unit 261 acquires an endoscopic image, the operation determination unit 265 determines the operation content of the endoscope 10, and the operation control unit 266 generates an operation control signal according to the determined operation content and supplies it to the drive unit 240.

[0089] As another timing example, the drive unit 240 may drive the operating mechanism of the endoscope 10 based on an operation control signal from the operation control unit 266, and at the timing when the drive is completed, the operation determination unit 265 may determine the operation content of the endoscope 10, and the operation control unit 266 may generate an operation control signal.

[0090] For example, when the driving unit 240 is driving the operation mechanism of the endoscope 10, the driving determination unit 267 determines that the driving unit 240 is driving the operation mechanism. When the driving by the driving unit 240 ends, the driving determination unit 267 determines that the driving of the operation mechanism has ended. At this time, the driving determination unit 267 notifies the recognition unit 264 that new operation content should be determined. The recognition unit 264 generates recognition results of various structures included in the endoscopic image, and the operation determination unit 265 determines the operation content of the endoscope 10 based on the recognition results. The operation determination unit 265 may determine the operation content of the endoscope 10 after or immediately before the driving of the operation mechanism is completed.

[0091] In the first embodiment, the result of the area division process by the area division unit 262 and the result of the depth estimation process by the depth information generation unit 263 may be corrected by various known techniques using frequency component information, brightness information of pixel values, etc., and then provided to the recognition unit 264.

[0092] The operation control unit 266 has a function of controlling the operation of the endoscope 10 based on the operation content determined by the operation determination unit 265. The operation control unit 266 may set an amount of operation in the determined operation content based on at least one of the insertion shape information output from the insertion shape detection device 30 and the external force information output from the external force information acquisition device 40. The operation control unit 266 generates an operation control signal according to the operation content determined by the operation determination unit 265 and the amount of operation in the operation content, and outputs the signal to the drive unit 240.

[0093] A description will be given of the operation of Example 1. In the following, a description will be given of control relating to the insertion operation of the insertion portion 11 inserted into the intestinal tract of the large intestine from the anus. After connecting each part of the endoscope system 1 and turning on the power, the user inserts the tip 12 of the endoscope 10 into the anus of the subject. At this time, the user operates the input device 50 to set the automatic insertion mode of the endoscope 10 to ON, which causes the processing device 20 to execute the automatic operation function of the endoscope 10.

[0094] The light source unit 210 supplies illumination light to the endoscope 10, and the imaging unit 110 captures an image of a subject illuminated with the illumination light at a predetermined period and transmits an imaging signal to the processing device 20. The signal processing unit 220 generates an endoscopic image from the imaging signal and supplies the image to the display processing unit 250 and the image acquisition unit 261.

[0095] Coil drive signal generation unit 230 supplies coil drive signals to the multiple source coils 18, receiving antenna 310 detects magnetic fields generated by each of the multiple source coils 18, and insertion shape information acquisition unit 320 generates insertion shape information of the insertion unit 11. The insertion shape information is supplied to control unit 260 and external force information acquisition device 40. External force information acquisition device 40 generates external force information at each position of the multiple source coils 18 from the insertion shape information and supplies the information to control unit 260.

[0096] The region dividing unit 262 divides the endoscopic image acquired by the image acquiring unit 261 into a plurality of regions to generate region information of the endoscopic image. The depth information generating unit 263 generates information indicating the depth of the endoscopic image acquired by the image acquiring unit 261. The recognition unit 264 receives the region information of the endoscopic image from the region dividing unit 262, receives the depth information of the endoscopic image from the depth information generating unit 263, and recognizes the situation around the tip of the endoscope. The recognition unit 264 uses the region information of the endoscopic image and the depth information of the endoscopic image to recognize structures such as the lumen direction and folds included in the endoscopic image, together with their positional relationship in the depth direction.

[0097] The operation determination unit 265 generates information regarding the traveling direction of the endoscope based on the situation around the endoscope tip part recognized by the recognition unit 264. Specifically, the operation determination unit 265 generates information regarding the traveling direction of the endoscope based on the directions in which the endoscope tip part can travel and the directions in which it must not travel, recognized by the recognition unit 264, and determines the operation content of the endoscope tip part.

[0098] The operation control unit 266 generates an operation control signal for controlling the operation of the endoscope 10, based on the operation content determined by the operation determination unit 265. At this time, the operation control unit 266 may perform processing for setting an amount of operation in the determined operation content, based on at least one of the insertion shape information output from the insertion shape detection device 30 and the external force information output from the external force information acquisition device 40. The operation control unit 266 generates an operation control signal according to the determined operation content and the set amount of operation, and outputs the signal to the drive unit 240.

[0099] The operation control unit 266 generates an operation control signal for controlling the operation of the endoscope 10 based on the operation content determined by the operation determination unit 265. At this time, the operation control unit 266 may perform processing for setting an amount of operation in the determined operation content based on at least one of the insertion shape information output from the insertion shape detection device 30 and the external force information output from the external force information acquisition device 40. The operation control unit 266 generates an operation control signal for performing operation control according to the determined operation content and the set amount of operation, and outputs the signal to the drive unit 240.

[0100] When the operation content determined by the operation determination unit 265 is an angle operation, the operation control unit 266 sets the bending angle CDS of the bending portion 13 as the amount of operation in the operation content. Then, the operation control unit 266 generates an operation control signal for executing control to bend the bending portion 13 by the bending angle CDS, and outputs the signal to the drive unit 240.

[0101] When the operation content determined by the operation determination unit 265 is an advance operation, the operation control unit 266 sets the movement amount MES of the insertion unit 11 as the operation amount in the operation content. Then, the operation control unit 266 generates an operation control signal for executing control to advance the insertion unit 11 by the movement amount MES, and outputs the signal to the drive unit 240. Note that the movement amount MES is preferably set as a value within a range in which the insertion unit 11 inserted into the intestinal tract can be safely advanced.

[0102] When the operation content determined by the operation determination unit 265 is a retreat operation, the operation control unit 266 sets the movement amount MFS of the insertion unit 11 as the operation amount in the operation content. Then, the operation control unit 266 generates an operation control signal for executing control to retreat the insertion unit 11 by the movement amount MFS, and outputs it to the drive unit 240. Note that the movement amount MFS is preferably set as a value within a range in which the insertion unit 11 inserted into the intestinal tract can be safely retreated.

[0103] When the operation content determined by the operation determination unit 265 is a search operation, the operation control unit 266 sets the movement amount MGS of the insertion unit 11 as the movement amount in the operation content. Then, the operation control unit 266 generates an operation control signal for executing control to point the tip portion 12 in multiple directions after moving the insertion unit 11 back by the movement amount MGS, and outputs the signal to the drive unit 240. At this time, the operation control unit 266 may generate an operation control signal for executing control to point the tip portion 12 in four or eight directions. In the process related to the search operation SES, a process is performed in which the tip portion 12 is pointed in multiple directions and a normal lumen is found from endoscopic images captured in each direction.

[0104] The operation control unit 266 sets the amount of operation based on at least one of the insertion shape information output from the insertion shape detection device 30 and the external force information output from the external force information acquisition device 40, but may also set the amount of operation by reading a setting value pre-stored in the storage medium 24.

[0105] <Example 2> 19 shows functional blocks of the control unit 260 in Example 2. The control unit 260 includes an image acquisition unit 261, an operation content selection unit 270, an area division unit 262, a depth information generation unit 263, a recognition unit 264, an operation content determination unit 271, a motion control unit 266, a drive determination unit 267, and an operation selection model 272.

[0106] The control unit 260 shown in Fig. 19 can be realized in hardware by one or more processors 22, memory, auxiliary storage device, and other LSIs, and in software by a program loaded into memory, but here, functional blocks realized by cooperation of these are depicted. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination of both. For example, a program for executing at least some of the functions of the control unit 260 may be stored in the storage medium 24, and the processor 22 may load the program from the storage medium 24 into the memory to realize each function of the control unit 260.

[0107] The image acquisition unit 261, the area division unit 262, the depth information generation unit 263, the recognition unit 264, the operation control unit 266 and the drive determination unit 267 in the second embodiment have the same or similar functions as the image acquisition unit 261, the area division unit 262, the depth information generation unit 263, the recognition unit 264, the operation control unit 266 and the drive determination unit 267 described in the first embodiment, and therefore, duplicated explanations will be omitted below as appropriate.

[0108] The image acquiring unit 261 acquires an endoscopic image captured by the endoscope 10 inserted into the subject from the signal processing unit 220. In the second embodiment, the image acquiring unit 261 supplies the acquired endoscopic image to the operation content selecting unit 270, the area dividing unit 262, and the depth information generating unit 263.

[0109] The operation content selection unit 270 has a function of selecting one or more operation contents from among a plurality of predetermined operation contents based on the endoscopic image acquired by the image acquisition unit 261. In other words, the operation content selection unit 270 selects an operation content to be performed from among a plurality of options for endoscopic operation based on an endoscopic image capturing the inside of a subject. The plurality of predetermined operation contents may be composed of at least one type of operation, including a forward operation, a backward operation, an angle operation, a twisting operation, an air supply operation, a water supply operation, and a suction operation.

[0110] The operation content selection unit 270 selects operation content recommended for the endoscope 10 capturing the endoscopic image by inputting input data acquired from the endoscopic image acquired by the image acquisition unit 261 to the operation selection model 272. The operation selection model 272 is a trained model generated by machine learning using, as teacher data, learning images that are endoscopic images captured in the past and labels indicating operation content for the endoscope that captured the learning images.

[0111] In the second embodiment, the operation selection model 272 is generated by learning each coupling coefficient (weight) in a CNN (Convolutional Neural Network), which corresponds to a multi-layer neural network including an input layer, one or more convolution layers, and an output layer, using a learning method such as deep learning.

[0112] The region division unit 262 has a function of dividing the endoscopic image acquired by the image acquisition unit 261 into a plurality of regions. Specifically, the region division unit 262 divides the endoscopic image into regions of each structure by executing semantic segmentation that labels each pixel in the endoscopic image. The region division unit 262 defines a region having a structure of a type (class) to be divided, and generates a region division result in which pixels of each structure are labeled. The region division unit 262 supplies region information indicating the region division result to the recognition unit 264.

[0113] The depth information generating unit 263 has a function of generating information indicating the depth of the endoscopic image acquired by the image acquiring unit 261. Various methods for estimating the depth of pixels or blocks included in an image have been proposed in the past. The depth information generating unit 263 may generate information indicating the depth of each pixel of the endoscopic image by using, for example, the technology disclosed in Non-Patent Document 2. The depth information generating unit 263 supplies the depth information of the endoscopic image to the recognition unit 264.

[0114] The recognition unit 264 recognizes the situation around the endoscope tip based on area information indicating the result of area division by the area division unit 262 and / or depth information of the endoscope image generated by the depth information generation unit 263. Specifically, the recognition unit 264 identifies at least directions in which the endoscope 10 can advance and directions in which it must not advance, as the situation around the endoscope tip.

[0115] The operation content determination unit 271 determines the operation content to be performed based on the operation content selected by the operation content selection unit 270 and the situation around the tip of the endoscope recognized by the recognition unit 264.

[0116] In the second embodiment, the operation content determination unit 271 may determine the operation content of the endoscope 10 each time the image acquisition unit 261 acquires an endoscopic image, and the operation control unit 266 may generate an operation control signal according to the determined operation content and supply it to the drive unit 240. As another timing example, the drive unit 240 may drive the operating mechanism of the endoscope 10 based on the operation control signal from the operation control unit 266, and at the timing when the drive is completed, the operation content determination unit 271 may determine the operation content of the endoscope 10, and the operation control unit 266 may generate an operation control signal.

[0117] (Selection of operation contents) To generate the operation selection model 272, machine learning is performed using training data including training images, which are endoscopic images previously taken with an endoscope of the inside of the intestinal tract or colon model, and labels indicating which of the 12 operation contents is most suitable for the situation shown by the training image.

[0118] Here, the 12 operation contents include the following: Angle operation UPS for bending the bending portion 13 to point the tip portion 12 upward. Angle operation RIS for bending the bending portion 13 and directing the tip portion 12 to the right Angle operation DOS for bending the bending portion 13 to point the tip portion 12 downward Angle operation LES for bending the bending portion 13 and directing the tip portion 12 to the left Angle operation URS for bending the bending portion 13 and directing the tip 12 in the upper right direction Angle operation DRS for bending the bending portion 13 to point the tip portion 12 in a lower right direction Angle operation DLS for bending the bending portion 13 to point the tip portion 12 in a lower left direction Angle operation ULS for bending the bending portion 13 to point the tip portion 12 in the upper left direction -Advance operation PSS for advancing the tip 12 · Retract operation PLS to retract the tip 12 A search operation SES for searching the lumen in multiple directions of the tip portion 12 Angle maintenance operation AMS for fixing the bending angle of the bending portion 13 and maintaining the direction of the tip portion 12 in the current direction

[0119] When creating the training data, the expert looks at the learning image, subjectively selects one operation content that is most likely to be performed in the situation shown in the learning image from the above-mentioned 12 operation contents, and assigns a label of the selected operation content to the learning image. The expert may be a doctor. For example, when the endoscopic image 70b shown in FIG. 3(b) is the learning image, since the center of the lumen is present at the top of the image, the expert determines that an operation to point the endoscope tip upward, that is, an angle operation UPS should be performed, and assigns a label of the angle operation UPS to the endoscopic image 70b. This labeling operation is performed on many past endoscopic images to create training data.

[0120] Below, an example of training data including learning images and labels is shown. Fig. 20 shows an example of training data. A "label of angle operation UPS" indicating an upward angle operation is added to each of the learning images shown in Fig. 20. The learning image shown in Fig. 20 is an image for which it has been determined that the bending portion 13 should be bent upward as the endoscope operation to be performed.

[0121] Fig. 21 shows another example of training data. All of the learning images shown in Fig. 21 are given a "label of angle operation RIS" indicating a rightward angle operation. The learning images shown in Fig. 21 are images for which it has been determined that the bending portion 13 should be bent to the right as the next endoscope operation to be performed.

[0122] Fig. 22 shows another example of training data. All of the learning images shown in Fig. 22 are given a "label of angle operation DOS" indicating a downward angle operation. The learning images shown in Fig. 22 are images for which it has been determined that the bending portion 13 should be bent downward as the endoscope operation to be performed.

[0123] Fig. 23 shows another example of training data. All of the learning images shown in Fig. 23 are given a "label of angle operation LES" indicating a leftward angle operation. The learning images shown in Fig. 23 are images for which it has been determined that the bending portion 13 should be bent to the left as the next endoscope operation to be performed.

[0124] Fig. 24 shows another example of training data. A "label of angle operation URS" indicating an angle operation in the upper right direction is added to each of the learning images shown in Fig. 24. The learning image shown in Fig. 24 is an image for which it has been determined that the bending portion 13 should be bent in the upper right direction as the endoscope operation to be performed.

[0125] Fig. 25 shows another example of training data. All of the learning images shown in Fig. 25 are given an "angle operation DRS label" indicating an angle operation in the lower right direction. The learning images shown in Fig. 25 are images for which it has been determined that the bending portion 13 should be bent in the lower right direction as the endoscope operation to be performed.

[0126] Fig. 26 shows another example of training data. All of the learning images shown in Fig. 26 are given a "label of angle operation DLS" indicating an angle operation in the lower left direction. The learning images shown in Fig. 26 are images for which it has been determined that the bending portion 13 should be bent in the lower left direction as the endoscope operation to be performed.

[0127] Fig. 27 shows another example of training data. The learning images shown in Fig. 27 are all given a "label of angle operation ULS" indicating an angle operation in the upper left direction. The learning images shown in Fig. 27 are images determined to indicate that the bending portion 13 should be bent in the upper left direction as the endoscope operation to be performed.

[0128] Fig. 28 shows another example of training data. A "push operation (forward movement) PSS label" indicating a forward movement is added to each of the learning images shown in Fig. 28. The learning image shown in Fig. 28 is an image that has been determined to indicate that the tip portion 12 should be moved forward as the next endoscope operation.

[0129] Fig. 29 shows another example of training data. All of the learning images shown in Fig. 29 are given a "pull operation (retreat operation) PLS label" indicating a retraction operation. The learning images shown in Fig. 29 are images determined to be endoscope operations that require retraction of the tip portion 12. Typical examples of situations in which a retraction operation is required include a situation in which the tip portion 12 is excessively close to the mucosal surface of the large intestine, or a situation in which the tip portion 12 comes into contact with the mucosal surface, which is commonly referred to as a "red ball" among endoscopists.

[0130] Fig. 30 shows another example of training data. A "search operation SES label" indicating a search operation is added to each of the learning images shown in Fig. 30. The learning images shown in Fig. 30 are images determined to be images of multiple directions by bending the bending portion 13 in multiple directions as an endoscope operation to be performed.

[0131] Although the teaching data for the angle maintaining operation AMS for fixing the bending angle of the bending portion 13 and maintaining the orientation of the tip portion 12 in the current orientation is not shown, for example, the learning image shown in FIG. 28 may be labeled with "angle maintaining operation AMS." The operation selection model 272 in the second embodiment is generated by machine learning using the training data shown in FIGS.

[0132] The operation content selection unit 270 selects one or more operation contents by inputting input data acquired from the endoscopic image acquired by the image acquisition unit 261 to one or more operation selection models 272 generated by machine learning using the learning image and the label indicating the operation content for the endoscope that captured the learning image as teacher data. Specifically, the operation content selection unit 270 acquires multidimensional data such as the pixel value of each pixel included in the endoscopic image acquired by the image acquisition unit 261, and inputs the multidimensional data as input data to the input layer of the neural network of the operation selection model 272. The operation selection model 272 outputs 12 likelihoods corresponding to 12 operation contents that can be selected as the operation content of the endoscope 10 from the output layer of the neural network. The operation content selection unit 270 can obtain the operation content corresponding to the highest likelihood among the 12 likelihoods included in the output data as the selection result of the operation content of the endoscope 10.

[0133] As described above, the operation content selection unit 270 is configured to input the input data acquired from the endoscopic image acquired by the image acquisition unit 261 to the operation selection model 272 and process it to obtain a selection result indicating one operation content selected from 12 operation contents including an operation for directing the orientation of the tip portion 12 in eight directions perpendicular to the insertion axis of the insertion unit 11, an operation for moving the tip portion 12 forward or backward, an operation for maintaining the orientation of the tip portion 12 in the current orientation, and an operation for searching for a lumen near the tip portion 12. The operation content selection unit 270 supplies the selected operation content to the operation content determination unit 271.

[0134] The operation selection model 272 is preferably constructed by sufficient learning so as to output appropriate operation contents, but the accuracy depends on the teacher data. Therefore, in the second embodiment, a method is realized in which whether or not the operation contents selected by the operation content selection unit 270 are appropriate is determined based on the situation around the tip of the endoscope recognized by the recognition unit 264.

[0135] (Area segmentation processing of endoscopic images) In parallel with the processing in the operation content selection unit 270, the region division unit 262 divides the endoscopic image acquired by the image acquisition unit 261 into a plurality of regions. As described in the first embodiment, the region division unit 262 executes semantic segmentation that labels each pixel in the endoscopic image, and divides the endoscopic image into regions of each structure. The region division unit 262 may execute the semantic segmentation using FCN.

[0136] Fig. 31(a) shows an example of an endoscopic image. When the image acquisition unit 261 acquires an endoscopic image captured by the endoscope 10 from the signal processing unit 220, it supplies the image to the operation content selection unit 270 and the area division unit 262. The operation content selection unit 270 inputs the input data acquired from the endoscopic image of Fig. 31(a) into the operation selection model 272, and as a result selects an "advance operation PSS" for advancing the tip portion 12 as the operation content. The operation content selection unit 270 supplies the selected operation content to the operation content determination unit 271.

[0137] The region dividing unit 262 performs semantic segmentation to divide the endoscopic image into a plurality of regions. FIG. 31(b) shows an example of the result of the region division by the region division unit 262. The region division unit 262 divides the endoscopic image into a plurality of regions and derives region information indicating the result of the region division. Here, the region information may be derived as a label value pa(x, y) of each pixel related to the structure. The region division unit 262 generates a region division result image using the derived label value. In the second embodiment, the endoscopic image has a size of 720×480, and the region division result image also has a size of 720×480. The region division unit 262 supplies the region division result image to the recognition unit 264 as region information indicating the result of the region division. In this region division result image, a region of a concentric fold edge and a region of a normal lumen are included. In another example, the region division unit 262 may supply the label value of each pixel to the recognition unit 264 as region information indicating the result of the region division.

[0138] The recognition unit 264 divides the region division result image of 720×480 size into a plurality of partial regions and recognizes the proportion of the normal lumen region contained in each partial region. In the second embodiment, the endoscopic image is divided into 5 equal parts in each of the horizontal and vertical directions, and divided into 5×5 partial regions, and each partial region is expressed as sub(i,j). Note that i,j=1,2,3,4,5. In another example, the recognition unit 264 may divide the region division result image into a division number other than 5×5.

[0139] FIG. 32 shows a state in which the region segmentation result image is segmented into a plurality of partial regions. In this example, the region extracted as the normal lumen is present across the partial regions sub(3,2), sub(2,3), sub(3,3), sub(2,4), and sub(3,4). The recognition unit 264 derives the proportion of pixels of the normal lumen in each partial region sub(i,j). This proportion is calculated by the following formula. Here, the total number of pixels in the partial regions is represented as an, and in this example, an=144×96. Normal lumen ratio (a) = (number of pixels in the normal lumen region in the partial region) / an

[0140] The recognition unit 264 may determine that a normal lumen region included in a partial region in which this ratio (a) exceeds a predetermined threshold (e.g., 0.4) is an area in which the endoscope can proceed. In this example, the normal lumen ratio (a) calculated for sub(3,3) exceeds the threshold, and therefore the recognition unit 264 recognizes that an area in which the endoscope can proceed exists at sub(3,3) in the endoscopic image. The recognition unit 264 recognizes the direction in which the endoscope can proceed by identifying the area in which the endoscope can proceed. The recognition unit 264 supplies this recognition result to the operation content determination unit 271.

[0141] In the above example, the recognition unit 264 determines the area into which the endoscope can proceed from the proportion of pixels of the normal lumen in each partial region sub(i,j). In another example, the recognition unit 264 may recognize a partial region sub(i,j) that contains a predetermined proportion or more of the total number of pixels of the normal lumen as a partial region into which the endoscope can proceed. In this case, the recognition unit 264 counts the number of pixels pl divided into the region as the normal lumen. The recognition unit 264 derives the proportion of the normal lumen region contained in each partial region sub(i,j) to the entire normal lumen region. This proportion is derived by the following formula. Normal lumen ratio (b) = (number of pixels in the normal lumen region in the partial region) / pl

[0142] The recognition unit 264 may determine that a partial region in which this ratio (b) exceeds a predetermined threshold (for example, 0.6) is a region in which the endoscope can proceed. In this example, the normal lumen ratio (b) calculated for sub(3,3) exceeds the threshold, and therefore the recognition unit 264 recognizes that a region in which the endoscope can proceed exists in sub(3,3) in the endoscopic image. The recognition unit 264 may specify a partial region in which the ratio (b) is maximum as a region in which the endoscope can proceed. The recognition unit 264 recognizes the direction in which the endoscope can proceed by specifying the region in which the endoscope can proceed. The recognition unit 264 supplies this recognition result to the operation content determination unit 271. The recognition unit 264 may specify the region and direction in which the endoscope can proceed based on either the normal lumen ratio (a) or the normal lumen ratio (b).

[0143] The operation content determination unit 271 receives the operation content selected from the operation content selection unit 270, and receives the recognition result of the situation around the tip from the recognition unit 264. The operation content determination unit 271 judges whether the operation content selected by the operation content selection unit 270 is appropriate or not based on the situation recognized by the recognition unit 264. Here, the operation content determination unit 271 selects the "forward movement operation PSS" for moving the tip 12 forward as the operation content, and the recognition unit 264 recognizes that an area in which the endoscope tip can advance exists at sub(3,3) located at the center of the endoscope image. Since sub(3,3) exists in the forward movement direction of the endoscope tip, the operation content determination unit 271 judges that the forward movement operation PSS selected by the operation content selection unit 270 is appropriate, and determines the forward movement operation PSS as the operation content to be performed.

[0144] Fig. 33(a) shows another example of an endoscopic image. When the image acquisition unit 261 acquires an endoscopic image captured by the endoscope 10 from the signal processing unit 220, it supplies the image to the operation content selection unit 270 and the area division unit 262. The operation content selection unit 270 inputs the input data acquired from the endoscopic image of Fig. 33(a) into the operation selection model 272, and as a result selects an "advance operation PSS" for advancing the tip portion 12 as the operation content. The operation content selection unit 270 supplies the selected operation content to the operation content determination unit 271.

[0145] FIG. 33(b) shows an example of the result of region division by the region division unit 262. The region division unit 262 divides the endoscopic image into a plurality of regions and derives region information indicating the result of the region division. The region information is derived as a label value pa(x, y) of each pixel related to the structure, and the region division unit 262 generates a region division result image using the derived label value. The region division unit 262 supplies the region division result image to the recognition unit 264 as region information indicating the result of the region division. The recognition unit 264 divides the region division result image into 5×5 partial regions and recognizes the proportion of normal lumen region included in each partial region.

[0146] FIG. 34 shows a state in which the region division result image is divided into a plurality of partial regions. In this example, the region extracted as the normal lumen exists across the partial regions sub(2,2), sub(3,2), sub(2,3), and sub(3,3). The recognition unit 264 derives the ratio of the number of pixels of the normal lumen in each partial region sub(i,j). In this example, the recognition unit 264 calculates the normal lumen ratio (b) and recognizes that in sub(3,2), the normal lumen ratio (b) exceeds a predetermined threshold (for example, 0.6). Therefore, the recognition unit 264 recognizes that in sub(3,2) in the endoscopic image, there is a region in which the endoscope can proceed. The recognition unit 264 supplies this recognition result to the operation content determination unit 271.

[0147] The operation content determination unit 271 receives the operation content selected from the operation content selection unit 270, and receives the recognition result of the situation around the tip from the recognition unit 264. Here, the operation content selection unit 270 selects the "forward movement operation PSS" for moving the tip 12 forward as the operation content, and the recognition unit 264 recognizes that an area in which the endoscope can proceed exists at sub(3,2) which is located higher than the central part of the endoscope image. Since sub(3,2) exists above the forward movement direction of the tip of the endoscope, the operation content determination unit 271 determines that the forward movement operation PSS selected by the operation content selection unit 270 is not appropriate. When it is determined that the selected forward movement operation PSS is not appropriate, the operation content determination unit 271 may forcibly end (interrupt) the automatic control of the operation of the endoscope 10.

[0148] At this time, the operation content determining unit 271 may determine, as the operation content to be performed, an "angle operation UPS" for bending the bending portion 13 and pointing the tip portion 12 upward. Alternatively, the operation content determining unit 271 may determine, as the operation content to be performed, performing an "advance operation PSS" after the "angle operation UPS". In this manner, the operation content determining unit 271 can correct the operation content that is inappropriate for the situation of the tip portion of the endoscope by determining whether or not the operation content selected by the operation content selecting unit 270 is appropriate based on the recognition result by the recognizing unit 264.

[0149] Fig. 35(a) shows another example of an endoscopic image. When the image acquisition unit 261 acquires an endoscopic image captured by the endoscope 10 from the signal processing unit 220, it supplies the image to the operation content selection unit 270 and the area division unit 262. The operation content selection unit 270 inputs the input data acquired from the endoscopic image of Fig. 35(a) into the operation selection model 272, and as a result selects an "advance operation PSS" for advancing the tip portion 12 as the operation content. The operation content selection unit 270 supplies the selected operation content to the operation content determination unit 271.

[0150] 35(b) shows an example of the region division result by the region division unit 262. The region division unit 262 generates a region division result image and supplies it to the recognition unit 264. The recognition unit 264 divides the region division result image into 5 × 5 partial regions and recognizes the proportion of normal lumen region contained in each partial region.

[0151] 36 shows a state in which the region segmentation result image is segmented into a plurality of partial regions. In this example, there is no region extracted as a normal lumen, and the region of the fold edge of the bent portion and the region of the lumen of the bent portion along the fold edge of the bent portion are extracted. The recognition unit 264 recognizes that there is no normal lumen region in each partial region sub(i,j), and that there is a lumen region of the bent portion.

[0152] First, the recognition unit 264 extracts a partial region including the lumen of the bent portion and the extracted region. In this example, the recognition unit 264 recognizes that the lumen region of the bent portion exists across the partial regions sub(2,2), sub(3,2), sub(4,2), sub(2,3), sub(3,3), sub(4,3), sub(5,3), and sub(5,4). From this, the recognition unit 264 recognizes that the lumen region of the bent portion exists in the center portion in the vertical direction.

[0153] Next, the recognition unit 264 extracts pixels of the fold edge of the bent portion, and identifies the direction of the circular arc of the lumen of the bent portion from the two end points and their midpoint. 37 shows two end points and their midpoints of the pleat edge of the bent portion. The recognition unit 264 identifies the direction of a circular arc formed by the pleat edge of the bent portion by identifying the direction of a perpendicular line drawn from the midpoint c to the line segment connecting the two end points k1 and k2. In this example, since a perpendicular line can be drawn from the midpoint c to the line segment connecting the end points k1 and k2 in the lower left direction, the recognition unit 264 recognizes that the pleat edge of the bent portion has an opening facing in the lower left direction with respect to the midpoint c, that is, the extension direction of the lumen is the upper right direction of the endoscopic image.

[0154] As described above, the recognition unit 264 recognizes from the region division result image shown in Figure 37 that a bent portion of the lumen is present in approximately the center of the endoscopic image, and that the lumen extension direction is toward the upper right of the image, and supplies the recognition result to the operation content determination unit 271.

[0155] When the operation content determination unit 271 receives the recognition result from the recognition unit 264, it determines that the forward movement operation PSS selected by the operation content selection unit 270 is not appropriate. The operation content determination unit 271 confirms from the recognition result that an operation of directing the tip portion 12 to the lower left with respect to the bent portion is preferable. Therefore, the operation content determination unit 271 may determine the "angle operation DLS" for bending the bending portion 13 and directing the tip portion 12 to the lower left direction as the operation content to be performed. Alternatively, the operation content determination unit 271 may determine the "forward movement operation PSS" to be performed after the "angle operation DLS" as the operation content to be performed. In this way, the operation content determination unit 271 can correct the operation content that is inappropriate for the situation of the tip portion of the endoscope by determining whether the operation content selected by the operation content selection unit 270 is appropriate based on the recognition result by the recognition unit 264.

[0156] The above describes a method in which the operation content determination unit 271 determines whether or not the operation content selected by the operation content selection unit 270 is appropriate based on the result of recognition of the area analysis result by the area division unit 262 by the recognition unit 264. Below, a method in which the operation content determination unit 271 determines whether or not the operation content selected by the operation content selection unit 270 is appropriate by further taking into account the depth information of the endoscopic image generated by the depth information generation unit 263 will be described.

[0157] FIG. 38 shows another example of an endoscopic image. At the bottom of this endoscopic image, a lumen is photographed linearly in the depth direction, and multiple folds exist surrounding the lumen. When the image acquisition unit 261 acquires an endoscopic image photographed by the endoscope 10 from the signal processing unit 220, it supplies the image to the operation content selection unit 270, the region division unit 262, and the depth information generation unit 263. The operation content selection unit 270 inputs the input data acquired from the endoscopic image of FIG. 38 into the operation selection model 272, and selects "angle operation DOS" for bending the bending portion 13 and pointing the tip portion 12 downward as the operation content. The operation content selection unit 270 supplies the selected operation content to the operation content determination unit 271.

[0158] Fig. 39 shows an example of the region division result by the region division unit 262. The region division unit 262 divides the endoscopic image into a plurality of regions, derives region information indicating the region division result, and generates a region division result image. The region division result image shown in Fig. 39 includes a region of a fold edge and a region of a normal lumen. The region division unit 262 supplies the region division result image to the recognition unit 264 as region information indicating the region division result.

[0159] 40 shows an example of a depth information estimation result by the depth information generating unit 263. The depth information generating unit 263 executes a depth estimation process on the endoscopic image to generate depth information indicating the depth of the endoscopic image, and generates a depth estimation result image. The depth information generating unit 263 supplies the depth estimation result image to the recognition unit 264 as the depth information of the endoscopic image.

[0160] The recognition unit 264 receives area information of the endoscope image from the area division unit 262, receives depth information of the endoscope image from the depth information generation unit 263, and recognizes the situation around the tip of the endoscope. 41 shows an example of recognition of a depth estimation result image. From the region segmentation result image, the recognition unit 264 recognizes that a normal lumen region exists in the lower center of the image. Furthermore, from the depth estimation result image, the recognition unit 264 recognizes that a region N closest to the endoscope tip exists in the lower right of the image, and that a region F farthest from the endoscope tip exists adjacent to region N. Furthermore, the recognition unit 264 refers to the region segmentation result image and the depth estimation result image and recognizes that a fold edge region exists at the boundary between region N and region F, and that a normal lumen region exists in region F.

[0161] As described above, the recognition unit 264 recognizes from the region segmentation result image shown in FIG. 39 and the depth estimation result image shown in FIG. 40 that a normal lumen region exists in the lower center of the endoscopic image, and that a fold region that is very close to the tip of the endoscope is adjacent to it on the right side, and supplies the recognition result to the operation content determination unit 271.

[0162] The operation content determination unit 271 receives the operation content selected from the operation content selection unit 270, and receives the recognition result of the situation around the tip from the recognition unit 264. Here, the operation content determination unit 271 selects the "angle operation DOS" of directing the tip 12 downward as the operation content. From the recognition result received from the recognition unit 264, the operation content determination unit 271 confirms that there is a fold in the lower right of the endoscope image that may come into contact when the tip 12 is directed downward. The operation content determination unit 271 may know in advance that the fold area assigned with the label value d0 is highly likely to come into contact. Therefore, the operation content determination unit 271 determines that the angle operation DOS selected by the operation content selection unit 270 is not appropriate, and changes the operation content to an angle operation DLS of directing the tip 12 to the lower left. The operation content determination unit 271 may determine the operation content of performing an angle operation upward so as to pass over the fold present in the lower right and then moving forward.

[0163] The above describes a method in which the operation content determination unit 271 determines whether the operation content selected by the operation content selection unit 270 is appropriate based on the result of the recognition unit 264 recognizing the region analysis result by the region division unit 262 and the depth estimation processing result by the depth information generation unit 263. In another example, the operation content determination unit 271 can also determine whether the operation content selected by the operation content selection unit 270 is appropriate based on the result of the recognition unit 264 recognizing the depth estimation processing result by the depth information generation unit 263. In this way, in the second embodiment, the operation content determination unit 271 may determine whether the operation content selected by the operation content selection unit 270 is appropriate based on the recognition state of the periphery of the tip of the endoscope by the recognition unit 264.

[0164] The present disclosure has been described above based on a number of examples. These embodiments and examples are merely illustrative, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present disclosure. In the examples, image processing when the endoscope 10 is inserted into the large intestine has been described, but the endoscope 10 may be inserted into other organs, or into piping, etc.

[0165] In the embodiment, an example has been described in which an endoscopic image is processed to determine the operation or manipulation of the endoscope 10, and automatic insertion control is applied. In a modified example, the determined operation or manipulation may be displayed on the display device 60 as guide information for a doctor to manually operate the endoscope 10. The determined operation or manipulation may also be recorded as log information. [Industrial Applicability]

[0166] The present disclosure can be applied to the technical field of endoscopic image processing. [Explanation of symbols]

[0167] DESCRIPTION OF SYMBOLS 1 Endoscope system, 2 Endoscope control device, 10 Endoscope, 20 Processing device, 22 Processor, 24 Storage medium, 260 Control unit, 261 Image acquisition unit, 262 Area division unit, 263 Depth information generation unit, 264 Recognition unit, 265 Operation determination unit, 266 Operation control unit, 267 Drive determination unit, 270 Operation content selection unit, 271 Operation content determination unit, 272 Operation selection model.

Claims

1. an image acquisition unit for acquiring an endoscopic image captured by an endoscope; a region dividing unit that divides the endoscopic image acquired by the image acquiring unit into regions corresponding to a plurality of structures; a depth information generating unit that generates depth information indicating a depth of the endoscopic image acquired by the image acquiring unit; a recognition unit that recognizes a region corresponding to a lumen and a region corresponding to a structure other than a lumen together with their positional relationship in the depth direction based on region information indicating the result of region division by the region division unit and depth information of the endoscopic image generated by the depth information generation unit, and identifies a direction in which the endoscope can proceed; An endoscopic image processing device comprising:

2. One of the regions of the plurality of structures is a region through which an endoscope can advance, the recognition unit recognizes that a region corresponding to a structure existing on the proximal side of a region corresponding to a lumen through which the endoscope can advance is a region corresponding to a structure that is an obstacle to the advancement of the endoscope.

2. The endoscopic image processing apparatus according to claim 1,

3. The recognition unit identifies directions in which the endoscope can advance and directions in which the endoscope must not advance from an area corresponding to a lumen through which the endoscope can advance and an area corresponding to a structure that is an obstacle to the advancement of the endoscope, and an operation determination unit that generates information regarding the traveling direction of the endoscope based on the directions in which the endoscope can travel and the directions in which the endoscope must not travel, which are identified by the recognition unit.

3. The endoscopic image processing apparatus according to claim 2.

4. an image acquisition unit for acquiring an endoscopic image captured by an endoscope; an operation content selection unit that selects one or more operation contents from a plurality of predetermined operation contents based on the endoscopic image acquired by the image acquisition unit; a region dividing unit that divides the endoscopic image acquired by the image acquiring unit into regions corresponding to a plurality of structures; a recognition unit that recognizes an area corresponding to a lumen and an area corresponding to a structure other than a lumen based on area information indicating the result of area division by the area division unit; an operation content determination unit that determines whether the operation content selected by the operation content selection unit is appropriate based on a recognition result by the recognition unit, and changes the operation content selected by the operation content selection unit when it is determined that the operation content is not appropriate; An endoscopic image processing device comprising:

5. When the operation content selected by the operation content selection unit is a forward movement operation for moving an endoscope forward, if the operation content determination unit determines that the forward movement operation selected by the operation content selection unit is not appropriate, the operation content determination unit changes the forward movement operation to another operation content.

5. The endoscopic image processing apparatus according to claim 4.

6. the operation content selection unit selects the one or more operation contents by inputting input data acquired from an endoscopic image acquired by the image acquisition unit into one or more operation selection models generated by machine learning using, as teacher data, learning images which are endoscopic images captured in the past and labels indicating operation contents for the endoscope that captured the learning images.

6. An endoscopic image processing apparatus according to claim 4 or 5.

7. A depth information generating unit generates depth information indicating a depth of the endoscopic image acquired by the image acquiring unit, the recognition unit recognizes a region corresponding to a lumen and a region corresponding to a structure other than a lumen together with a positional relationship in the depth direction based on region information indicating the result of region division by the region division unit and depth information of the endoscopic image generated by the depth information generation unit, and specifies a direction in which the endoscope can proceed.

7. An endoscopic image processing apparatus according to claim 4, wherein the image processing apparatus is a digital camera.

8. A method for operating an endoscopic image processing device, comprising: The endoscopic image processing device includes: an image acquisition unit for acquiring an endoscopic image captured by an endoscope; a region dividing unit that divides the endoscopic image acquired by the image acquiring unit into a plurality of regions; a depth information generating unit that generates depth information indicating a depth of the endoscopic image acquired by the image acquiring unit; a recognition unit that identifies a direction in which the endoscope can proceed based on area information indicating the result of area division by the area division unit and depth information of the endoscopic image generated by the depth information generation unit, an operation of the image acquisition unit acquiring an endoscopic image captured by the endoscope; an operation of the region dividing unit dividing an acquired endoscopic image into regions corresponding to a plurality of structures; an operation of the depth information generating unit generating depth information indicating a depth of an acquired endoscopic image; an operation of the recognition unit recognizing an area corresponding to a lumen and an area corresponding to a structure other than a lumen together with their positional relationship in the depth direction based on area information indicating the result of dividing the area into areas corresponding to a plurality of structures and depth information of the endoscopic image, and specifying a direction in which the endoscope can proceed; A method of operating an endoscopic image processing device, comprising:

9. On the computer, A function to acquire endoscopic images taken by the endoscope; A function to divide the acquired endoscopic image into regions corresponding to multiple structures; A function for generating depth information indicating the depth of an acquired endoscopic image; A function of recognizing an area corresponding to a lumen and an area corresponding to a structure other than a lumen, together with their positional relationship in the depth direction, based on area information indicating the results of the area division and depth information of the endoscopic image, and identifying a direction in which the endoscope can proceed; A program to achieve this.

Images