Endoscope control system and method of operating an endoscope

Abstract

An image acquisition unit (262) acquires an endoscopic image captured by an endoscope. An image classification unit (264) classifies the acquired endoscopic image as one of a plurality of types. When the endoscopic image is classified as being of a first type, an operation content determination unit (300) determines the operation content of the endoscope using an operation selection model generated by machine learning. When the endoscopic image is classified as being of a second type, the operation content determination unit (300) determines the operation content of the endoscope using an algorithm for determining the operation content.

Description

Technical Field

[0001] The present disclosure relates to a technique for determining the operation content of an endoscope from an image captured by the endoscope.

Background Art

[0002] In endoscopic observation, a flexible and elongated insertion portion is inserted into a subject's body to capture an image of the inside of the subject's body. In recent years, research has been conducted on automating the operation of the insertion portion. Patent Document 1 discloses a technique for controlling the bending angle of a bending portion in an electronic endoscope device provided with a bending portion that can be bent vertically, horizontally, and in all directions so that the tip of the insertion portion faces the center of the lumen being imaged.

[0003] Patent Document 2 discloses an endoscope system that uses an operation selection model generated by machine learning to determine one or more operation contents from a plurality of operation contents and controls the operation of the endoscope based on the determined one or more operation contents. The operation selection model is generated by performing machine learning using, as teacher data, learning images captured in the past and labels indicating the operation contents for the endoscope that captured the learning images. By inputting input data obtained from the captured endoscope image into this operation selection model, the operation content is determined.

[0004] In recent years, as a technique related to deep learning, a method for estimating depth direction information from an image has been proposed (Non-Patent Document 1), and research has also been conducted on generating depth direction information from an endoscope image (Non-Patent Document 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0006] [Non-Patent Document 1] Lei He, Guanghui Wang and Zhanyi Hu,“Learning Depth from Single Images with Deep Neural Network Embedding Focal Length”, 27 Mar 2018<URL:https: / / arxiv.org / pdf / 1803.10039.pdf> [Non-Patent Document 2] Faisal Mahmood, Richard Chen, Nicholas J. Durr, “Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training”, 29 Nov 2017<URL:https: / / arxiv.org / pdf / 1711.06606.pdf> [Overview of the project] [Problems that the invention aims to solve]

[0007] Figure 1 shows a schematic diagram of the large intestine. Areas such as the sigmoid colon, splenic flexure, and hepatic flexure have significantly curved shapes. The sigmoid colon, in particular, is not fixed and moves considerably, and although the splenic and hepatic flexures are fixed, they are not sufficiently fixed and also move, making endoscopy difficult in these areas. Endoscopy is also difficult when the intestinal tract has a complex shape or when the lumen is narrowed due to stenosis, etc. Therefore, there is a need for the development of technology in automated endoscopy insertion control that allows the endoscope tip to safely pass through areas with high insertion difficulty. Furthermore, in endoscopic examinations where physicians manually insert the endoscope, providing information on how to safely pass through areas with high insertion difficulty would be beneficial for improving physician skills.

[0008] This disclosure is made in view of the above circumstances and aims to provide technology for generating appropriate endoscope operation instructions according to the situation. [Means for solving the problem]

[0009] To solve the above problems, an endoscope control system according to one aspect of the present invention is an endoscope control system for determining the operation content of an endoscope, and comprises one or more processors having hardware. The one or more processors acquire endoscope images taken by the endoscope, classify the acquired endoscope images into one of a plurality of types, and when the endoscope image is classified into a first type, the operation content of the endoscope is determined using an operation selection model generated by machine learning, and when the endoscope image is classified into a second type, the operation content of the endoscope is determined using an algorithm for determining the operation content.

[0010] Another aspect of the present invention is an endoscope control method which acquires an endoscope image taken by the endoscope, classifies the acquired endoscope image into one of several types, determines the operation of the endoscope using an operation selection model generated by machine learning when the endoscope image is classified into a first type, and determines the operation of the endoscope using an algorithm for determining the operation when the endoscope image is classified into a second type.

[0011] Furthermore, any combination of the above components, as well as conversions of the expressions of this disclosure between methods, apparatus, systems, recording media, computer programs, etc., are also valid forms of this disclosure. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of the large intestine. [Figure 2] This is a diagram showing the configuration of the endoscope control system. [Figure 3] This diagram shows the functional blocks of the endoscope control system. [Figure 4] This figure shows an example of an endoscopic image. [Figure 5] This is a schematic diagram of the large intestine. [Figure 6] This figure shows an example of an endoscopic image. [Figure 7]It is a diagram showing an example of an endoscopic image. [Figure 8] It is a diagram showing the functional blocks of the control unit. [Figure 9] It is a diagram showing an example of teacher data. [Figure 10] It is a diagram showing another example of teacher data. [Figure 11] It is a diagram showing another example of teacher data. [Figure 12] It is a diagram showing another example of teacher data. [Figure 13] It is a diagram showing another example of teacher data. [Figure 14] It is a diagram showing another example of teacher data. [Figure 15] It is a diagram showing another example of teacher data. [Figure 16] It is a diagram showing another example of teacher data. [Figure 17] It is a diagram showing another example of teacher data. [Figure 18] It is a diagram showing another example of teacher data. [Figure 19] It is a diagram showing another example of teacher data. [Figure 20] It is a diagram showing a flowchart for determining the operation content of the endoscope. [Figure 21] It is a diagram showing an example of a learning image to which a third type of label is assigned. [Figure 22] It is a diagram showing an example of a learning image to which a fourth type of label is assigned. [Figure 23] (a) is a diagram showing an endoscopic image, and (b) is a diagram showing a region division result image. [Figure 24] (a) is a diagram showing an endoscopic image, and (b) is a diagram showing a region division result image. [Figure 25] (a) is a diagram showing an endoscopic image, and (b) is a diagram showing a depth estimation result image. [Figure 26] It is a flowchart showing the procedure for the endoscope tip to pass through the bending part. [Figure 27](a) is a figure showing an endoscopic image, (b) is a figure showing the region segmentation result image, and (c) is a figure showing the depth estimation result image. [Figure 28] This diagram illustrates a method for determining the centroid of a group of pixels. [Figure 29] (a) is a figure showing an endoscopic image, (b) is a figure showing the region segmentation result image, and (c) is a figure showing the depth estimation result image. [Figure 30] (a) is a figure showing an endoscopic image, (b) is a figure showing the region segmentation result image, and (c) is a figure showing the depth estimation result image. [Figure 31] (a) is a figure showing an endoscopic image, (b) is a figure showing the region segmentation result image, and (c) is a figure showing the depth estimation result image. [Figure 32] This flowchart shows the procedure for the tip of the endoscope to pass through the narrowed area. [Figure 33] This flowchart shows the procedure for the tip of the endoscope to pass through the area with multiple diverticula. [Figure 34] This figure shows a fourth type of endoscopic image containing one or more diverticula. [Figure 35] This figure shows an image with the lumen region and fold edge region superimposed on the endoscopic image. [Figure 36] This figure shows an example of an endoscopic image. [Modes for carrying out the invention]

[0013] The embodiments of this disclosure will be described below with reference to the drawings. Figure 2 shows the configuration of the endoscope control system 1 of the embodiment. The endoscope control system 1 is installed in the endoscopy room and comprises an endoscope control device 2, an endoscope 10, an input device 50, and a display device 60. The endoscope control device 2 has a processing device 20, an insertion shape detection device 30, and an external force information acquisition device 40, and has the function of automatically operating the endoscope 10 inserted into the patient's body. The automatic operation of the endoscope 10 is performed by the processing device 20, which comprises one or more processors 22 and a recording medium 24.

[0014] The input device 50 is an input interface operated by the user and is configured to output instructions to the processing unit 20 in response to the user's operation. The input device 50 may include, for example, an operating device such as a mouse, keyboard, or touch panel. The display device 60 is a device that displays endoscopic images or the like output from the processing unit 20 on a screen, and may be a liquid crystal display or an organic EL display.

[0015] The endoscope 10 includes an imaging unit containing a solid-state image sensor (for example, a CCD image sensor or a CMOS image sensor). The solid-state image sensor converts incident light into an electrical signal and outputs it to the processing unit 20. The processing unit 20 has a signal processing unit that performs signal processing such as A / D conversion and noise reduction on the imaging signal photoelectrically converted by the solid-state image sensor, and generates an endoscope image. Alternatively, the signal processing unit may be provided on the endoscope 10 side, and the endoscope 10 may generate the endoscope image. The processing unit 20 displays the video captured by the endoscope 10 on the display device 60 in real time.

[0016] The endoscope 10 comprises an insertion section 11 inserted into the subject, an operating section 16 provided on the proximal end of the insertion section 11, and a universal cord 17 extending from the operating section 16. The endoscope 10 is detachably connected to the processing device 20 by a scope connector (not shown) provided at the end of the universal cord 17.

[0017] The elongated insertion section 11 has a rigid tip section 12, a bendable curved section 13, and a flexible, elongated flexible tube section 14, arranged in order from the tip to the base. Inside the tip section 12, the curved section 13, and the flexible tube section 14, a plurality of source coils 18 are arranged at predetermined intervals along the longitudinal direction of the insertion section 11, and the source coils 18 generate a magnetic field in response to a coil drive signal supplied from the processing unit 20.

[0018] With the endoscope 10 inserted into the subject, when a user such as a physician operates the release switch on the control unit 16, the processing unit 20 captures the endoscopic image and transmits it to an image server (not shown) for recording. The release switch may also be provided on the input device 50. Inside the endoscope 10, there is a light guide (not shown) that transmits illumination light supplied from the processing unit 20 to illuminate the inside of the subject, and the tip 12 is provided with an illumination window for emitting the illumination light transmitted by the light guide onto the subject, and an imaging unit that photographs the subject at predetermined intervals and outputs the imaging signal to the processing unit 20.

[0019] In the endoscope control system 1 of this embodiment, the processing device 20 automatically operates the endoscope 10 and automatically controls the movement of the endoscope 10 inside the subject, but it is also possible for the user to grasp the operating unit 16 and manually operate the endoscope 10.

[0020] The operating section 16 may include operating members for the user to operate the endoscope 10. The operating section 16 includes at least angle knobs for bending the bending section 13 in eight directions intersecting the longitudinal axis of the insertion section 11. The following shows an example of basic operation of the endoscope 10. • “Forward operation (pushing operation)” to advance the insertion part 11. • A "retraction operation (pulling operation)" to retract the insertion part 11. • Angle operation to curve the curved section 13 • A "twisting operation" to rotate the insertion part 11 around the insertion axis. • "Air supply operation" to eject gas forward from the tip 12. • "Water supply operation" to eject liquid forward from the tip 12. • A "suction operation" to aspirate objects such as tissue fragments located near the tip 12. • A “search operation” to explore the center of the lumen by curving the curved portion 13 in multiple directions and directing the tip portion 12 in multiple directions.

[0021] In this embodiment, the vertical direction of the tip portion 12 is set to be perpendicular to the insertion axis of the insertion portion 11 and to correspond to the vertical direction of the solid-state image sensor provided in the imaging unit. The left-right direction of the tip portion 12 is set to be perpendicular to the insertion axis of the insertion portion 11 and to correspond to the horizontal direction of the solid-state image sensor provided in the imaging unit. Therefore, in this embodiment, the vertical direction of the tip portion 12 coincides with the vertical direction of the endoscopic image output from the signal processing unit 220, and the left-right direction of the tip portion 12 coincides with the left-right direction of the endoscopic image.

[0022] The processing unit 20 is detachably connected to each of the components of the insertion shape detection device 30, the external force information acquisition device 40, the input device 50, and the display device 60. The processing unit 20 receives user instructions input from the input device 50 and performs processing corresponding to those instructions. The processing unit 20 also acquires imaging signals periodically output from the endoscope 10 and displays the endoscopic image on the display device 60.

[0023] The insertion shape detection device 30 has the function of detecting the magnetic field generated by each of the multiple source coils 18 provided in the insertion section 11 and acquiring the position of each of the multiple source coils 18 based on the strength of the detected magnetic field. The insertion shape detection device 30 generates insertion shape information indicating the acquired positions of the multiple source coils 18 and outputs it to the processing device 20 and the external force information acquisition device 40.

[0024] The external force information acquisition device 40 stores data on the curvature (or radius of curvature) and bending angle of a predetermined number of positions on the insertion section 11 when no external force is applied, and data on the curvature (or radius of curvature) and bending angle of the same predetermined number of positions when a predetermined external force is applied to any position on the insertion section 11 from any conceivable direction. Based on the insertion shape information output from the insertion shape detection device 30, the external force information acquisition device 40 identifies the positions of a plurality of source coils 18 provided on the insertion section 11 and acquires the curvature (or radius of curvature) and bending angle at each of the plurality of source coils 18. The external force information acquisition device 40 may acquire external force information indicating the magnitude and direction of the external force at each of the plurality of source coils 18 from the acquired curvature (or radius of curvature) and bending angle and various data stored in advance. The external force information acquisition device 40 outputs the acquired external force information to the processing device 20.

[0025] Figure 3 shows the functional blocks of the endoscope control system 1 according to an embodiment. The endoscope control system 1 comprises an endoscope 10, a processing device 20, an insertion shape detection device 30, an external force information acquisition device 40, an input device 50, and a display device 60.

[0026] The endoscope 10 comprises a source coil 18, an imaging unit 110, a forward / backward mechanism 141, a bending mechanism 142, an AWS mechanism 143, and a rotation mechanism 144. The forward / backward mechanism 141, the bending mechanism 142, the AWS mechanism 143, and the rotation mechanism 144 constitute the operating mechanism of the endoscope 10.

[0027] The imaging unit 110 includes an observation window into which reflected light from a subject illuminated by illumination light enters, and a solid-state image sensor (for example, a CCD image sensor or a CMOS image sensor) that captures the reflected light and outputs an imaging signal.

[0028] The forward / backward mechanism 141 has a mechanism for realizing the operation of moving the insertion portion 11 forward and backward. For example, the forward / backward mechanism 141 may be configured to have a pair of rollers positioned opposite each other on either side of the insertion portion 11, and a motor for rotating the pair of rollers. The forward / backward mechanism 141 drives the motor in response to a forward / backward control signal output from the processing unit 20 to rotate the pair of rollers, thereby performing either the operation of moving the insertion portion 11 forward or the operation of moving the insertion portion 11 backward.

[0029] The bending mechanism 142 has a mechanism for realizing the action of bending the curved portion 13. For example, the bending mechanism 142 may be configured to have a plurality of curved pieces provided on the curved portion 13, a plurality of wires connected to the plurality of curved pieces, and a motor for pulling the plurality of wires. The bending mechanism 142 drives the motor in accordance with the bending control signal output from the processing device 20 and changes the amount of pulling of the plurality of wires, thereby bending the curved portion 13 in any of the eight directions intersecting the longitudinal axis of the insertion portion 11.

[0030] The AWS (Air feeding, Water feeding and Suction) mechanism 143 has a mechanism for realizing air supply, water supply, and suction operations. For example, the AWS mechanism 143 may be configured to have two conduits, an air supply conduit and a suction conduit, provided inside the insertion section 11, the operating section 16, and the universal cord 17, and a solenoid valve that performs an operation to open one of the two conduits while closing the other.

[0031] When the AWS mechanism 143 operates a solenoid valve to open the air supply and water supply pipeline in response to the AWS control signal output from the processing device 20, it allows a fluid containing at least one of water and air supplied from the processing device 20 to flow through the air supply and water supply pipeline and discharges the fluid from the outlet formed at the tip 12. Also, when the AWS mechanism 143 operates a solenoid valve to open the suction pipeline in response to the AWS control signal output from the processing device 20, it applies the suction force generated in the processing device 20 to the suction pipeline and uses that suction force to attract objects near the suction port formed at the tip 12.

[0032] The rotation mechanism 144 has a mechanism for realizing the operation of rotating the insertion part 11 with the insertion shaft of the insertion part 11 as the axis of rotation. For example, the rotation mechanism 144 may be configured to have a support member that rotatably supports the insertion part 11 at the base end side of the flexible tube part 14, and a motor for rotating the support member. The rotation mechanism 144 rotates the insertion part 11 around the insertion shaft by driving the motor in accordance with the rotation control signal output from the processing device 20 to rotate the support member.

[0033] The insertion shape detection device 30 includes a receiving antenna 310 and an insertion shape information acquisition unit 320. The receiving antenna 310 is configured with multiple coils that three-dimensionally detect the magnetic field generated by each of the multiple source coils 18. When the receiving antenna 310 detects the magnetic field generated by each of the multiple source coils 18, it outputs a magnetic field detection signal corresponding to the strength of the detected magnetic field to the insertion shape information acquisition unit 320.

[0034] The insertion shape information acquisition unit 320 acquires the position of each of the multiple source coils 18 based on the magnetic field detection signal output from the receiving antenna 310. Specifically, the insertion shape information acquisition unit 320 acquires multiple 3D coordinate values ​​as the positions of the multiple source coils 18 in a virtual spatial coordinate system with a predetermined position of the subject (such as the anus) as the origin or reference point. The insertion shape information acquisition unit 320 generates insertion shape information including the 3D coordinate values ​​of the multiple source coils 18 and outputs it to the control unit 260 and the external force information acquisition device 40.

[0035] The external force information acquisition device 40 acquires the curvature (or radius of curvature) and bending angle of each of the multiple source coils 18 at each position based on the insertion shape information output from the insertion shape detection device 30. The external force information acquisition device 40 may acquire external force information indicating the magnitude and direction of the external force at each of the multiple source coils 18 from the acquired curvature (or radius of curvature) and bending angle and various data stored in advance. The external force information acquisition device 40 outputs the acquired external force information to the control unit 260.

[0036] The processing unit 20 comprises a light source unit 210, a signal processing unit 220, a coil drive signal generation unit 230, a drive unit 240, a display processing unit 250, and a control unit 260. In this embodiment, the processing unit 20 functions as an image processing unit for processing endoscopic images. Specifically, the processing unit 20 generates information related to the operation or control of the endoscope 10 based on the endoscopic images and automatically controls the operation of the endoscope 10.

[0037] The light source unit 210 generates illumination light to illuminate the inside of the subject and supplies the illumination light to the endoscope 10. The light source unit 210 may have one or more LEDs or one or more lamps as light sources. The light source unit 210 may change the amount of illumination light in accordance with the operation control signal supplied from the control unit 260.

[0038] The signal processing unit 220 has a signal processing circuit that performs predetermined processing on the imaging signal output from the endoscope 10 to generate an endoscopic image, and outputs the generated endoscopic image to the display processing unit 250 and the control unit 260.

[0039] The coil drive signal generation unit 230 generates a coil drive signal for driving the source coil 18. The coil drive signal generation unit 230 has a drive circuit and generates a coil drive signal based on the operation control signal supplied from the control unit 260 and supplies it to the source coil 18.

[0040] The drive unit 240 generates control signals corresponding to the basic operation of the endoscope 10 based on the operation control signals supplied from the control unit 260, and drives the operation mechanism of the endoscope 10. Specifically, the drive unit 240 controls at least one of the following operations: forward and backward movement by the forward and backward movement mechanism 141, bending movement by the bending mechanism 142, AWS movement by the AWS mechanism 143, and rotational movement by the rotation mechanism 144. The drive unit 240 comprises a forward and backward drive unit 241, a bending drive unit 242, an AWS drive unit 243, and a rotational drive unit 244.

[0041] The forward / backward drive unit 241 generates and outputs forward / backward control signals to control the operation of the forward / backward mechanism 141 based on the operation control signals supplied from the control unit 260. Specifically, the forward / backward drive unit 241 generates and outputs forward / backward control signals to control the rotation of the motor provided on the forward / backward mechanism 141 based on the operation control signals supplied from the control unit 260.

[0042] The bending drive unit 242 generates and outputs a bending control signal to control the operation of the bending mechanism 142 based on the operation control signal supplied from the control unit 260. Specifically, the bending drive unit 242 generates and outputs a bending control signal to control the rotation of the motor provided in the bending mechanism 142 based on the operation control signal supplied from the control unit 260.

[0043] The AWS drive unit 243 generates and outputs AWS control signals to control the operation of the AWS mechanism 143 based on the operation control signals supplied from the control unit 260. Specifically, the AWS drive unit 243 generates and outputs AWS control signals to control the operating state of the solenoid valve provided in the AWS mechanism 143 based on the operation control signals supplied from the control unit 260.

[0044] The rotary drive unit 244 generates and outputs a rotation control signal to control the operation of the rotary mechanism 144 based on the operation control signal supplied from the control unit 260. Specifically, the rotary drive unit 244 generates and outputs a rotation control signal to control the rotation of the motor provided on the rotary mechanism 144 based on the operation control signal supplied from the control unit 260.

[0045] The display processing unit 250 generates a display image including the endoscopic image output from the signal processing unit 220, and displays the generated display image on the display device 60. The display processing unit 250 may also display on the display device 60 the result image of the control unit 260 processing the endoscopic image, or information regarding the operation of the endoscope 10.

[0046] The control unit 260 of this embodiment has the function of controlling the drive unit 240 in either a manual insertion mode in which a physician operates the endoscope 10, or an automatic insertion mode in which the endoscope 10 is automatically operated. When the manual insertion mode of the endoscope 10 is set to ON, the control unit 260 has the function of generating an operation control signal to cause the endoscope 10 to perform an action in response to instructions from the operation unit 16 and the input device 50, and outputting it to the drive unit 240. Furthermore, when the automatic insertion mode of the endoscope 10 is set to ON, the control unit 260 has the function of automatically controlling the operation of the endoscope 10 based on the endoscope image generated by the signal processing unit 220. Before describing the automatic insertion control in this embodiment, a brief explanation of manual operation of the endoscope by a physician will be given below.

[0047] In manual insertion mode, the physician observes the endoscopic image displayed on the display device 60 to check the situation around the tip of the endoscope and operates the endoscope accordingly. The physician instantly makes decisions such as avoiding obstacles near the tip of the endoscope, not letting the tip of the endoscope touch the mucosal surface, not putting stress on the digestive tract, and deciding on the current route by anticipating the route further ahead, and then operates the endoscope.

[0048] Figure 4(a) shows an example of an endoscopic image. Endoscopic image 70a is an image taken with an endoscope of a colon model with an intestinal tract made of a material such as rubber or silicone. When the physician looks at endoscopic image 70a and confirms that the lumen (that is, the center of the lumen, in other words the direction in which the center of the lumen is located) is in the center of the image, he decides that it is OK to advance the tip of the endoscope and moves the tip of the endoscope forward.

[0049] Figure 4(b) shows another example of an endoscopic image. Endoscopic image 70b is also an image of a colon model. The physician looks at endoscopic image 70b and confirms that the center of the lumen is located at the top of the image, and determines that if the tip of the endoscope is advanced in this state, it will come into contact with the folds in the center of the image. The physician then operates the angle knob to curve the bending section 13 upward so that the center of the lumen is captured approximately in the center of the image. Since the center of the lumen is captured approximately in the center of the image, the situation is the same as that of endoscopic image 70a shown in Figure 4(a), so the physician determines that it is OK to advance the tip of the endoscope and moves the tip of the endoscope forward.

[0050] The above judgments and operations can be easily performed by a physician. To automate the movement of this endoscope with a device, it is necessary to correctly recognize the situation around the tip of the endoscope and to identify or estimate the position of the center of the lumen.

[0051] Figure 5 shows a schematic diagram of the large intestine. In the sigmoid colon, the apex of the sigmoid colon (sometimes called the S-top among Japanese endoscopists) and the descending sigmoid colon are formed by inserting an endoscope. A bend (a curved section of the intestine) often appears at the colic junction (SDJ). In longer patients, even more and sharper bends may occur. P1 to P6 indicate the positions where the tip of the endoscope passes through the sigmoid colon, and the endoscopic images taken at each position are described below.

[0052] Figure 6(a) shows an endoscopic image taken when the tip of the endoscope is positioned at P1. Due to the fold in the intestinal tract, the endoscopic image shown in Figure 6(a) captures two structural components made up of the intestinal wall (or folds) and the arc-shaped boundary between the two structural components, but the center of the lumen (hereinafter also simply referred to as the "lumen") is not captured.

[0053] Figure 6(b) shows an endoscopic image taken when the tip of the endoscope is located at P2. As the tip of the endoscope advances from P1 to P2 and approaches the intestinal wall, only the intestinal wall is captured in the endoscopic image shown in Figure 6(b). If the tip of the endoscope advances further from P2, it will hit the intestinal wall and cause pain to the patient, so the control unit 260 needs to control the movement of the tip of the endoscope to prevent this. In this embodiment, the control unit 260 changes the orientation of the tip of the endoscope at P2 so that it faces the direction in which the lumen center is located (more precisely, the direction in which the lumen center is presumed to be located), and once the lumen center is captured, it advances the tip of the endoscope again toward the lumen center.

[0054] Figure 7(a) shows an endoscopic image taken when the tip of the endoscope is located at P3. As the tip of the endoscope passed through the sigmoid colon apex (S-top), which is a bend in the colon, as shown in Figure 7(a) The endoscopic image clearly shows the center of the lumen in the sigmoid colon. Since the center of the lumen is captured approximately in the center of the endoscopic image, the control unit 260 may advance the tip of the endoscope toward the center of the lumen. The control unit 260 may also perform a suction operation as needed after advancement to remove air from the lumen. By not introducing too much air into the lumen, insertion can be performed with less strain on the large intestine. Note that the sigmoid colon-descending colon junction (SDJ), the next bend, is captured in the background in the direction of the lumen in Figure 7(a).

[0055] Referring to Figure 5, the tip of the endoscope moves from P3 to P near the sigmoid colon descending junction (SDJ). When the endoscope tip advances to P4, an endoscopic image like the one shown in Figure 6(a) is taken again. When the endoscope tip advances further to P5, the endoscope tip comes into close proximity to the intestinal wall, and an endoscopic image like the one shown in Figure 6(b) is taken.

[0056] Figure 7(b) shows an endoscopic image taken when the tip of the endoscope is located at P6. The endoscope tip has passed through the sigmoid colon-descending colon junction (SDJ), which is a bend in the colon, as shown in Figure 7(b). The endoscopic image shown clearly captures the center of the lumen in the descending colon. Since the center of the lumen is captured approximately in the center of the endoscopic image, the control unit 260 may advance the tip of the endoscope toward the center of the lumen.

[0057] As shown in Figures 7(a) and (b), if the endoscopic image includes the center of the lumen, the control unit 260 only needs to control the endoscope's movement to advance toward the center of the lumen. In other words, when the endoscopic image includes the center of the lumen, the control unit 260 can control the endoscope's movement relatively simply. On the other hand, as shown in Figures 6(a) and (b), when the endoscopic image does not include the center of the lumen, the control unit 260 needs to estimate the position and direction of the center of the lumen and control a series of movements of the endoscope. Even when the endoscopic image includes the center of the lumen, depending on the conditions around the tip of the endoscope, it may not be desirable to advance it as usual. Therefore, the control unit 260 in this embodiment is equipped with a function to classify the captured endoscopic image into one of several types, determine the conditions around the tip of the endoscope, and control the endoscope's movement using a control method appropriate to the conditions.

[0058] Figure 8 shows the functional blocks of the control unit 260 in the embodiment. The control unit 260 comprises an image acquisition unit 262, an image classification unit 264, an image classification model 266, a first generation unit 270, a second generation unit 280, an operation content determination unit 300, and an operation control unit 302. The first generation unit 270 has an operation content selection unit 272 and an operation selection model 274. The second generation unit 280 has a region division unit 282, a depth information generation unit 284, an operation content generation unit 286, and an algorithm holding unit 288.

[0059] The control unit 260 shown in Figure 8 includes a computer, and the various functions shown in Figure 8 are realized by the computer executing a program. The computer includes, as hardware, a memory for loading the program, one or more processors for executing the loaded program, auxiliary storage devices, and other LSIs. The processor is composed of multiple electronic circuits, including semiconductor integrated circuits and LSIs, and these multiple electronic circuits may be mounted on one chip or on multiple chips. The functional blocks shown in Figure 8 are realized through the cooperation of hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination thereof.

[0060] The image acquisition unit 262 acquires endoscopic images captured by the endoscope 10 from the signal processing unit 220. The imaging unit 110 of the endoscope 10 supplies imaging signals to the signal processing unit 220 at a predetermined period (for example, 30 frames / second), and the signal processing unit 220 generates endoscopic images from the imaging signals and supplies them to the image acquisition unit 262. The image acquisition unit 262 supplies the acquired endoscopic images to the image classification unit 264, the first generation unit 270, and the second generation unit 280, respectively.

[0061] First, we will explain the operation of the first generation unit 270, which is supplied with endoscopic images. <Generation process of operation content by the first generation unit> The operation selection unit 272 has the function of selecting the operation to be performed from a predetermined set of operations based on the endoscopic image acquired by the image acquisition unit 262. The predetermined set of operations may consist of at least one of the following operations: forward movement, backward movement, angle movement, twisting movement, air insufflation, water insufflation, and suction.

[0062] The operation selection unit 272 selects the recommended operation for the endoscope 10 that is taking the endoscopic image by inputting input data obtained from the endoscopic image acquired by the image acquisition unit 262 to the operation selection model 274. The operation selection model 274 is a trained model generated by machine learning using training images, which are endoscopic images taken in the past, and labels indicating the operation content of the endoscope for the training images, as training data.

[0063] In one embodiment, the operation selection model 274 corresponds to a multilayer neural network (CNN) including an input layer, one or more convolutional layers, and an output layer. The network is generated by training each connection coefficient (weight) using a learning method such as deep learning. The training data used includes training images, which are endoscopic images taken in the past of the inside of a human intestine or colon model, and labels indicating which of the 12 possible procedures is most appropriate for the situation shown in the training images.

[0064] The 12 operations are as follows: • Angle operation UPS to curve the curved section 13 and point the tip section 12 upwards • Angle operation RIS to curve the curved section 13 and point the tip section 12 to the right. - Angle operation DOS to curve the curved section 13 and point the tip section 12 downwards • Angle operation LES to curve the curved section 13 and point the tip section 12 to the left. • Angle operation URS to curve the curved section 13 and point the tip section 12 towards the upper right. • Angle operation DRS to curve the curved section 13 and point the tip section 12 downwards to the right. • Angle operation DLS to curve the curved section 13 and point the tip section 12 downward to the left. • Angle operation ULS to curve the curved section 13 and point the tip section 12 towards the upper left. • Forward operation PSS to advance the tip 12 • Retraction operation PLS to move the tip 12 backward • Search operation SES for exploring the lumen by pointing the tip 12 in multiple directions. - Angle maintenance operation (AMS) to fix the curvature angle of the curved section 13 and maintain the orientation of the tip section 12 in its current orientation.

[0065] The annotation of endoscopic images is performed by annotators with specialized knowledge, such as physicians. The annotator examines a training image, subjectively selects the most likely operation to be performed in the situation shown in the training image from the 12 operations described above, and assigns a label for the selected operation to the training image to create training data.

[0066] For example, if the endoscopic image 70a shown in Figure 4(a) is a training image, the annotator determines that the tip of the endoscope should be advanced because the center of the lumen is located approximately in the center of the image, and assigns the label "Forward Operation PSS" to the endoscopic image 70a. On the other hand, if the endoscopic image 70b shown in Figure 4(b) is a training image, the annotator determines that the tip of the endoscope should be pointed upward because the center of the lumen is located in the upper part of the image, and assigns the label "Angle Operation UPS" to the endoscopic image 70b. This labeling process is performed on a large number of past endoscopic images to create training data.

[0067] Below is an example of training data, including training images and labels. Figure 9 shows an example of training data. All of the training images shown in Figure 9 are labeled "Angle Operation UPS" to indicate an upward angle operation. The training images shown in Figure 9 are images in which the system determined that the bending section 13 should be bent upward as the endoscopic operation to be performed.

[0068] Figure 10 shows another example of training data. All of the training images shown in Figure 10 are labeled with an "angle operation RIS label" indicating an angle operation to the right. The training images shown in Figure 10 are images in which it was determined that the bending section 13 should be bent to the right as the endoscopic operation to be performed.

[0069] Figure 11 shows another example of training data. All of the training images shown in Figure 11 are labeled with the "Angle Operation DOS" label, indicating a downward angle operation. The training images shown in Figure 11 are images in which the endoscopic operation to be performed was determined to involve bending the bending section 13 downwards.

[0070] Figure 12 shows another example of training data. All of the training images shown in Figure 12 are labeled with the "Angle Operation LES" label, indicating an angle operation to the left. The training images shown in Figure 12 are images in which it was determined that the bending section 13 should be curved to the left as the endoscopic operation to be performed.

[0071] Figure 13 shows another example of training data. All of the training images shown in Figure 13 are labeled with an "angle operation URS" indicating an angle operation in the upper right direction. The training images shown in Figure 13 are images in which it was determined that the bending section 13 should be curved in the upper right direction as the endoscopic operation to be performed.

[0072] Figure 14 shows another example of training data. All of the training images shown in Figure 14 are labeled with an "Angle Operation DRS" indicating an angle operation in the lower right direction. The training images shown in Figure 14 are images in which it was determined that the bending section 13 should be curved in the lower right direction as the endoscopic operation to be performed.

[0073] Figure 15 shows another example of training data. All of the training images shown in Figure 15 are labeled "Angle Operation DLS" to indicate an angle operation in the lower left direction. The training images shown in Figure 15 are images in which it was determined that the bending section 13 should be curved in the lower left direction as the endoscopic operation to be performed.

[0074] Figure 16 shows another example of training data. All of the training images shown in Figure 16 are labeled "Angle Operation ULS" to indicate an angle operation in the upper left direction. The training images shown in Figure 16 are images in which it was determined that the bending section 13 should be curved in the upper left direction as the endoscopic operation to be performed.

[0075] Figure 17 shows another example of training data. All of the training images shown in Figure 17 are labeled with a "push operation (forward operation) PSS" label, indicating an advancement operation. The training images shown in Figure 17 are images in which it was determined that the tip 12 should be advanced as the next endoscopic operation to be performed.

[0076] Figure 18 shows another example of training data. All of the training images shown in Figure 18 are labeled "Pull-out (Retraction) PLS" to indicate a retraction operation. The training images shown in Figure 18 are images in which it was determined that the tip 12 should be retracted as the endoscopic operation to be performed. Typical examples of situations in which a retraction operation is necessary include situations in which the tip 12 is too close to the mucosal surface of the large intestine, and situations in which the tip 12 has come into contact with the mucosal surface, commonly referred to as a "red ball" among endoscopists.

[0077] Figure 19 shows another example of training data. All of the training images shown in Figure 19 are labeled with the "Exploration Operation SES" label, indicating an exploration operation. The training images shown in Figure 19 are images for which it was determined that the bending section 13 should be bent in multiple directions and images should be taken in multiple directions as part of the endoscopic operation to be performed.

[0078] The training data for the angle maintenance operation AMS, which fixes the curvature angle of the curved portion 13 and maintains the orientation of the tip portion 12 in its current orientation, is not shown in the illustration, but for example, the training image shown in Figure 17 may be labeled "Angle Maintenance Operation AMS".

[0079] As described above, the operation selection model 274 is generated by machine learning using the training data shown in Figures 9 to 19. The curvature angle in the eight-way curvature operation, the amount of movement (forward) in the push operation, and the amount of movement (backward) in the pull operation are each set to predetermined values. For example, the curvature angle may be set to 20 degrees, and the forward and backward amounts may be set to 20 mm.

[0080] The operation selection unit 272 selects one operation from multiple options by inputting input data obtained from the captured endoscopic image to the operation selection model 274. Specifically, the operation selection unit 272 acquires multidimensional data such as the pixel value of each pixel included in the endoscopic image and inputs this multidimensional data as input data to the input layer of the neural network of the operation selection model 274. The operation selection model 274 outputs 12 likelihoods corresponding to each of the 12 possible operation contents that can be selected as operation contents for the endoscope 10 from the output layer of the neural network. The operation selection unit 272 can obtain the operation contents corresponding to the single highest likelihood among the 12 likelihoods included in the output data as the selected operation contents for the endoscope 10.

[0081] As described above, the operation content selection unit 272 is configured to input input data acquired from the endoscope image into the operation selection model 274 for processing, thereby obtaining one selection result from 12 operation contents, which include an operation to tilt the orientation of the tip portion 12 by a predetermined angle in eight directions perpendicular to the insertion axis of the insertion portion 11, an operation to advance or retract the tip portion 12 by a predetermined distance, an operation to maintain the orientation of the tip portion 12 in its current orientation, and an operation to search for the lumen near the tip portion 12. The operation content selection process using the operation selection model 274 has the advantage of requiring little computation and being able to be executed in a short time. The operation content selection unit 272 supplies the selected operation contents to the operation content determination unit 300.

[0082] Generally, the operation selection model 274 can improve its output accuracy by training with a large amount of training data. However, in special situations (unusual situations), such as when the center of the lumen is not imaged or is not clearly imaged, the operation determined using the operation selection model 274 may not be appropriate.

[0083] Referring to Figure 5, the process of the tip 12 passing through the bend will be explained. After capturing an endoscopic image including the boundary of the bend at P1 (see Figure 6(a)), the tip 12 advances to P2, which is close to the intestinal wall, and captures an endoscopic image including only the intestinal wall (see Figure 6(b)). Then, the tip 12 changes its orientation to face the direction in which the lumen center is located (the direction in which the lumen center is presumed to be located), and once the lumen center is captured in the image, it advances toward the lumen center. In summary, for the tip 12 to pass through the bend, a series of actions are required: advancing to a position close to the intestinal wall while keeping the boundary of the arc-shaped fold in the field of view, changing orientation at that position to find the lumen center, and advancing toward the lumen center to exit the bend.

[0084] However, performing this series of actions using the operation selection model 274 is not easy. For example, when the endoscopic image shown in Figure 6(b) is taken, the operation selection model 274 may select a retraction (retraction) PLS to avoid collision between the tip and the intestinal wall, resulting in a situation where the tip 12 retracts. At this time, the tip 12 may repeat the action of retracting once, then advancing again, then retracting again, and then advancing again.

[0085] Therefore, it is conceivable to prepare many class labels to respond to various situations and train the operation selection model 274 with them, but this would require preparing a huge amount of training data corresponding to each individual situation, which is not practically feasible. Therefore, the endoscope control system 1 of this embodiment recognizes the current situation in which the tip 12 of the endoscope 10 is located from the endoscope image acquired by the image acquisition unit 262, and if the situation allows, it determines the operation content using the operation selection model 274, and if the situation does not allow, it determines the operation content using another method by the second generation unit 280.

[0086] Figure 20 shows a flowchart for determining the operation of the endoscope. When the image acquisition unit 262 acquires an endoscope image (S10), the image classification unit 264 classifies the endoscope image into one of several types and recognizes the situation around the tip 12 (S12).

[0087] The image classification unit 264 identifies the type of endoscopic image by inputting input data obtained from the endoscopic image to the image classification model 266. The image classification model 266 is a trained model generated by machine learning using training images, which are endoscopic images taken in the past, and labels indicating the type of training image, as training data. In this embodiment, the image classification model 266 is generated by training each connection coefficient (weight) in a CNN (Convolutional Neural Network), which is a multi-layer neural network including an input layer, one or more convolutional layers, and an output layer, using a learning method such as deep learning.

[0088] In this embodiment, four types of class labels are used. Type 1: Standard image (primarily including the lumen) Type 2: Image of the bent area (image including the bent area) Type 3: Image of the constricted area (image including the constricted area) Type 4: Diverticular images (images including diverticula)

[0089] Type 1 endoscopic images include a clearly captured lumen and are suitable for determining the procedure using the operation selection model 274. However, even if an endoscopic image includes a clearly captured lumen, if it contains structural components (obstacles) that would hinder the passage of the endoscope tip, it is not classified as Type 1 and may be classified as another type (for example, Type 3 or Type 4).

[0090] Endoscopic images that do not fall under the first category do not include a clearly captured lumen and / or include structural components that obstruct passage, and are therefore unsuitable for determining the procedure using the procedure selection model 274. For example, a second-category endoscopic image may include at least an image that includes the boundary of a bend and does not include a clearly defined lumen; a third-category endoscopic image may include at least an image that includes a narrowed and very narrowed lumen; and a fourth-category endoscopic image may include at least an image that includes a lumen and one or more diverticula.

[0091] The annotator examines the training images, identifies the type of training image, and assigns a label of the identified type to the training image to create training data. The training images shown in Figures 7(a) and 7(b) are assigned the first type of label, while the training image shown in Figure 6(a) is assigned the second type of label.

[0092] Figures 21(a) and (b) show examples of training images to which a third type of label is assigned. In the embodiment, "stenosis" refers to a portion of the lumen that has narrowed due to some cause. Figure 21(a) shows an image of a stenosis caused by a large lesion, and Figure 21(b) shows an image of a stenosis caused by the effects of surgery or inflammation. Both images show that the lumen is very narrow. The amount of forward movement determined using the operation selection model 274 is fixed at 20 mm, but when passing through a stenosis, it is preferable to move forward carefully with an amount of forward movement smaller than 20 mm.

[0093] Figures 22(a) and (b) show examples of training images to which the fourth type of label is assigned. Both Figures 22(a) and (b) show images of multiple diverticula. While a human physician can distinguish between lumens and diverticula when viewing such endoscopic images, it is difficult for a neural network to distinguish between lumens and diverticula, and the possibility of misidentification or mis-extraction cannot be ruled out. In this situation, there are multiple candidate lumens, and it is not clear which candidate lumen is the true lumen, making it difficult to determine the procedure using the operation selection model 274.

[0094] The image classification model 266 is generated by machine learning using training data. The image classification unit 264 identifies the type of endoscopic image by inputting input data obtained from the endoscopic image into the image classification model 266. Specifically, when the image classification unit 264 inputs multidimensional data such as the pixel values ​​of each pixel included in the endoscopic image into the input layer of the neural network of the image classification model 266, the image classification model 266 outputs four likelihoods corresponding to each of the four possible image types from the output layer of the neural network. The image classification unit 264 acquires the image type corresponding to the single highest likelihood among the four likelihoods included in the output data as the type of endoscopic image.

[0095] As described above, the image classification unit 264 uses the image classification model 266 to classify the endoscopic image into one of four types and recognizes the situation around the tip 12. When the image classification unit 264 classifies the endoscopic image into the first type, it recognizes that the situation around the tip 12 is normal (Y in S14) and notifies the operation content determination unit 300 and the second generation unit 280 of this.

[0096] Upon receiving this notification, the operation content determination unit 300 determines the operation content selected by the first generation unit 270 as the operation content for the endoscope 10 (S16), and the operation control unit 302 controls the operation of the endoscope 10 according to the operation content determined using the operation selection model 274 (S20). In this way, when the endoscope image is classified into the first type, the operation content determination unit 300 determines the operation content for the endoscope 10 using the operation selection model 274. The operation content selection process using the operation selection model 274 has the advantage of requiring less computation and being able to be executed in a short time, thereby improving the stability of the automatic insertion control of the endoscope 10.

[0097] On the other hand, when the image classification unit 264 classifies the endoscopic image into a type different from the first type, it recognizes that the surrounding conditions of the tip 12 are special (N in S14) and notifies the second generation unit 280 and the operation content determination unit 300 of this.

[0098] Upon receiving this notification, the second generation unit 280 generates the operation content for the endoscope 10 using an algorithm suitable for the surrounding conditions of the tip 12, and the operation content determination unit 300 determines the operation content generated by the second generation unit 280 as the operation content for the endoscope 10 (S18). An algorithm for determining the operation content is provided for each surrounding condition, and the second generation unit 280 generates a series of operation content for the endoscope 10 using an algorithm suitable for the surrounding condition. The operation control unit 302 controls the operation of the endoscope 10 according to the operation content determined using the algorithm (S20). Even if a special situation is recognized, the first generation unit 270 may continue to supply the operation content determination unit 300 with the operation content selected using the operation selection model 274, but the operation content determination unit 300 may discard the information provided by the first generation unit 270.

[0099] During the examination (N in S22), once the operation control by S20 is complete, the process returns to step S10 and continues. When the examination is complete (Y in S22), the operation control of the endoscope is terminated.

[0100] The operation of the second generation unit 280 will be described below. <Generation process of operation content by the second generation unit> The second generation unit 280 has the function of extracting various structural components such as lumens and folds from endoscopic images using an image recognition method for structural image analysis, and generating appropriate operation content based on their size and positional relationship. The algorithm holding unit 288 holds a program that implements an algorithm for determining operation content for each situation around the tip of the endoscope. In this embodiment, the algorithm holding unit 288 holds a program that implements a bending section passage algorithm, a stenosis passage algorithm, and a diverticulum passage algorithm. The operation content generation unit 286 generates a series of endoscopic operation contents using an algorithm suitable for the situation at the tip of the endoscope, based on information about the structural components contained in the endoscopic image.

[0101] When the image acquisition unit 262 acquires an endoscopic image taken by the endoscope 10, it supplies it to the second generation unit 280. In the second generation unit 280, the endoscopic image is processed by the region division unit 282 and the depth information generation unit 284.

[0102] (Endoscopic image region segmentation processing) The region segmentation unit 282 has the function of dividing the endoscopic image acquired by the image acquisition unit 262 into multiple regions. Specifically, the region segmentation unit 282 performs semantic segmentation, which involves labeling each pixel in the endoscopic image, to divide the endoscopic image into regions corresponding to a predetermined number of structures. The region segmentation unit 282 defines regions having the type (class) of structure to be divided and generates region segmentation results with the pixels of each structure labeled. Semantic segmentation is implemented using FCN (Fully Convolutional Neural Network) or BiSeNet (Bilateral Segmentation Network), but the region segmentation unit 282 may use FCN to perform semantic segmentation.

[0103] Label values ​​from 0 to 255 may be provided as the type (class) of the region to be divided. In this embodiment, label values ​​are assigned to the following structure. Label value 0: Background pixels Label value 1: Lumen Label value 2: Fold edge (contour) Label value 3: Bending edge

[0104] In semantic segmentation, a label value of 0 generally means "regions not to be extracted," but in this embodiment, a label value of 0 means the mucosal surface. A "lumen" to which a label value of 1 is assigned means a structure in the endoscopic image that the endoscope can advance through, and is defined as a structure that indicates the direction of advancement of the endoscope tip. Specifically, a structure defined as a "lumen" represents the direction of extension of the lumen. In addition to these classes, classes may also be set for structures such as residues, polyps, and blood vessels that appear during colonoscopy, and label values ​​may be assigned to each of these classes.

[0105] Figure 23(a) shows an example of an endoscopic image. The endoscopic image has a size of 720 x 480 pixels, and each RGB pixel is represented by 8 bits. This endoscopic image captures a lumen extending in the depth direction, with folds surrounding the lumen.

[0106] Figure 23(b) shows an example of the region division result by the region division unit 282. The region division unit 282 divides the endoscopic image into multiple regions and derives region information indicating the results of the region division. The region information is derived as label values ​​pa(x,y) for each pixel relating to the structure, and the region division unit 282 generates a region division result image using the derived label values. The region division result image may be displayed on the display device 60 and presented to the user.

[0107] The region division unit 282 may set the pixel values ​​(R, G, B) corresponding to the label values ​​of the divided regions as follows. In order to distinguish them from the label values ​​related to depth information, label values ​​0 to 3 of the divided regions will be expressed as label values ​​a0 to a3 below. Label value a0 (background pixel) → (255,255,255) Label value a1 (lumen) → (128,0,0) Label value a2 (fold edge) → (0,0,128) Label value a3 (bending edge) → (128,128,128)

[0108] By setting the pixel values ​​in this manner, the region division unit 282 generates a region division result image in which the mucosal surface (label value a0), which occupies the majority, is colored white, and the extracted structural parts are colored. The region division unit 282 supplies the region division result image to the operation content generation unit 286 as region information indicating the results of region division. In the example shown in Figure 23(b), the region of the fold edge and the region of the lumen surrounded by the fold edge are displayed so that the user can see them. In another example, the region division unit 282 may supply the label value of each pixel to the operation content generation unit 286 as region information indicating the results of region division.

[0109] Figure 24(a) shows another example of an endoscopic image. In this endoscopic image, the lumen is not captured, but the boundary of the bend is. Figure 24(b) shows an example of the region division result by the region division unit 282. This region division result image includes the region of the edge (boundary) of the bent portion that extends laterally near the center. The region division unit 282 supplies the region division result image to the operation content generation unit 286 as region information indicating the result of region division.

[0110] (Processing for generating depth information from endoscopic images) The depth information generation unit 284 has the function of generating information indicating the depth of the endoscopic image acquired by the image acquisition unit 262. Various methods have been proposed to estimate the depth of pixels or blocks contained in an image. Non-patent document 2 uses 3D information from CT colonography as training data for distance information, but the depth information generation unit 284 may generate information indicating the depth of each pixel in the endoscopic image using the technology disclosed in Non-patent document 2.

[0111] The depth information generation unit 284 may generate a learning model for depth estimation processing based on simply created training data. For example, the creator of the training data (annotator) may create the training data by visually specifying label values ​​from 0 to 4 for each region of the image, according to their positional relationship in the depth direction. In this case, relative positional relationships in the depth direction based on human perception can be obtained. While it is not easy to obtain absolute numerical distance information from a normal endoscopic image, it is easy for someone skilled in viewing endoscopic images to intuitively judge whether something is close or far away. Furthermore, since physicians actually use the intuitive distance information obtained from images to perform insertion operations, the reliability of the training data created in this way is high, and it is possible to generate a learning model capable of accurately estimating depth.

[0112] In the depth estimation method by the depth information generation unit 284, classes are set according to the distance range from the endoscope tip 12. In this embodiment, a label value is assigned to each distance range. Label value 0: Depth < 1st distance Label value 1: 1st distance ≤ depth < 2nd distance Label value 2: Second distance ≤ Depth < Third distance Label value 3: 3rd distance ≤ depth < 4th distance Label value 4: 4th distance ≤ depth A label value of 0 indicates the region closest to the tip 12, while a label value of 4 indicates the region furthest from the tip 12.

[0113] Figure 25(a) shows another example of an endoscopic image. In this endoscopic image, a lumen extending in the depth direction is captured. Figure 25(b) shows an example of the depth information estimation result by the depth information generation unit 284. The depth information generation unit 284 performs depth estimation processing on the endoscope image to generate depth information indicating the depth of the endoscope image. Here, the depth information may be derived as a label value for each pixel relating to the depth (distance from the tip of the endoscope). In this embodiment, the depth information generation unit 284 generates a depth estimation result image using the derived label values. The depth estimation result image may be displayed on the display device 60 together with the region division result image and presented to the user.

[0114] The depth information generation unit 284 may set the (R, G, B) pixel values ​​corresponding to the label values ​​representing the depth stages as follows. In order to distinguish them from the label values ​​related to the divided regions, the depth information label values ​​0 to 4 will be expressed as label values ​​d0 to d4 below. Label value d0 (less than 1st distance) → (200,0,0) Label value d1 (greater than or equal to the 1st distance, less than the 2nd distance) → (160,0,0) Label value d2 (2nd distance or greater, less than 3rd distance) → (120,0,0) Label value d3 (3rd distance or greater, less than 4th distance) → (80,0,0) Label value d4 (4th distance or greater) → (40,0,0)

[0115] The depth information generation unit 284 generates a depth estimation result image in which distant areas are colored darker red and closer areas are colored brighter red by setting the pixel values ​​in this way. The depth information generation unit 284 supplies the depth estimation result image to the operation content generation unit 286 as depth information for the endoscope image. In another example, the depth information generation unit 284 may supply the label value of each pixel to the operation content generation unit 286 as depth information for the endoscope image.

[0116] (Method of passing through the curved section) The following describes how to pass through a bend using an algorithm. Figure 26 is a flowchart showing the procedure for the tip of the endoscope to pass through a bent section. When the image classification unit 264 classifies the captured endoscope image into a second type, the second generation unit 280 is activated, and the operation content generation unit 286 reads a program that implements the bent section passage algorithm from the algorithm holding unit 288 and starts the process of generating a series of operation contents for the tip of the endoscope to pass through the bent section.

[0117] The second generation unit 280 receives a second type of endoscopic image, including the boundary of the bent portion, from the image acquisition unit 262 (S40). Figure 27(a) shows a second type of endoscopic image including the boundary of the bend. The region division unit 282 divides the endoscopic image into multiple regions, generates region information showing the results of the region division, and provides it to the operation content generation unit 286. Figure 27(b) shows the region division result image generated by the region division unit 282. The depth information generation unit 284 also performs depth estimation processing on the endoscopic image, generates depth information showing the depth of the endoscopic image, and provides it to the operation content generation unit 286. Figure 27(c) shows the depth estimation result image generated by the depth information generation unit 284. From the region division result image shown in Figure 27(b), the operation content generation unit 286 recognizes the presence of a bend edge (Y in S42), and from the depth estimation result image shown in Figure 27(c), it recognizes that the structural components on the upper side of the image are located closer to the user, and the structural components on the lower side of the image are located further away from the user.

[0118] Since a bent section is formed when two structural components (folds or intestinal wall) overlap in the depth direction, it is highly likely that a lumen exists in a hidden position behind the front structural component, extending from the back structural component toward the front structural component. For example, in the bent section shown in Figure 27(a), the front structural component is in the upper part of the image and the back structural component is in the lower part of the image. Therefore, it is highly likely that a lumen exists behind the upper structural component, extending from the lower structural component toward the upper structural component. The operation content generation unit 286 uses this inference theory to infer that the lumen is located above based on region information and depth information, and stores the inferred direction of the lumen in memory (not shown) (S44).

[0119] Next, the operation content generation unit 286 determines whether or not the centroid of the bent edge is located near the center of the image (S46). The centroid of the bent edge being located near the center of the image is a condition for advancing the tip of the endoscope. The operation content generation unit 286 determines from the region information whether or not the centroid of the group of pixels extracted as the bent edge is located near the center of the image.

[0120] Figure 28 is a diagram illustrating a method for determining the centroid B of a pixel group A extracted as a bent edge. If the centroid B is located on pixel group A, the operation content generation unit 286 uses the position of the centroid B as is. On the other hand, if the centroid B is not located on pixel group A but is located outside of pixel group A, the operation content generation unit 286 may use the position on pixel group A closest to the centroid B as the centroid position. If the centroid position is included in the "central region", the operation content generation unit 286 determines that the centroid of the bent edge is located near the center of the image. On the other hand, if the centroid position is not included in the "central region", the operation content generation unit 286 determines that the centroid of the bent edge is not located near the center of the image.

[0121] The operation content generation unit 286 determines the centroid position of the bent edge from the region division result image shown in Figure 27(b) and determines that the centroid of the bent edge is not located near the center of the image (N in S46). At this time, the operation content generation unit 286 generates an operation to point the tip upward in order to move the edge centroid towards the center of the image and notifies the operation content determination unit 300. The operation content determination unit 300 decides that the upward angle operation generated by the operation content generation unit 286 is the operation to be performed (S48), and the operation control unit 302 generates an operation control signal corresponding to the determined operation and supplies it to the drive unit 240. In this case, the operation control unit 302 supplies an operation control signal to the drive unit 240 to bend the angle upward, and the drive unit 240 bends the bent portion 13 upward, changing the direction of the tip 12 upward. For example, the bending angle may be 10 degrees.

[0122] After changing the orientation of the tip portion 12 upward, the second generation unit 280 receives the endoscopic image from the image acquisition unit 262 (S40). Figure 29(a) shows an endoscopic image taken after changing the orientation of the tip 12. The region division unit 282 generates region information showing the result of region division and provides it to the operation content generation unit 286. Figure 29(b) shows the region division result image generated by the region division unit 282. The depth information generation unit 284 generates depth information showing the depth of the endoscopic image and provides it to the operation content generation unit 286. Figure 29(c) shows the depth estimation result image generated by the depth information generation unit 284. The operation content generation unit 286 determines from the region division result image shown in Figure 29(b) that a bent edge exists (Y in S42). The operation content generation unit 286 also uses inference theory to infer that the lumen is located on the upper side based on the region information and depth information, and stores the direction in which the inferred lumen is located in memory (not shown) (S44).

[0123] Next, the operation content generation unit 286 determines the centroid position of the bent edge from the region division result image shown in Figure 29(b) and determines that the centroid of the bent edge is located near the center of the image (Y in S46). Since the location of the centroid of the bent edge near the center of the image is a condition for advancing the endoscope tip, the operation content generation unit 286 generates an operation to advance and notifies the operation content determination unit 300. The operation content determination unit 300 decides that the advancement operation generated by the operation content generation unit 286 is the operation to be performed (S50), and the operation control unit 302 supplies an operation control signal to the drive unit 240 to advance the tip 12, and the drive unit 240 advances the tip 12. For example, the amount of advancement may be set to 20 mm.

[0124] After the tip 12 is advanced, the second generation unit 280 receives the endoscopic image from the image acquisition unit 262 (S40). Figure 30(a) shows an endoscopic image taken after the tip 12 has been advanced. The region division unit 282 generates region information showing the results of the region division and provides it to the operation content generation unit 286. Figure 30(b) shows the region division result image generated by the region division unit 282. The depth information generation unit 284 also generates depth information showing the depth of the endoscopic image and provides it to the operation content generation unit 286. Figure 30(c) shows the depth estimation result image generated by the depth information generation unit 284.

[0125] The endoscopic image shown in Figure 30(a), like the endoscopic image shown in Figure 6(b), is an image taken in close proximity to the intestinal tract. In other words, at S50, the tip 12 has advanced, and the tip 12 has entered the bent section. The operation content generation unit 286 determines from the region segmentation result image shown in Figure 30(b) that there is no bent section edge (N at S42).

[0126] The operation content generation unit 286 determines whether or not a lumen exists from the region division result image shown in Figure 30(b) (S52). Since the region division result image shown in Figure 30(b) does not include a lumen, the operation content generation unit 286 determines that a lumen does not exist (N in S52). At this time, the operation content generation unit 286 generates an operation content to point the tip in the direction stored in memory in S44 and notifies the operation content determination unit 300. In this example, "upward" is stored in memory, so the operation content generation unit 286 notifies the operation content determination unit 300 of an operation content to point the tip upward. The operation content determination unit 300 decides that the upward angle operation generated by the operation content generation unit 286 is the operation content to be performed (S54), and the operation control unit 302 generates an operation control signal according to the determined operation content and supplies it to the drive unit 240. In this case, the motion control unit 302 supplies an motion control signal to the drive unit 240 to bend the angle upward, and the drive unit 240 bends the curved portion 13 upward, changing the orientation of the tip portion 12 upward. For example, the bending angle may be 20 degrees.

[0127] After changing the orientation of the tip portion 12 upward, the second generation unit 280 receives the endoscopic image from the image acquisition unit 262 (S40). Figure 31(a) shows an endoscopic image taken after changing the orientation of the tip portion 12 to upward. The region division unit 282 generates region information showing the result of region division and provides it to the operation content generation unit 286. Figure 31(b) shows the region division result image generated by the region division unit 282. The depth information generation unit 284 also generates depth information showing the depth of the endoscopic image and provides it to the operation content generation unit 286. Figure 31(c) shows the depth estimation result image generated by the depth information generation unit 284. The operation content generation unit 286 determines from the region division result image shown in Figure 31(b) that there is no bending edge (N in S42).

[0128] The right side of the region segmentation result image shown in Figure 31(b) includes a lumen, and the operation content generation unit 286 determines that a lumen is present (Y in S52). When the operation content generation unit 286 determines that a lumen is included in the endoscopic image, it terminates the operation content generation process using the bend passage algorithm and notifies the operation content determination unit 300 of this. At this time, the image classification unit 264 classifies the endoscopic image shown in Figure 31(a) into a first type and notifies the operation content determination unit 300 and the second generation unit 280 that the surrounding conditions of the tip 12 are normal. Upon receiving this notification, the operation content generation unit 286 may terminate the operation content generation process. After receiving the notification, the operation content determination unit 300 determines that the operation content selected by the first generation unit 270, rather than the operation content generated by the second generation unit 280, is the operation content for the endoscope 10, and the operation control unit 302 controls the operation of the endoscope 10 according to the operation content determined using the operation selection model 274.

[0129] As described above, when a bent section is included in the endoscopic image, the second generation unit 280 performs a process to generate the operation content using a bent section passage algorithm, thereby determining a series of operations suitable for passing through the bent section, and contributing to a reduction in examination time.

[0130] In addition, there are situations where it is difficult for the tip 12 to smoothly pass through the bend due to reasons such as an excessively long sigmoid colon. In such situations, it is preferable to avoid forcing the insertion operation to continue. For example, if angle operations in the same direction continue for a predetermined number of times (e.g., 7 times), or if forward operations continue for a predetermined number of times (e.g., 5 times), it may be determined that passage through the bend has failed, and the tip 12 may be temporarily retracted. After retraction, the operation content generation unit 286 may resume the process of generating operation content using the bend passage algorithm.

[0131] In the flowchart shown in Figure 26, for the process of estimating the direction in which the lumen exists (S44), it is also possible to use a CNN trained with training data that uses the estimated direction of the lumen as a class for images of the bend.

[0132] (Method for passing through a narrowed area) The following describes how to pass through the narrow section using an algorithm. Figure 32 is a flowchart illustrating the procedure for the tip of the endoscope to pass through a narrowed area. A narrowed area occurs when the lumen is blocked by a lesion or when the lumen itself becomes narrower due to the effects of surgery or inflammation. When performing endoscopic movements to pass through a narrowed area, it is desirable to accurately point the tip 12 towards the narrowed lumen and carefully insert it into the lumen while avoiding contact with the lesion or other affected area.

[0133] When the image classification unit 264 classifies the captured endoscopic image into a third type, the second generation unit 280 is activated, and the operation content generation unit 286 reads a program that implements the stenosis passage algorithm from the algorithm holding unit 288 and starts the process of generating a series of operation contents for the tip of the endoscope to pass through the stenosis.

[0134] The second generation unit 280 receives a third type of endoscopic image, including the stricture, from the image acquisition unit 262 (S60). The third type of endoscopic image includes the stricture (Y in S62).

[0135] The operation content generation unit 286 determines whether the centroid of the lumen is located in the center of the image (S64). The operation content generation unit 286 determines whether the centroid of the pixel group extracted as a lumen is located in the center of the image from the region information generated by the region division unit 282.

[0136] If the center of gravity of the lumen is not in the center of the image (N in S64), the operation content generation unit 286 generates an operation to change the orientation of the tip in order to move the center of gravity of the lumen to the center of the image, and notifies the operation content determination unit 300. The operation content determination unit 300 decides on the angle operation generated by the operation content generation unit 286 as the operation to be performed (S66), and the motion control unit 302 generates an operation control signal according to the determined operation and supplies it to the drive unit 240. The drive unit 240 bends the curved section 13 according to the operation control signal and changes the orientation of the tip section 12. For example, the bending angle may be 10 degrees.

[0137] After changing the orientation of the tip 12, the second generation unit 280 receives an endoscopic image including the stenosis (Y in S60 and S62). If the center of gravity of the lumen is in the center of the endoscopic image (Y in S64), the operation content generation unit 286 determines from the depth information generated by the depth information generation unit 284 whether or not there are structural components (folds or intestinal wall) around the lumen that would obstruct the advancement of the tip 12 (S68).

[0138] If there is a structural component (obstacle) that hinders forward movement (Y in S68), the operation content generation unit 286 generates an operation to change the orientation of the tip in order to move the obstacle toward the outside of the image (away from the lumen) while maintaining the position of the lumen in the center of the image, and notifies the operation content determination unit 300. The operation content determination unit 300 decides on the angle operation generated by the operation content generation unit 286 as the operation to be performed (S70), and the motion control unit 302 generates an operation control signal according to the determined operation and supplies it to the drive unit 240. The drive unit 240 bends the curved section 13 according to the operation control signal and changes the orientation of the tip 12.

[0139] If no obstacles are present (N in S68), the operation content generation unit 286 generates an operation to move forward and notifies the operation content determination unit 300. The operation content determination unit 300 determines the forward operation generated by the operation content generation unit 286 as the operation to be performed (S72), and the operation control unit 302 supplies an operation control signal to the drive unit 240 to move the tip 12 forward, and the drive unit 240 moves the tip 12 forward. For example, it is preferable to set the amount of forward movement to 10 mm or 5 mm and move it forward carefully.

[0140] After advancing the tip 12, the operation content generation unit 286 terminates the operation content generation process using the stenosis passage algorithm when the endoscopic image includes a lumen that is not stenotic (N in S62), and notifies the operation content determination unit 300 of this. At this time, the image classification unit 264 classifies the endoscopic image acquired by the image acquisition unit 262 into a first type and notifies the operation content determination unit 300 and the second generation unit 280 that the surrounding conditions of the tip 12 are normal. Upon receiving this notification, the operation content generation unit 286 may terminate the operation content generation process. After receiving the notification, the operation content determination unit 300 decides that the operation content selected by the first generation unit 270, rather than the operation content generated by the second generation unit 280, will be the operation content for the endoscope 10, and the operation control unit 302 controls the operation of the endoscope 10 according to the operation content determined using the operation selection model 274.

[0141] As described above, when a stricture is included in the endoscopic image, the second generation unit 280 performs a process to generate the operation content using a stricture passage algorithm, thereby determining a series of operations suitable for passing through the stricture, and contributing to a reduction in examination time.

[0142] (Method of passing through areas with multiple diverticula) The following describes how to pass through areas with multiple diverticula using an algorithm. Figure 33 is a flowchart showing the procedure for the endoscope tip to pass through a region with multiple diverticula. Diverticula are areas where the intestinal wall partially protrudes outside the large intestine, sometimes exhibiting a structure similar to a lumen. When one or more diverticula are present in the endoscopic image, it is desirable for the endoscopic operation to locate the true lumen from among multiple candidate lumen regions, and for the tip 12 to pass through the true lumen.

[0143] When the image classification unit 264 classifies the captured endoscopic image into the fourth type, the second generation unit 280 is activated, and the operation content generation unit 286 reads a program that implements the diverticulum passage algorithm from the algorithm holding unit 288 and starts the process of generating a series of operations for the tip of the endoscope to pass through the diverticulum. The diverticulum passage algorithm extracts multiple regions that are candidates for lumen from the endoscopic image, sets a reference position for the tip 12, and generates a series of operations that involve slightly inserting the tip 12 into each region from the reference position to investigate whether or not it is a true lumen.

[0144] The second generation unit 280 receives a fourth type endoscopic image containing one or more diverticula from the image acquisition unit 262 (S80). Figure 34 shows a fourth type of endoscopic image containing one or more diverticula. The region division unit 282 divides the endoscopic image into multiple regions, generates region information showing the results of the region division, and provides it to the operation content generation unit 286. Figure 35(a) shows an image in which the lumen regions and fold edge regions extracted by the region division unit 282 are superimposed on the endoscopic image shown in Figure 34. In the image shown in Figure 35(a), three lumen regions have been extracted, which means that the region division unit 282 has mistakenly extracted at least two diverticula as lumens. Therefore, the operation content generation unit 286 generates a series of operation contents to search for the true lumen region from the multiple lumen regions.

[0145] The operation content generation unit 286 assigns numbers to the lumen regions derived by the region division unit 282 and treats them as candidate lumen regions 1 to k (S82). In this example, k=3. Figure 35(b) shows the state after region numbers have been added to the candidate lumen regions.

[0146] In this algorithm, the tip 12 is controlled to enter each of the lumen candidate regions 1 to k from the reference position and check whether or not it is a true lumen. If it is determined that it is not a lumen after entering, the tip 12 returns to the reference position and enters another lumen candidate region to check whether or not it is a true lumen. For this reason, the operation content generation unit 286 records the endoscopic image shown in Figure 34 and information for identifying the reference position when this endoscopic image was taken (reference position information) in memory (not shown) (S84). The reference position information may also be information for identifying the state of the endoscope 10 when the tip 12 is at the reference position, and may be, for example, the insertion length and insertion shape of the endoscope 10.

[0147] The following describes the operation for confirming the lumen candidate region. Setting k to 1 (S86), the operation content generation unit 286 sequentially generates operations for inserting the tip 12 into region 1 and notifies the operation content determination unit 300. The operation content determination unit 300 determines the operations generated by the operation content generation unit 286 as the operations to be performed, and the operation control unit 302 generates an operation control signal corresponding to the determined operations and supplies it to the drive unit 240. When the tip 12 reaches region 1, the drive unit 240 advances it by a predetermined amount. The advance amount is set to, for example, 5 mm, and the tip 12 is carefully advanced so that it does not come into contact with the mucosal surface if region 1 is not a true lumen (S88). If region 1 is a diverticulum, the tip 12 approaches the mucosal surface, and an endoscopic image like the one shown in Figure 36 is taken. When the operation content generation unit 286 captures an image close to the mucosal surface, it determines that it is a dead end and determines that region 1 is not a lumen (N in S90).

[0148] Next, the operation content generation unit 286 sequentially generates operations to return the tip 12 to the reference position based on the information recorded in S84, notifies the operation content determination unit 300, and moves the tip 12 to the reference position. The operation content generation unit 286 increments k by 1 (S94), sequentially generates operations to insert the tip 12 into region 2, and notifies the operation content determination unit 300. If region 2 is not a lumen (N in S90), the operation content generation unit 286 generates similar operations for region 3 and determines whether region 3 is a lumen or not (S90). If region 3 is a lumen (Y in S90), this algorithm terminates. This algorithm also terminates if no lumen is found after investigating all candidate lumen regions.

[0149] As described above, when a diverticulum is included in the endoscopic image, the second generation unit 280 performs the process of generating the operation content using the diverticulum passage algorithm, thereby determining a series of operations suitable for passing through the diverticulum, and contributing to a reduction in examination time.

[0150] The present disclosure has been described above based on embodiments. These embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications are also within the scope of the present disclosure. In the embodiments, the process of inserting the endoscope 10 into the large intestine was described, but the endoscope 10 may be inserted into other organs, or into piping, etc. In the embodiments, the operation content of the endoscope was described using an operation selection model generated by machine learning for images classified as type 1, but it is also possible to configure the system to use control such as extracting lumens by semantic segmentation for images classified as type 1 and performing angle operations in that direction.

[0151] In this embodiment, an example was described in which endoscopic images are processed to determine the operation content of the endoscope 10 and applied to automatic insertion control. In a modified example, the display processing unit 250 may display information regarding the operation content determined by the operation content determination unit 300 on the display device 60 as guide information for when a physician manually operates the endoscope 10. The display processing unit 250 may also display the region division result image generated by the region division unit 282 and the depth estimation result image generated by the depth information generation unit 284 on the display device 60 as guide information. By displaying information regarding the operation content, the region division result image, and / or the depth estimation result image on the display device 60, the display processing unit 250 can support the physician's safe operation of the endoscope. The operation content determined by the operation content determination unit 300 may be recorded as log information. [Industrial applicability]

[0152] This disclosure can be used in the field of technology for processing endoscopic images. [Explanation of Symbols]

[0153] 1...Endoscope control system, 2...Endoscope control device, 10...Endoscope, 11...Insertion section, 12...Tip section, 13...Bending section, 20...Processing device, 22...Processor, 24...Recording medium, 30...Insertion shape detection device, 40...External force information acquisition device, 50...Input device, 60...Display device, 110...Imaging unit, 141...Advance / retraction mechanism, 142...Bending mechanism, 143...AWS mechanism, 144...Rotation mechanism, 210...Light source unit, 220...Signal processing unit, 230...Coil drive signal generation unit, 240...Drive unit, 241...Advance / retraction drive unit, 2 42...Bending drive unit, 243...AWS drive unit, 244...Rotation drive unit, 250...Display processing unit, 260...Control unit, 262...Image acquisition unit, 264...Image classification unit, 266...Image classification model, 270...First generation unit, 272...Operation content selection unit, 274...Operation selection model, 280...Second generation unit, 282...Region division unit, 284...Depth information generation unit, 286...Operation content generation unit, 288...Algorithm holding unit, 300...Operation content determination unit, 302...Motion control unit, 310...Receiving antenna, 320...Insertion shape information acquisition unit.

Claims

1. An endoscope control system for determining the insertion procedure of an endoscope, comprising one or more processors having hardware, The one or more processors described above are: The endoscope captures an image of the lumen, The acquired lumen images are classified into one of several types. If the lumen image is classified as a normal image (Type 1), the insertion procedure of the endoscope is determined using an insertion procedure selection model generated by machine learning. If the lumen image is classified as a second type, which is an image that does not fall under the first type and contains structural components that obstruct the passage of the endoscope tip, then the insertion procedure of the endoscope is determined using an algorithm for determining the insertion procedure. An endoscope control system characterized by the following features.

2. The one or more processors described above are: When the lumen image is of type 1, the operation of the endoscope is controlled according to the insertion operation content determined using the insertion operation selection model. When the lumen image is of type 2, the operation of the endoscope is controlled according to the insertion operation determined using the algorithm. The endoscopic control system according to claim 1.

3. The one or more processors described above are: When the lumen image is of type 1, information regarding the insertion operation determined using the insertion operation selection model is displayed on the display device. If the lumen image is of type 2, information regarding the insertion procedure determined using the algorithm is displayed on the display device. The endoscopic control system according to claim 1.

4. The one or more processors described above are: When the lumen image includes the boundary of a bend, the lumen image is classified into a second type. The endoscopic control system according to claim 1.

5. The one or more processors described above are: If the lumen image includes a narrowed portion, the lumen image is classified into a second type. The endoscopic control system according to claim 1.

6. The one or more processors described above are: If the lumen image includes a diverticulum, the lumen image is classified into a second type. The endoscopic control system according to claim 1.

7. The aforementioned insertion procedure selection model is generated by machine learning using training images, which are previously captured images of the tubular cavity, and labels indicating the content of the endoscopic insertion procedure for those training images, as training data. The endoscopic control system according to claim 1.

8. The one or more processors described above are: When the lumen image is classified into a second type, the algorithm is used to determine the insertion procedure of the endoscope based on the information regarding the structural components within the lumen image. The endoscopic control system according to claim 1.

9. The one or more processors described above are: Based on the information indicating the depth of the lumen image, the algorithm is used to determine the insertion procedure of the endoscope. The endoscopic control system according to claim 8.

10. The one or more processors described above are: The aforementioned tubular images are classified into a first type or a second type using an image classification model generated by machine learning. The endoscopic control system according to claim 1.

11. The one or more processors described above are: When the lumen image is classified into the second type, the sequence of insertion operations for the endoscope is determined using the algorithm, which is suitable for the conditions around the tip of the endoscope. The endoscopic control system according to claim 1.

12. The one or more processors described above are: The completion of the process of determining the sequence of insertion operations of the endoscope using the aforementioned algorithm is determined based on the lumen image. The endoscopic control system according to feature 11.

13. The one or more processors described above are: The following insertion operations are determined: an insertion operation to change the orientation of the tip of the endoscope so that the boundary of the bent portion is moved to near the center of the lumen image; an insertion operation to advance the tip; and an insertion operation to change the orientation of the tip after it has entered the bent portion so that it faces the direction in which the lumen is presumed to exist. The endoscopic control system according to feature 4.

14. The one or more processors described above are: The system determines the following insertion operations: changing the orientation of the tip of the endoscope so that the lumen is moved to the center of the lumen image; changing the orientation of the tip of the endoscope so that an obstacle that would hinder the advancement of the tip is moved toward the outside of the lumen image; and advancing the endoscope. The endoscope control system according to claim 5, characterized in that it is as described above.

15. The one or more processors described above are: The system determines the insertion procedure for inserting the tip of the endoscope into one of several candidate lumen regions, and, if the candidate lumen region into which the tip was inserted is not a lumen, the system determines the insertion procedure for inserting the tip into another candidate lumen region. The endoscopic control system according to claim 6.

16. A method for operating an endoscope, One or more processors having hardware, The endoscope captures an image of the lumen, The acquired lumen images are classified into one of several types. If the lumen image is classified as a normal image (Type 1), the insertion procedure of the endoscope is determined using an insertion procedure selection model generated by machine learning. If the lumen image is classified as a second type, which is an image that does not fall under the first type and contains structural components that obstruct the passage of the endoscope tip, then the insertion procedure of the endoscope is determined using an algorithm for determining the insertion procedure. A method for operating an endoscope, characterized by the features described above.

17. On the computer, The function of acquiring images of the lumen captured by the endoscope, A function to classify the acquired lumen image into one of several types, When the lumen image is classified as a normal image (Type 1), the function determines the content of the endoscope insertion operation using an insertion operation selection model generated by machine learning. If the lumen image is classified as a second type, which is an image that does not fall under the first type and contains structural components that obstruct the passage of the endoscope tip, then a function is provided to determine the insertion procedure of the endoscope using an algorithm for determining the insertion procedure. A recording medium containing a program to achieve this.

Images