Endoscope insertion control device, endoscope insertion control method, and recording medium on which endoscope insertion control program is recorded

Abstract

An endoscope insertion control device is provided with: a processor that performs a process of selecting an operation of an endoscope, recording the operation, generating a control signal for executing the selected operation, classifying insertion states on the basis of an image captured by the endoscope, determining whether or not the classified insertion states correspond to the recorded operation, and if so, generating a control signal for executing the control signal; when the classified insertion state corresponds to the recorded operation, the processor selects the operation of the endoscope.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201980100076.2, filed on October 1, 2019, entitled "Endoscope Insertion Control Device, Endoscope Insertion Control Method and Recording Medium Recording Endoscope Insertion Control Program". Technical Field

[0002] The present invention relates to an endoscope insertion control device, an endoscope insertion control method, and a recording medium containing an endoscope insertion control program, which enables reliable and easy insertion of the endoscope insertion section. Background Technology

[0003] In addition, medical endoscopes are widely used for observing deep organs within body cavities without cutting the body surface by inserting a slender insertion part into the convoluted body cavity, or for performing various treatments and procedures by using a treatment instrument inserted through the treatment instrument channel of the endoscope insertion part as needed.

[0004] Doctors insert the endoscope into body cavities using their hands or other means to observe organs. However, the tubing inside the body cavity through which the endoscope is inserted is elastic. At bends and other points in the tubing, the tubing can become kinked by the force of pushing the endoscope in, sometimes preventing the endoscope from advancing smoothly.

[0005] Therefore, the insertion of an endoscope requires skill. Especially in endoscopic insertion of the colon, the long intestinal tract exhibits complex movement patterns. Furthermore, the transverse and S-shaped colonic portions are not fixed within the body cavity but are mobile, making insertion difficult and potentially causing pain to the patient depending on the insertion method. Therefore, skillful insertion of an endoscope into the colon is essential. Consequently, Japanese Patent No. 3645223 discloses an endoscope bending device that facilitates insertion. In this proposed solution, dark areas are detected using images from a camera unit located at the front end of the insertion section, thereby detecting the center of the lumen and calculating a bending angle indicator value, ensuring that the bending section of the insertion section is oriented towards the center of the lumen.

[0006] Furthermore, in recent years, automated insertion endoscopes capable of automatically inserting endoscopes into lumens have been developed. In such automated insertion endoscopes, the tip of the endoscope can be aligned with the direction of travel by detecting the direction of travel in the lumen, thereby advancing the tip. As a result, the insertion portion of the endoscope can reach, for example, the deep part of the large intestine.

[0007] However, whether insertion is performed by a physician or by an automated endoscope, the insertion state of the inserter does not always change or progress as the physician desires. For example, even after insertion (advancement) is performed, the insertion state may not be optimal. Sometimes, due to endoscope flexion or friction with the intestinal wall, it may only advance slightly, and in cases where a so-called "rod phenomenon" occurs in the splenic bend, the tip may move away from the direction of travel. In such situations, continuing to advance the endoscope may worsen the insertion and cause pain to the patient. Therefore, it is difficult to complete endoscopic insertion without constantly monitoring the behavior of the inserter or tip relative to the desired operation and performing appropriate actions accordingly. However, this has not been considered in existing endoscopic insertion support devices and automated endoscopes. Therefore, even using technologies such as Japanese Patent No. 3645223, insertion may not be smooth. Furthermore, for example, Patent Publication No. 4855901 discloses an endoscope insertion support device that detects rings or flexures based on the shape of the endoscope insertion and prompts a release operation. However, although it can provide support information after the rings or flexures are generated, it is difficult to detect the insertion status based on changes caused by a single operation of the doctor or the automatic endoscope insertion device. In addition, there is a problem that it may not be aware of conditions such as friction with the mucosal surface that are not reflected in the insertion shape.

[0008] The purpose of this invention is to provide an endoscope insertion control device, an endoscope insertion control method, and a recording medium recording an endoscope insertion control program, which evaluates the behavior of the endoscope insertion unit for a certain insertion operation based on an image, and selectively performs subsequent operations based on the evaluation results, thereby effectively supporting reliable and smooth endoscope insertion. Summary of the Invention

[0009] means for solving technical problems

[0010] An endoscope insertion control device according to one aspect of the present invention comprises: an image acquisition unit that acquires a plurality of images related to the insertion of an endoscope obtained in a time sequence; a classification unit that classifies the insertion state of the endoscope based on the plurality of images; and an operation selection unit that selects a subsequent operation related to the insertion of the endoscope with reference to the classification result, wherein the operation selection unit relaxes the advancement conditions when the classification unit classifies the insertion state as the insertion state related to advancement.

[0011] One aspect of the present invention provides an endoscope insertion control method, wherein: a plurality of images relating to endoscope insertion are acquired in a time-series manner; the insertion states of the endoscope are classified based on the acquired plurality of images; a subsequent operation relating to the insertion of the endoscope is selected with reference to the classification results; and when the insertion state is classified as an insertion state related to forward movement, forward movement conditions are relaxed.

[0012] One aspect of the present invention includes a recording medium having an endoscope insertion control program that causes a computer to perform the following processes: acquiring multiple images related to endoscope insertion obtained in a time sequence; classifying the insertion state of the endoscope based on the multiple images; and selecting a subsequent operation related to the insertion of the endoscope with reference to the classification results, wherein the process of selecting the next operation is a process of relaxing the advance conditions when the insertion state is classified as an insertion state related to advance. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating an endoscope system incorporating an endoscope insertion control device according to a first embodiment of the present invention.

[0014] Figure 2A This is a block diagram used to illustrate the specific structure of the endoscope system 1.

[0015] Figure 2B This is a diagram illustrating an example of a forward / reverse mechanism.

[0016] Figure 3A It is an explanatory diagram used to illustrate time series images and the labels attached to them.

[0017] Figure 3B It is an explanatory diagram used to illustrate time series images and the labels attached to them.

[0018] Figure 3C It is an explanatory diagram used to illustrate time series images and the labels attached to them.

[0019] Figure 3D It is an explanatory diagram used to illustrate time series images and the labels attached to them.

[0020] Figure 3E It is an explanatory diagram used to illustrate time series images and the labels attached to them.

[0021] Figure 3F It is an explanatory diagram used to illustrate time series images and the labels attached to them.

[0022] Figure 4 This is a flowchart used to explain the actions of the implementation method.

[0023] Figure 5 This is a block diagram illustrating the second embodiment of the present invention.

[0024] Figure 6 This is an illustrative diagram showing the procedure performed during an endoscopy.

[0025] Figure 7 This is a flowchart used to explain the operation of the second embodiment.

[0026] Figure 8 This is an explanatory diagram used to illustrate the operation of the second embodiment. Detailed Implementation

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0028] (First Embodiment)

[0029] Figure 1 This is a block diagram illustrating an endoscope system incorporating an endoscope insertion control device according to a first embodiment of the present invention. The first embodiment is applied to an automatic insertion endoscope that automatically inserts the insertion portion of an endoscope into the large intestine of a patient being examined.

[0030] In this embodiment, a recognizer determines whether the operation selected among various operations for inserting the endoscope insertion section has achieved the expected insertion state, and what kind of insertion state has been achieved. This recognizer uses a model obtained through machine learning. The recognizer uses a model that estimates the class corresponding to the insertion state based on time-series images. Multiple images (time-series images) taken continuously and over time are classified according to their insertion states and used as teaching data. A neural network is used to learn from the teaching data to generate the model.

[0031] Figure 1 This is a diagram illustrating the structure of a key component of an endoscope system, including an endoscope insertion control device. Endoscope system 1, for example, is shown below. Figure 1 As shown, it is configured to include an endoscope 10, a main body device 20, an insertion shape detection device 30, an external force information acquisition device 40, an input device 50, and a display device 60.

[0032] As described below, endoscope 10 is an automated insertion endoscope that automates insertion. Endoscope 10 is configured to have an insertion section 11 for insertion into the patient, an operating section 16 located at the base of the insertion section 11, and a universal cable 17 extending from the operating section 16. Furthermore, endoscope 10 is configured to be detachably connected to the main unit 20 via a scope connector (not shown) located at the end of the universal cable 17. Additionally, light guides (not shown) for transmitting illumination light supplied from the main unit 20 are provided inside the insertion section 11, the operating section 16, and the universal cable 17.

[0033] The insertion portion 11 is configured to have a flexible and elongated shape. Furthermore, the insertion portion 11 is configured to have, from the front end side, a rigid front end portion 12, a freely bendable curved portion 13, and a flexible, elongated tube portion 14. Additionally, within the front end portion 12, the curved portion 13, and the flexible tube portion 14, a plurality of source coils 18 are arranged at predetermined intervals along the length direction of the insertion portion 11. These source coils 18 generate a magnetic field corresponding to a coil drive signal supplied from the main body device 20.

[0034] An illumination window (not shown) is provided at the front end 12, which is used to emit illumination light transmitted through a light guide provided inside the insertion part 11 onto the subject. Furthermore, a camera part 110 is provided at the front end 12. Figure 1 (Not shown in the figure), the camera unit 110 is configured to perform actions corresponding to the camera control signal supplied from the main unit 20, and to capture images of a subject illuminated by illumination light emitted through the illumination window and output a camera signal.

[0035] The bending portion 13 is configured to bend under the control of the bending control unit 242, which will be described later. Furthermore, the bending portion 13 is configured to bend according to the operation of the angle knob (not shown) provided on the operation unit 16.

[0036] The operating unit 16 is configured to be shaped so that it can be operated by a user. Furthermore, the operating unit 16 may be equipped with an angle knob for bending the bending portion 13 along four or eight directions, such as up, down, left, and right, intersecting the length axis of the insertion portion 11. Additionally, the operating unit 16 may be equipped with one or more mirror switches (not shown) capable of providing indications corresponding to user input operations.

[0037] The main unit 20 is configured to have one or more processors 20P and storage media 20M. Furthermore, the main unit 20 is configured to be detachably connected to the endoscope 10 via a universal cable 17. Furthermore, the main unit 20 is configured to be detachably connected to each of the insertion shape detection device 30, the input device 50, and the display device 60. Furthermore, the main unit 20 is configured to perform actions corresponding to instructions from the input device 50. Furthermore, the main unit 20 is configured to generate an endoscope image based on the camera signal output from the endoscope 10, and to perform actions to display the generated endoscope image on the display device 60. Furthermore, the main unit 20 is configured to generate and output various control signals for controlling the operation of the endoscope 10. Furthermore, the main unit 20 is configured to function as an endoscope control device, using insertion shape information (described later) output from the insertion shape detection device 30 to control the insertion operation of the insertion section 11. In addition, the main device 20 can also perform the operation of generating an insertion shape image corresponding to the insertion shape information output from the insertion shape detection device 30 and displaying the generated insertion shape image on the display device 60.

[0038] The insertion shape detection device 30 is configured to detect the magnetic fields generated by the source coils 18 provided in the insertion portion 11, and obtain the positions of each of the multiple source coils 18 based on the strength of the detected magnetic fields. Furthermore, the insertion shape detection device 30 is configured to generate insertion shape information representing the positions of the multiple source coils 18 obtained as described above, and output this information to the main body device 20 and the external force information acquisition device 40. In other words, the insertion shape detection device 30 is configured to detect the insertion shape of the insertion portion inserted into the test body to obtain insertion shape information, and output this obtained insertion shape information to the main body device 20 and the external force information acquisition device 40.

[0039] The external force information acquisition device 40 stores, for example, data on the curvature (or radius of curvature) and bending angle of a predetermined plurality of positions of the insertion portion 11 in a state where no external force is applied, and data on the curvature (or radius of curvature) and bending angle of the predetermined plurality of positions obtained when a predetermined external force is applied to any position of the insertion portion 11 from all anticipated directions. Furthermore, the external force information acquisition device 40 is configured to, for example, determine the position of each of the plurality of source coils 18 provided on the insertion portion 11 based on the insertion shape information output from the insertion shape detection device 30, and obtain the magnitude and direction of the external force at each position of the plurality of source coils 18 by referring to various pre-stored data based on the curvature (or radius of curvature) and bending angle at each position of the plurality of source coils 18. Furthermore, the external force information acquisition device 40 is configured to generate external force information representing the magnitude and direction of the external force at each position of the plurality of source coils 18 obtained as described above, and output it to the main unit 20.

[0040] Furthermore, in this embodiment, as a method for the external force information acquisition device 40 to calculate the external force at the respective positions of the plurality of source coils 18 provided in the insertion part 11, the method disclosed in Japanese Patent No. 5851204 or the method disclosed in Japanese Patent No. 5897092 may be used. In addition, in this embodiment, for example, when electronic components such as strain sensors, pressure sensors, accelerometers, gyroscopes, and wireless components are provided in the insertion part 11, the external force information acquisition device 40 is configured to calculate the external force at the respective positions of the plurality of source coils 18 based on signals output from these electronic components.

[0041] The input device 50 is configured to have one or more user-operated input interfaces, such as a mouse, keyboard, and touch panel. Furthermore, the input device 50 is configured to output instructions corresponding to the user's operation to the main unit 20.

[0042] The display device 60 is configured to include, for example, a liquid crystal monitor. Furthermore, the display device 60 is configured to display endoscopic images, etc., output from the main unit 20 on a screen.

[0043] Figure 2A This is a block diagram used to illustrate the specific structure of the endoscope system 1. (Refer to...) Figure 2A An example of the specific structure of the endoscope 10 and the main device 20 will be described.

[0044] The endoscope 10 is configured to have multiple source coils 18, a camera unit 110, an advance and retraction mechanism 141, a bending mechanism 142, an AWS (Air feeding, Water feeding, and Suction) mechanism 143, and a rotation mechanism 144.

[0045] The camera unit 110 is configured, for example, to have an observation window for receiving reflected light from the illuminated subject, and an image sensor such as a color CCD that captures the reflected light and outputs an image signal.

[0046] Figure 2BThis figure shows an example of the specific structure of the advance / retract mechanism 141. The advance / retract mechanism 141 is configured, for example, to have: a pair of rollers 141a and 141b arranged opposite each other across the insertion portion 11; and a motor (not shown) that supplies rotational driving force for rotating the pair of rollers 141a and 141b. Furthermore, the advance / retract mechanism 141 is configured, for example, to drive the motor according to an advance / retract control signal output from the advance / retract control unit 241 of the main body device 20, and to rotate the pair of rollers 141a and 141b about axes C1 and C2 according to the rotational driving force supplied from the motor. Thus, it selectively performs either an action to advance the insertion portion 11 in the direction of arrow A1 or an action to retract the insertion portion 11 in the direction of arrow A2.

[0047] The bending mechanism 142 is configured, for example, to have a plurality of bending members provided on the bending section 13, a plurality of lines connected to the plurality of bending members, and a motor supplied with a rotational driving force for pulling the plurality of lines. Furthermore, the bending mechanism 142 is configured, for example, to drive the motor according to a bending control signal output from the main body device 20, and to change the traction amount of each of the plurality of lines according to the rotational driving force supplied from the motor, thereby enabling the bending section 13 to bend in four directions: up, down, left, and right.

[0048] The AWS mechanism 143 is configured, for example, to have two conduits (not shown) disposed inside the endoscope 10 (insertion section 11, operation section 16, and universal cable 17), namely an air / water supply conduit and a suction conduit, and a solenoid valve for opening one of the two conduits and closing the other. Furthermore, the AWS mechanism 143 is configured, for example, to allow fluid containing at least one of water and air supplied from the main body 20 to flow through the air / water supply conduit and be discharged from the outlet formed at the front end 12 when the solenoid valve is opened according to an AWS control signal output from the main body 20. Furthermore, the AWS mechanism 143 is configured, for example, to allow the suction force generated in the main body 20 to act on the suction conduit when the solenoid valve is opened according to an AWS control signal output from the main body 20, and to allow the suction force to draw out objects present near the suction port formed at the front end 12.

[0049] The rotating mechanism 144 is configured, for example, to have a gripping member that grips the insertion portion 11 at the base end of the flexible tube portion 14 and a motor that supplies a rotational driving force to rotate the gripping member. Furthermore, the rotating mechanism 144 is configured, for example, to drive the motor according to a rotational control signal output from the main body device 20, and to rotate the gripping member according to the rotational driving force supplied from the motor, thereby enabling the insertion portion 11 to rotate about the insertion axis (length axis).

[0050] like Figure 2A As shown, the main device 20 is configured to include a light source unit 210, an image processing unit 220, a coil drive signal generation unit 230, an insertion action control unit 240, a display control unit 250, and a system control unit 260.

[0051] The light source unit 210 is configured, for example, to have one or more LEDs or one or more lamps as a light source. Furthermore, the light source unit 210 is configured to generate illumination light for illuminating the specimen into which the insertion unit 11 is inserted, and to supply this illumination light to the endoscope 10. Furthermore, the light source device 210 is configured to vary the amount of illumination light according to a system control signal supplied from the system control unit 260.

[0052] The image processing unit 220, which together with the camera unit 110 constitutes the image acquisition unit, is configured to include an image processing circuit, for example. Furthermore, the image processing unit 220 is configured to generate an endoscope image by performing predetermined processing on the camera signal output from the endoscope 10, and output the generated endoscope image to the display control unit 250 and the system control unit 260.

[0053] The coil drive signal generation unit 230 is configured, for example, to have a drive circuit. Furthermore, the coil drive signal generation unit 230 is configured to generate and output a coil drive signal for driving the source coil 18 based on a system control signal supplied from the system control unit 260.

[0054] The insertion action control unit 240 is configured to include an advance / retreat control unit 241, a bending control unit 242, an AWS control unit 243, and a rotation control unit 244. The insertion action control unit 240 is configured to perform actions to control the functions implemented by the endoscope 10 based on insertion control signals supplied from the system control unit 260. Specifically, the insertion action control unit 240 is configured to perform at least one of the following functional actions: the advance / retreat function implemented by the advance / retreat mechanism 141, the bending function implemented by the bending mechanism 142, the AWS function implemented by the AWS mechanism 143, and the rotation function implemented by the rotation mechanism 144.

[0055] The forward / backward control unit 241 is configured to generate and output forward / backward control signals for controlling the operation of the forward / backward mechanism 141 based on the insertion control signal supplied from the system control unit 260. Specifically, the forward / backward control unit 241 is configured to generate and output, for example, forward / backward control signals for controlling the rotational state of the motor provided on the forward / backward mechanism 141 based on the insertion control signal supplied from the system control unit 260.

[0056] The bending control unit 242 is configured to generate and output a bending control signal for controlling the operation of the bending mechanism 142 based on the insertion control signal supplied from the system control unit 260. Specifically, the bending control unit 242 is configured to generate, for example, a bending control signal for controlling the rotational state of the motor provided in the bending mechanism 142 based on the insertion control signal supplied from the system control unit 260.

[0057] The AWS control unit 243 is configured to control a pump (not shown) or similar device based on an insertion control signal supplied from the system control unit 260, thereby enabling selective operation of either an action to supply a fluid containing at least one of water and air to the endoscope 10 or an action to generate a suction force to aspirate an object present near the suction port of the front end 12. Furthermore, the AWS control unit 243 is configured to generate and output AWS control signals for controlling the operation of the AWS mechanism 143. Specifically, the AWS control unit 243 is configured to generate and output, for example, AWS control signals for controlling the operating state of a solenoid valve provided in the AWS mechanism 143 based on the insertion control signal supplied from the system control unit 260.

[0058] The rotation control unit 244 is configured to generate and output a rotation control signal for controlling the operation of the rotation mechanism 144 based on the insertion control signal supplied from the system control unit 260. Specifically, the rotation control unit 244 is configured to generate, for example, a rotation control signal for controlling the rotation state of the motor provided on the rotation mechanism 144 based on the insertion control signal supplied from the system control unit 260.

[0059] That is, the insertion action control unit 240 is configured to generate and output control signals corresponding to the operations performed by the endoscope 10 based on the insertion control signals supplied from the system control unit 260. These control signals correspond to the following operations: a push operation equivalent to an operation to advance the insertion part 11; a pull operation equivalent to an operation to retract the insertion part 11; an angle operation equivalent to an operation to bend the bending part 13 so that the direction of the front end 12 is oriented in a direction that intersects the insertion axis (length axis) of the insertion part 11 (for example, one of eight directions consisting of four directions: up, down, left, right, and four directions in the middle of each direction); a torsion operation equivalent to an operation to rotate the insertion part 11 around the insertion axis (length axis); a gas delivery operation for ejecting gas in front of the front end 12; a water delivery operation for ejecting liquid in front of the front end 12; and a suction operation for aspirating tissue in front of the front end 12.

[0060] The display control unit 250 is configured to perform processing for generating a display image including the endoscopic image output from the image processing unit 220, and to perform processing for displaying the generated display image on the display device 60. Furthermore, the display control unit 250 may also perform processing for displaying an insertion shape image output from the system control unit 260 on the display device 60.

[0061] like Figure 2A As shown, the insertion shape detection device 30 is configured to have a receiving antenna 310 and an insertion shape information acquisition unit 320.

[0062] The receiving antenna 310 is configured, for example, to have multiple coils for three-dimensionally detecting magnetic fields emitted from the multiple source coils 18 respectively. Furthermore, the receiving antenna 310 is configured to detect the magnetic fields emitted from the multiple source coils 18 respectively, generate a magnetic field detection signal corresponding to the strength of the detected magnetic field, and output it to the insertion shape information acquisition unit 320.

[0063] The insertion shape information acquisition unit 320 is configured to acquire the positions of each of the plurality of source coils 18 based on the magnetic field detection signal output from the receiving antenna 310. Furthermore, the insertion shape information acquisition unit 320 generates insertion shape information representing the positions of each of the plurality of source coils 18 acquired as described above, and outputs it to the system control unit 260.

[0064] Specifically, the insertion shape information acquisition unit 320 acquires, for example, multiple three-dimensional coordinate values ​​in a spatial coordinate system as the positions of the multiple source coils 18, which is virtually set with the origin or reference point being a predetermined position (such as the anus) of the subject into which the insertion unit 11 is inserted. Furthermore, the insertion shape information acquisition unit 320 generates insertion shape information containing the multiple three-dimensional coordinate values ​​acquired as described above, and outputs the insertion shape information to the system control unit 260. Then, in this case, for example, the system control unit 260 performs the following processes: processing for acquiring multiple two-dimensional coordinate values ​​corresponding to each of the multiple three-dimensional coordinate values ​​included in the insertion shape information output from the insertion shape information acquisition unit 320; processing for interpolating the acquired multiple two-dimensional coordinate values; and processing for generating an insertion shape image corresponding to the interpolated multiple two-dimensional coordinate values.

[0065] In this embodiment, at least a portion of the insertion shape detection device 30 may be configured as an electronic circuit, or as a circuit module in an integrated circuit such as an FPGA. Furthermore, in this embodiment, for example, the insertion shape detection device 30 may be configured to have one or more processors (CPUs, etc.).

[0066] The system control unit 260 is configured to include an insertion control unit 261, a control content recording unit 262, an insertion status classification unit 263, and an operation database (DB) unit 264. The system control unit 260 is configured to generate and output system control signals for performing actions corresponding to instructions from the operation unit 16 and the input device 50.

[0067] As an operation selection unit, the insertion control unit 261 generates control signals (hereinafter referred to as basic control signals) for controlling various operations (hereinafter referred to as basic insertion operations) for inserting the insertion unit 11 into the desired lumen based on the automatic insertion control based on the output of the insertion shape detection device 30, the external force information acquisition device 40 and the image processing unit 220.

[0068] The basic insertion operation based on the insertion control unit 261 is an operation performed by selecting from at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape detection device 30. For example, it is an operation performed by selecting from various basic insertion operations implemented by the endoscope 10, such as the aforementioned forward operation (push operation), backward operation (pull operation), stop operation, angle (bending) operation, rotation operation (torsion operation), air delivery operation, water delivery operation, and suction operation. The basic control signals from the insertion control unit 261 also include control information related to the amount of movement, movement speed, rotation angle, rotation direction, and operating force during the basic insertion operation.

[0069] The control content recording unit 262 is composed of a specified recording medium. The control content recording unit 262 records the control content of the insertion control unit 261, i.e., the basic insertion operation, one after another.

[0070] Furthermore, in endoscopic insertion procedures without automatic insertion capabilities, skilled surgeons pay close attention to the response of the proximal endoscope to manipulations at the tip. For example, if the tip advances smoothly with no resistance relative to the right hand, advancing the same amount as intended—that is, the surgeon's intended amount—the surgeon recognizes that the endoscope is in a good insertion position. In this case, it is easier for the surgeon to select the intended insertion method in subsequent procedures.

[0071] Such a good insertion state is sometimes also characterized by "a sense of freedom with the endoscope" and "a one-to-one relationship of movement between the proximal and distal ends." Hereafter, this good insertion state will be recorded as "endoscope in a free state." On the other hand, if there is insufficient advance of the distal end relative to the amount of insertion, or if the distal end remains almost motionless, or if movements different from advance, such as retraction or rotation, occur, the physician judges this as an obstacle to advancement, such as friction or flexion with the mucosa. In this case, a skilled physician will choose rotational manipulation or jiggling manipulation (repeated subtle back-and-forth movements) to eliminate / reduce these obstacles. These judgments are mainly based on the image or the force / feel transmitted from the endoscope to the right hand, with the former being particularly important.

[0072] For example, when an α-ring or similar ring is formed, even if the right-hand insertion operation can be performed with relatively little resistance, the force used for forward movement will be absorbed by the ring. Therefore, the camera unit 110 obtains an image (moving image) showing that the front end 12 is almost stationary, or a moving image showing movements other than forward movement, such as a small amount of rotation. A skilled physician can determine the actual insertion state relative to the operation based on such moving images.

[0073] Therefore, in this embodiment, an insertion state classification unit 263 with a recognizer is employed. This recognizer has been trained to perform the same judgment process as a skilled physician. The insertion state classification unit 263 classifies (also called infers) the insertion state of the insertion unit 11 based on a series of images captured by the camera unit 110 and provided by the image processing unit 220, obtaining a classification result. Furthermore, the series of images can be multiple images obtained in a time sequence, such as moving images at a specified frame rate or continuously captured still images (time-series images). If multiple images are obtained in a time sequence, even multiple images that do not change at all are considered time-series images.

[0074] In this embodiment, the insertion state classification unit 263 can identify whether an image change corresponding to the intended insertion state has been obtained relative to the basic insertion operation performed. Based on the identification result, an operation considered to be effective for smooth insertion is selected as the next operation (hereinafter referred to as auxiliary insertion operation).

[0075] In the operation DB unit 264, information on auxiliary insertion operations deemed effective for smooth insertions is registered based on the relationship between the basic insertion operation and the insertion state obtained from the classification results. The insertion control unit 261 is configured to obtain auxiliary insertion operations by referring to the operation DB unit 264 based on the basic insertion operation recorded in the control content recording unit 262 and the classification results of the insertion states output from the insertion state classification unit 263, and outputs a control signal (hereinafter referred to as an auxiliary control signal) for implementing the obtained auxiliary insertion operation.

[0076] Here, a specific example of the structure of the insertion state classification unit 263 in this embodiment will be described.

[0077] The insertion state classification unit 263 is configured, for example, to process the data using a recognizer to obtain a classification result that classifies the insertion states, which are classified according to the time-series images output from the image processing unit 220, into one of several categories. This recognizer is generated by learning the 3D combination coefficients (weights) of a 3D-CNN (Convolutional Neural Network), which is equivalent to a multi-layer neural network containing an input layer, one or more convolutional layers, and an output layer, using learning methods such as deep learning. Here, 3D-CNN is a method that extends the CNN (Convolutional Neural Network), widely used for the recognition or classification of ordinary (two-dimensional) images, to also be applicable to three-dimensional images (voxel images) or time-series images.

[0078] When generating the recognizer, machine learning is performed using teaching data, which includes, for example, a series of multiple camera images (time-series images) identical to those generated by the image processing unit 220; and labels indicating the classification result of the insertion state determined based on the time-series images as one of a predetermined plurality of categories. Furthermore, the aforementioned predetermined plurality of categories are, for example, set to be characteristic insertion states or categories of success of operation that help determine whether the insertion operation of the insertion unit 11 is successful when performed manually or automatically, during the period from the start of insertion of the insertion unit 11 into the subject to the end of insertion of the insertion unit 11 into the subject.

[0079] For example, the insertion state is not limited to "forward," but also includes categories such as "good forward" or "stopped," which encompass whether the operation was successful. When generating teaching data, for example, a task is also performed to assign a label to the time series image, where the label corresponds to the judgment result of a skilled person visually determining which of a number of categories the insertion state of the insertion unit 11 based on the time series image belongs to.

[0080] Figures 3A to 3F It is an explanatory diagram used to illustrate time series images and the labels attached to them. Figures 3A to 3F These are time-series images captured by the camera unit 110 during the insertion operation of the insertion unit 11. Times t1, ..., t5 indicate the time at which each image constituting the time-series image was acquired. Regarding the time-series image, it is possible to acquire it from multiple images obtained during the period from the start to the end of a certain operation, for example, at fixed time intervals. Figures 3A to 3F An example of such a time-series image is shown. Furthermore, when generating the time-series image, for example, in the case of using a line or air pressure in the mechanism during angle operation, the movement of the insertion part may have a slight delay relative to the operation. Therefore, a margin of about 1 to 2 seconds may be added to the end time of the operation. In this embodiment, in the following description, a set of time-series images is assumed to consist of 5 images.

[0081] Figure 3A The images shown are time-series images of the forward movement, in which the tip of the insertion section 11 moves (advances) towards the depth of the lumen. In the images sampled on the time axis, the following changes occur: as the tip 12 advances, dark areas or folds in the lumen on the distal side gradually approach or disappear from the image field of view.

[0082] In this embodiment, it is assumed that the creator of the teaching data is a skilled physician who performs colonoscopy or someone with sufficient knowledge of endoscopic procedures as a skilled physician.

[0083] For example, the creator of the teaching data might label a time-series image as "good progress." "Good progress" could mean, for example, that after advancing the insertion section 11 by 5 cm, the tip of the insertion section 11 has advanced approximately 5 cm. Alternatively, an insertion where the tip of the insertion section 11 advances approximately 2.5 cm after advancing the insertion section 11 by 5 cm could be labeled as "insufficient progress." Furthermore, an insertion where the tip of the insertion section 11 advances less than 1 cm after advancing the insertion section 11 by 5 cm could be labeled as "stopped."

[0084] Furthermore, while the above description quantitatively represents the amount of forward movement relative to the operation, the actual amount of forward movement can also be subjectively evaluated. For example, in a time-series image obtained when the insertion part 11 is advanced by 5 cm for the advance / retraction mechanism 141, if the teaching data creator evaluates it as sufficient forward movement, it can be labeled as "good forward movement"; if it is almost stationary, it can be labeled as "stopped"; and if it is perceived as an intermediate amount of forward movement, it can be labeled as "insufficient forward movement".

[0085] Furthermore, the actions of the endoscope for a particular operation can be categorized into multiple classes based on the magnitude of their changes. Classification can also be performed for retraction or rotation, for example, by setting "right rotation less than 90 degrees" and "right rotation greater than 90 degrees", thereby enabling more accurate classification of the operation.

[0086] Figure 3B The image shows a time-series image of the endoscope in a free state during retraction, with the tip of the insertion section 11 retracting relative to the depth of the lumen. For example, the teaching data creator... Figure 3B The time series images are labeled with tags such as "backward". Figure 3C The image shows a time-series image of the endoscope in a free state during right-hand rotation, with the tip of the insertion section 11 rotated right relative to the depth of the lumen. For example, the teaching data creator... Figure 3C The time-series images are labeled with tags such as "right rotation". Furthermore, Figure 3D The image shows a time-series image of the endoscope in a free state during leftward rotation, with the tip of the insertion section 11 rotated leftward relative to the depth of the lumen. For example, the teaching data creator... Figure 3D The time series images are labeled with tags such as "left rotation".

[0087] In addition, in the above Figures 3B to 3D The description includes a graph representing the time-series images of the endoscope in a free state, but this is unrelated to whether the endoscope is in a free state. Figures 3B to 3D The time series images are labeled with tags such as "backward," "right rotation," and "left rotation." Figure 3A The text describes labeling such as "good progress," but this assumes that the creator of the teaching data knew what operations were performed to obtain the time series image.

[0088] also, Figure 3E The image shown is a time-series image labeled with terms like "parallel movement to the right" by the creator of the teaching data. Figure 3FThe image shown is a time-series image labeled with tags such as "angle operation to the right" by the creator of the teaching data. Additionally, in Figure 3F In the example, only the angle operation that bends the curved portion 13 to the right as a defined direction is shown. However, for example, time series images with angle operations in eight directions can also be used to label the eight directions respectively.

[0089] Furthermore, as a result of the operation, the tip of the insertion part 11 sometimes becomes an insertion state accompanied by deformation of the intestinal side. By using time-series images of this situation, it is also possible to label it with a tag such as "intestinal side deformation". In addition, the types of intestinal side deformation are not limited to one.

[0090] In this way, machine learning is performed using teaching data obtained by labeling each time-series image with a label corresponding to the type of insertion state. The result is a recognizer that classifies the insertion state. During learning, for example, 14 classes are used as recognition objects: good forward movement, insufficient forward movement, backward movement, rotation in two directions, angular manipulation in eight directions, and stopping, indicating a state of near-immobility. Approximately 1000 sets of time-series images are used as teaching data for each class, and the number of learning iterations (called epochs) is set to 100. Based on such a recognizer, for example, multi-dimensional data such as the pixel values ​​of each pixel contained in the time-series image output from the image processing unit 220 can be obtained. This multi-dimensional data is input into the input layer of the neural network, thereby obtaining multiple likelihoods corresponding to each class that can be classified as the type of insertion state of the insertion unit 11 obtained from the time-series image, and output data is output from the output layer of the neural network. Furthermore, based on the processing using this recognizer, for example, it is possible to obtain the type of an insertion state corresponding to the highest likelihood among the multiple likelihoods contained in the output data output from the output layer of the neural network, and the classification result of the insertion state of the insertion unit 11 obtained as a result of the operation.

[0091] That is, the insertion state classification unit 263 is configured to obtain a classification result representing the type of insertion state of the insertion unit 11 by performing processing using a recognizer. The recognizer is generated by machine learning using teaching data, which includes: a time series image from the image processing unit 220; and a label representing a classification result that classifies the insertion state of the insertion unit 11 obtained from the time series image into one of a number of specified types.

[0092] As described above, the insertion control unit 261 reads the insertion state obtained from the classification result from the insertion state classification unit 263, reads the basic insertion operation that generated the insertion state from the control content recording unit 262, and refers to the operation DB unit 264 to obtain the auxiliary insertion operation. In addition, normally, when the endoscope is in a free state and the insertion state expected for the basic insertion operation is obtained, the auxiliary insertion operation is not performed, and the subsequent basic insertion operation is performed.

[0093] Furthermore, in this embodiment, at least a portion of the functions of the main device 20 can be implemented by the processor 20P. Additionally, in this embodiment, at least a portion of the main device 20 can be configured as various electronic circuits, or as a circuit module in an integrated circuit such as an FPGA (Field Programmable Gate Array). Furthermore, the structure of this embodiment can be appropriately modified, such that, for example, a computer reads a program from a storage medium 20M, such as a memory, to execute at least a portion of the functions of the main device 20, and performs actions corresponding to the read program.

[0094] Next, refer to Figure 4 The operation of this implementation method will be explained. Figure 4 This is a flowchart used to illustrate the actions of the implementation method.

[0095] System control unit 260 Figure 4 In step S1, a basic insertion operation is selected. The system control unit 260 records the selected basic insertion operation in the control content recording unit 262 (step S2). The insertion control unit 261 outputs a basic control signal for executing the selected basic insertion operation to the insertion action control unit 240. As a result, the insertion action control unit 240 controls each mechanism of the endoscope 10 to execute the basic insertion operation (step S3).

[0096] During insertion, the imaging unit 110 captures an image of the subject (step S4) and outputs the imaging signal to the image processing unit 220. The image processing unit 220 provides the imaging image (endoscopic image) based on the imaging signal to the display control unit 250 and displays it on the display device 60, and also provides it to the insertion state classification unit 263. The image processing unit 220 sequentially provides the images captured by the imaging unit 110 to the insertion state classification unit 263, and the insertion state classification unit 263 acquires time-series images corresponding to the insertion state of the front end of the imaging unit 110 during the basic insertion operation.

[0097] Furthermore, in step S6, the system control unit 260 determines whether the predetermined number of images has been reached. If the predetermined number of images has not been reached, the process returns to step S3; if the predetermined number of images has been reached, the process proceeds to step S7. Thus, in step S7, the insertion state classification unit 263 uses a time-series image composed of the predetermined number of captured images to classify the insertion state. This classification indicates which insertion state corresponds to the changes in the image generated by the operation related to the insertion of the endoscope 10.

[0098] For example, based on multiple time-series images taken for the operation of advancing the front end 12, it is possible to classify the movement of the front end 12 for the operation, such as "the front end has advanced well", "it has not advanced sufficiently", "it has not advanced", "it has retreated (the lumen has moved away)", "it has rotated".

[0099] The insertion control unit 261 reads the contents of the basic insertion operation obtained from the classification result from the control content recording unit 262. Based on the insertion state obtained through classification and the read basic insertion operation, it refers to the operation DB unit 264 to execute the next auxiliary insertion operation. In this case, the insertion control unit 261 determines whether the read basic insertion operation corresponds to the insertion state obtained through classification, that is, whether the expected insertion state is obtained (step S8). If the expected insertion state is obtained, the insertion control unit 261 returns the process to step S1 and selects the next basic insertion operation. If the expected insertion state is not obtained, the auxiliary insertion operation is determined in step S9.

[0100] For example, if the determination result is other than "the front end has advanced well" for the operation of advancing the front end 12, as an auxiliary insertion operation, an operation such as shaking or rotating is selected and performed to improve the situation, so that the front end 12 of the endoscope 10 can advance smoothly (step S10).

[0101] For example, for a determination of "not advancing," options such as deflection removal or jitter can be selected to allow advancement. Furthermore, for a determination of "slightly advancing," the next operation is typically selected, or an operation that advances while applying a rotational force can be selected. Additionally, for a determination of "advanced," the usual next operation can be selected, or the advancement amount can be increased, or the advancement conditions can be relaxed, such as allowing the anterior end 12 to advance even if the lumen deviates slightly from the center of the field of view.

[0102] The system control unit 260 determines whether the front end 12 has reached the target position (step S11), and repeats the processing of steps S1 to S10 until the target position is reached.

[0103] Thus, in this embodiment, a classifier that classifies time-series images according to the insertion state of the endoscope line relative to the lumen is used. This allows for the determination of whether the desired insertion state has been achieved and, if so, what that state has been achieved, among various operations for inserting the endoscope insert. The determination result is used to decide the next insertion operation, thereby understanding how the endoscope insert or its tip behaves relative to the desired operation. This allows for the selection of an appropriate operation corresponding to the situation, preventing deterioration of the insertion state or causing pain to the patient, and ensuring more reliable and smooth insertion of the endoscope insert into the lumen. Furthermore, this classifier achieves high-precision classification results by using a model in which the classification of time-series images according to the insertion state of the endoscope line relative to the lumen is used as teaching data, and the model is generated by learning from the teaching data using a neural network.

[0104] (Second Implementation)

[0105] Figure 5 This is a block diagram illustrating the second embodiment of the present invention. Furthermore, Figure 6 This is an illustrative diagram showing the procedure performed during an endoscopy. Figure 5 and Figure 6 China and Figure 2A The same structure is labeled with the same number and the explanation is omitted.

[0106] In the first embodiment, an example of application to an automated insertion endoscope is described. This embodiment applies to a general endoscope in which a physician inserts the insertion part into the patient.

[0107] exist Figure 5 and Figure 6 In this system, the endoscope system 400 includes an endoscope 410 and a main unit 420. For example... Figure 5 As shown, endoscope 410 from Figure 2A The endoscope 10 omits the advance / retract mechanism 141, the AWS mechanism 143, and the rotation mechanism 144. Furthermore, the main unit 420... Figure 2A The insertion action control unit 240 is omitted from the main body device 20. Furthermore, the main body device 420 replaces... Figure 2A Instead of the system control unit 260, a system control unit 460 is used. The system control unit 460 omits the insertion control unit 261, the control content recording unit 262 and the operation DB unit 264, and adds an insertion shape image generation unit 461 and a bending control unit 462.

[0108] The endoscope 410 includes: an elongated and flexible insertion part 410b that is inserted into the body cavity of the subject P, which is the subject to be examined; an operation part 410a connected to the base end of the insertion part 410b and provided with various manipulators; and a cable 410c for connecting the operation part 410a and the main body device 420.

[0109] Figure 6 This shows the insertion part 410b inserted into the large intestine through the anus of the subject P, who is lying on the examination bed 6. Figure 6 The image shows a doctor O holding the operating part 410a and the insertion part 410b of the endoscope 410, wherein the endoscope 410 is connected to the main body device 420 on the medical cart 4 via a cable 410c.

[0110] The structure of the insertion part 410b and Figure 1 Similarly, the insertion part 11 is provided with a camera part 110 at the front end and a plurality of source coils 18 for insertion state detection. In addition, a bending part is provided at the front end of the insertion part 410b, which is configured to be bent and driven by a bending mechanism 142.

[0111] A bending knob 410d constituting the input device 50 is provided in the operation unit 410a. By operating the bending knob 410d, an operation signal is supplied to the system control unit 460. The bending control unit 462 of the system control unit 460 is configured to generate and output a bending control signal for controlling the operation of the bending mechanism 142 based on the operation of the bending knob 410d. By using this bending control signal, for example, the rotation state of the motor provided in the bending mechanism 142 is controlled, and a bending action corresponding to the operation of the bending knob 410d is performed. In this way, the doctor can operate the bending knob 410d to bend the bending part and push the insertion part 410b into the body cavity.

[0112] The insertion state of the insertion part 410b is observed using a known insertion shape detection device 30. The insertion shape detection device 30, consisting of a receiving antenna 310 and an insertion shape information acquisition unit 320, is disposed near the bed 6. The insertion shape detection device 30 is connected to the main unit 420 via a cable 7a. The insertion shape detection device 30 generates insertion shape information of the insertion part 410b and outputs it to the system control unit 460.

[0113] On the other hand, the insertion operation performed by the doctor O can be detected by the operation detection sensor 70. The operation detection sensor 70 detects, for example, the forward and backward direction, movement amount, rotation direction, and rotation amount of the insertion part 410b near the anus of the patient P. The operation detection sensor 70 outputs the detection results to the doctor's operation detection device 71.

[0114] The doctor's operation detection device 71 determines the start timing of various doctor's operations based on the detection results of the operation detection sensor 70. Furthermore, the doctor's operation detection device 71 determines the type of doctor's operation based on the detection results of the operation detection sensor 70 within a predetermined period from the start timing of the operation. This predetermined period is set according to the time-series image used to obtain the model for the insertion state classification unit 263 during learning. The doctor's operation detection device 71 outputs information about the type of doctor's operation and its start timing to the system control unit 460.

[0115] The insertion shape image generation unit 461 of the system control unit 460 performs, for example, the following processes: acquiring multiple two-dimensional coordinate values ​​corresponding to multiple three-dimensional coordinate values ​​contained in the insertion shape information output from the insertion shape information acquisition unit 320; interpolating the acquired multiple two-dimensional coordinate values; and generating an insertion shape image corresponding to the interpolated multiple two-dimensional coordinate values. The insertion shape image generation unit 461 outputs the generated insertion shape image to the display control unit 250. Thus, the display control unit 250 can display the insertion shape image on the display screen of the display device 60.

[0116] In this embodiment, the system control unit 460 includes, for example, an insertion state classification unit 263 that uses a model obtained through the same learning process as in the first embodiment. The system control unit 460 outputs information indicating the classification result of the insertion state type obtained through the insertion state classification unit 263 to the display control unit 250 based on the doctor's operation detected by the doctor's operation detection device 71. Consequently, the display control unit 250 displays a display indicating the type of insertion state on the display screen of the display device 60.

[0117] Next, refer to Figure 7 and Figure 8 The operation of this implementation method will be explained. Figure 7 This is a flowchart used to explain the operation of the second embodiment. Figure 7 China and Figure 4 The same steps are labeled with the same numbers and explanations are omitted. Furthermore, Figure 8 This is an explanatory diagram used to illustrate the operation of the second embodiment.

[0118] In this embodiment, since an endoscope 410 that cannot be automatically inserted is used, therefore, Figure 7 No further action will be taken. Figure 4 Steps S1 to S3. In Figure 7In step S7, the insertion state classification unit 263 uses a time-series image composed of a predetermined number of photographic images to classify the insertion state. This classification indicates which insertion state corresponds to the changes in the image resulting from the insertion operation of the endoscope 10 performed by the doctor. The insertion state classification unit 263 provides the classification result to the display control unit 250, which displays the classification result on the screen of the display device 60 (step S22).

[0119] Figure 8 An example of this situation is shown where the insertion shape image 61 of the insertion part 11 generated by the insertion shape image generation unit 461 is displayed on the display screen 60a of the display device 60, and a display 62 showing the type of insertion state of the classification result is also shown. Figure 8 In the example, the display 62 indicating the type of insertion state is "almost not advanced". A doctor can easily identify, for example, a situation where an advancing operation has been performed but the tip of the insertion section 11 has barely advanced by using display 62.

[0120] Thus, in this embodiment, by employing a recognizer, it is possible to determine whether the desired insertion state has been achieved, and what kind of insertion state has been achieved, for an operation selected from various procedures for inserting the endoscope insertion site. This recognizer uses a model that classifies time-series images according to the insertion state of the endoscope line relative to the lumen, and uses this as teaching data to generate the model through a neural network. By displaying the determination result, effective support can be provided for selecting subsequent insertion procedures that can improve the insertion for the physician and prevent pain for the patient.

[0121] Furthermore, the insertion action control unit 240, system control unit 260, system control unit 460, etc., in the above embodiments can be configured by combining dedicated circuits or multiple general-purpose circuits. They can also be configured by combining microprocessors and CPUs, or sequencers, which operate according to pre-programmed software, as needed. Additionally, it can be designed so that an external device undertakes part or all of the control; in this case, a wired or wireless communication circuit is provided. Embodiments in which external devices such as servers or personal computers perform the characteristic or supplementary processing of each embodiment are also envisioned. That is, this application also covers situations where the features of the present invention are achieved through the cooperation of multiple devices. In such cases, communication uses Bluetooth (registered trademark), Wi-Fi (registered trademark), telephone lines, etc. Furthermore, communication can also be performed via USB, etc. The dedicated circuit, general-purpose circuit, or control unit can also be integrated into an ASIC.

[0122] Furthermore, in the technology described herein, most of the controls and functions illustrated in the flowchart can be programmed, read and executed by a computer to perform the aforementioned controls and functions. This program can be recorded, in whole or in part, as a computer program product on removable media such as floppy disks or CD-ROMs (non-volatile memory), or storage media such as hard disks or volatile memory, and can be distributed or provided at the time of product shipment or via removable media or communication lines. Users can download the program via a communication network and install it on a computer, or install it from a recording medium on a computer, thereby easily implementing the endoscope insertion control device of this embodiment.

[0123] This invention is not directly limited to the embodiments described above. During implementation, structural elements can be modified and modified to be more specific without departing from its main intent. Furthermore, various inventions can be formed by appropriately combining the multiple structural elements disclosed in the above embodiments. For example, some structural elements may be deleted from all the constituent elements shown in the embodiments. Moreover, structural elements from different embodiments may be appropriately combined.

[0124] Furthermore, in the above description, the case of using 3D-CNN, as one of the deep learning methods, as a method for detecting the state of the endoscope insertion part based on time series images has been described. However, various methods that can achieve the same effect in the recognition of time series images, such as known methods such as dynamic image recognition using optical flow, can also be used.

[0125] Furthermore, the endoscopic insertion control device and method of the present invention can also be used to examine organs other than the large intestine, such as the small intestine or trachea, and can also be used in industrial endoscopes for examining piping, etc.

Claims

1. An endoscope insertion control device, characterized in that, The endoscope insertion control device has a processor. The processor performs the following processing: Choose the endoscopic procedure. Record the operation. Generate control signals for performing the selected operation. The insertion status is classified based on the images captured by the endoscope. Determine whether the insertion state obtained from the classification corresponds to the recorded operation. When the insertion state obtained from the classification corresponds to the recorded operation, the processor selects the operation of the endoscope.

2. The endoscope insertion control device according to claim 1, characterized in that, If the classification result does not correspond to the recorded operation, the processor outputs an auxiliary insertion operation.

3. The endoscope insertion control device according to claim 1, characterized in that, The processor acquires time-series images, and, The processor determines whether the number of time-series images has reached a predetermined number. If the predetermined number has been reached, the processor performs the classification.

4. The endoscope insertion control device according to claim 1, characterized in that, The processor selects an operation that changes the condition of the endoscope as the auxiliary insertion operation.

5. The endoscope insertion control device according to claim 1, characterized in that, When the processor classifies an insertion state related to forward movement as the insertion state, it selects the forward movement operation even if the lumen deviates from the center of the field of view.

6. The endoscope insertion control device according to claim 1, characterized in that, The processor determines whether the tip of the endoscope has reached the target area and repeats the process until the target area is reached.

7. The endoscope insertion control device according to claim 1, characterized in that, The processor performs the classification based on a machine learning-equipped recognizer.

8. The endoscope insertion control device according to claim 7, characterized in that, The recognizer uses the following model: it is obtained by classifying multiple images acquired by the endoscope in a time-series manner according to the type of insertion state and using them as teaching data, and then using a neural network to learn from the teaching data. The insertion state shows the correspondence between the operation related to the insertion of the endoscope insertion part and the insertion state of the endoscope insertion part.

9. The endoscope insertion control device according to claim 1, characterized in that, The processor is categorized into a first category and a second category, where the first category corresponds to the case where the endoscope has advanced sufficiently, and the second category corresponds to the case where the endoscope has not advanced sufficiently. The processor modifies the operation to be selected based on whether the classification result is the first category or the second category.

10. A control method for a control device, characterized in that, Insert the control unit to select the endoscope operation. The control content recording unit records the operation selected by the insertion control unit. The insertion control unit generates a control signal for performing the selected operation. The insertion shape classification unit classifies the images captured by the endoscope. The insertion control unit determines whether the classification result corresponds to the operation recorded by the control content recording unit. If they correspond, the insertion control unit selects the operation of the endoscope.

11. The control method of the control device according to claim 10, characterized in that, If the classification result does not correspond to the recorded operation, the insertion control unit outputs an auxiliary insertion operation.

12. The control method of the control device according to claim 10, characterized in that, The inserted shape classification unit acquires time-series images, and, The insertion shape classification unit determines whether the number of time series images has reached a predetermined number. If the predetermined number of images has been reached, the insertion shape classification unit performs the classification.

13. The control method of the control device according to claim 10, characterized in that, The insertion control unit selects an operation that changes the state of the endoscope as the auxiliary insertion operation.

14. The control method of the control device according to claim 10, characterized in that, When the insertion state related to forward movement is classified as the insertion state, the insertion control unit selects forward movement even if the lumen deviates from the center of the field of view.

15. The control method of the control device according to claim 10, characterized in that, The insertion control unit determines whether the tip of the endoscope has reached the target area and repeats the process until the target area is reached.

16. A computer program product, characterized in that, The computer program product causes the computer to perform the following methods: Choose the endoscopic procedure. Record the operation. Generate control signals for performing the selected operation. The insertion status is classified based on the images captured by the endoscope. Determine whether the insertion state obtained from the classification corresponds to the recorded operation. When the insertion state obtained from the classification corresponds to the recorded operation, the processor selects the operation of the endoscope.

17. The computer program product according to claim 16, characterized in that, A method for performing an output-assisted insertion operation when the classification result does not correspond to the recorded operation.

18. The computer program product according to claim 16, characterized in that, Execute the following method: Obtain time series images, and, Determine whether the number of time-series images has reached a specified number. If the specified number of images has reached the specified number, perform the classification.

19. The computer program product according to claim 16, characterized in that, Execute the following method: The operation that changes the state of the endoscope is selected as the auxiliary insertion operation.

20. The computer program product according to claim 16, characterized in that, Execute the following method: When the insertion state related to forward movement is classified as the insertion state, the forward movement operation is selected even if the lumen deviates from the center of the field of view.

Images