Information processing device and its operating method, endoscope control device and its operating method, and storage medium

Abstract

The endoscope control device includes: an insertion shape classification unit that obtains a classification result that classifies the type of insertion shape of the endoscope insertion part inserted into the patient into one of a number of specified types; and a control unit that performs control related to the insertion operation of the endoscope insertion part based on the classification result.

Description

TECHNICAL FIELD

[0001] The present application relates to an information processing apparatus, an endoscope control apparatus, a method of operating an information processing apparatus, a method of operating an endoscope control apparatus, and a storage medium. BACKGROUND

[0002] In endoscope observation, an insertion operation for inserting an elongated insertion section having flexibility into a deep portion of a subject is performed. Further, in the field of endoscopes, a technique for assisting the insertion operation of an insertion section has been conventionally proposed.

[0003] Specifically, for example, in Japanese Patent No. 4274854, a structure is disclosed in which, in an endoscope insertion shape analysis apparatus that analyzes an insertion shape of an endoscope insertion section inserted into a body cavity, when a loop is formed by the insertion operation of the endoscope insertion section, an operation method for straightening the endoscope insertion section by canceling the loop is displayed.

[0004] Here, in the field of endoscopes, a technique for automating the insertion operation of an insertion section has been studied in recent years.

[0005] However, in Japanese Patent No. 4274854, a specific control method or the like for automating the insertion operation of an insertion section is not particularly mentioned. Therefore, according to the structure disclosed in Japanese Patent No. 4274854, for example, the following problem arises: it is not possible to perform appropriate insertion control corresponding to the insertion condition of the insertion section, such as individual differences in the internal state of a subject into which the insertion section is inserted and temporal changes in the insertion shape of the insertion section in the inside of the subject.

[0006] The present application was achieved in view of the foregoing, and an object thereof is to provide an information processing apparatus, an endoscope control apparatus, a method of operating an information processing apparatus, a method of operating an endoscope control apparatus, and a storage medium, which can perform appropriate insertion control corresponding to the insertion condition of an insertion section. SUMMARY

[0007] An information processing apparatus of one embodiment of the present application classifies a type of an insertion shape of an endoscope insertion section inserted into a subject using information about the insertion shape of the endoscope insertion section, and has an insertion shape classification section that obtains a classification result of classifying the type of the insertion shape of the endoscope insertion section inserted into the subject into one of a predetermined plurality of types, and an output section that outputs the classification result.

[0008] Another aspect of the present invention is an endoscope control device that uses information related to the insertion shape of an endoscope insertion portion inserted into a patient to perform control related to the insertion operation of the endoscope insertion portion, comprising: an insertion shape element extraction unit that extracts one or more structural elements related to the insertion shape of the endoscope insertion portion inserted into the patient to obtain an extraction result; and a control unit that performs control related to the insertion operation of the endoscope insertion portion based on the extraction result.

[0009] An endoscope control device according to one aspect of the present invention uses information related to the insertion shape of an endoscope insertion portion inserted into a patient to perform control related to the insertion operation of the endoscope insertion portion, comprising: an insertion shape element extraction unit that extracts one or more structural elements related to the insertion shape of the endoscope insertion portion inserted into the patient to obtain an extraction result; and a control unit that performs control related to the insertion operation of the endoscope insertion portion based on the extraction result.

[0010] An information processing apparatus according to one aspect of the present invention provides a method for operating such an apparatus, wherein the apparatus classifies the types of insertion shapes of an endoscope insertion portion inserted into a patient body using information relating to the insertion shape of the endoscope insertion portion inserted into the patient body, wherein an insertion shape classification unit performs processing to obtain a classification result that classifies the types of insertion shapes of the endoscope insertion portion inserted into the patient body into one of a predetermined plurality of types, and an output unit outputs the classification result.

[0011] Another aspect of the present invention discloses an endoscopic control device, wherein the endoscopic control device uses information related to the insertion shape of an endoscopic insertion portion inserted into a patient to perform control related to the insertion operation of the endoscopic insertion portion, wherein an insertion shape element extraction unit performs processing to extract one or more structural elements related to the insertion shape of the endoscopic insertion portion inserted into the patient and obtains extraction results; and a control unit performs control related to the insertion operation of the endoscopic insertion portion based on the extraction results.

[0012] In one aspect of the present invention, a program stored in a storage medium is used to cause a computer to execute: a process for obtaining a classification result that classifies the type of insertion shape of an endoscope insertion portion inserted into a subject into one of a predetermined plurality of types; and a process for outputting the classification result.

[0013] The storage medium of one aspect of the present invention stores a program for causing a computer to execute: processing for extracting one or more structural elements related to the insertion shape of an endoscope insert inserted into a patient and obtaining extraction results; and control related to the insertion operation of the endoscope insert based on the extraction results. Attached Figure Description

[0014] Figure 1 This is a diagram showing the structure of a key part of an endoscope system including an endoscope control device according to the first embodiment of the present invention.

[0015] Figure 2 This is a block diagram illustrating the specific structure of the endoscope system of the first embodiment.

[0016] Figure 3 This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0017] Figure 4A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0018] Figure 4B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0019] Figure 5A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0020] Figure 5B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0021] Figure 6A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0022] Figure 6B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0023] Figure 7A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0024] Figure 7B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0025] Figure 8A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0026] Figure 8B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0027] Figure 9A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0028] Figure 9B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0029] Figure 10 This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0030] Figure 11A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0031] Figure 11B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0032] Figure 12A This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0033] Figure 12B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0034] Figure 13A This is a diagram illustrating an example of a situation where information recorded in the endoscope system of the first embodiment is used to visualize the type of insertion shape of the insertion part over time.

[0035] Figure 13B This is a diagram illustrating an example of a situation where information recorded in the endoscope system of the first embodiment is used to visualize the type of insertion shape of the insertion part over time.

[0036] Figure 13C This is a diagram illustrating an example of a situation where information recorded in the endoscope system of the first embodiment is used to visualize the type of insertion shape of the insertion part over time.

[0037] Figure 14 This is a flowchart illustrating the outline of the control performed in the endoscopic system of the modified example of the first embodiment.

[0038] Figure 15A This is a diagram illustrating an example of an endoscopic image generated in an endoscopic system according to an embodiment.

[0039] Figure 15B It is shown in response to Figure 15A An example of a processed image obtained when an endoscopic image has undergone processing to detect the location of a lumen region.

[0040] Figure 15C It is used to explain that after obtaining Figure 15BThe control diagram is shown in the case of the processed image.

[0041] Figure 16 This is a block diagram illustrating the specific structure of the endoscope system according to the second embodiment.

[0042] Figure 17A This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment.

[0043] Figure 17B This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment.

[0044] Figure 17C This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment.

[0045] Figure 17D This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment. Detailed Implementation

[0046] The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0047] (First Embodiment)

[0048] Figures 1 to 15C This is an accompanying drawing of the first embodiment.

[0049] For example, such as Figure 1 As shown, the endoscope system 1 is configured to include an endoscope 10, a main body device 20, an insertion shape detection device 30, an external force information acquisition device 40, an input device 50, and a display device 60. Figure 1 This is a diagram showing the structure of a key part of an endoscope system including an endoscope control device according to an embodiment.

[0050] The endoscope 10 is configured to have an insertion part 11 that is inserted into the subject, an operation part 16 provided at the base end of the insertion part 11, and a universal cable 17 extending from the operation part 16. Here, the endoscope 10 is configured to be detachably connected to the main body device 20 via a scope connector (not shown) provided at the end of the universal cable 17.

[0051] In addition, a light guide (not shown) for transmitting illumination light supplied from the main unit 20 is provided inside the aforementioned insertion part 11, operation part 16 and universal cable 17.

[0052] The insertion part 11 is configured to have a flexible and elongated shape. Furthermore, the insertion part 11 is configured to have, from the front end side, a rigid front end 12, a bendable part 13, and a flexible elongated tube part 14.

[0053] Furthermore, inside the front end portion 12, the curved portion 13, and the flexible tube portion 14, a plurality of source coils 18 are arranged at predetermined intervals along the length direction of the insertion portion 11. These multiple source coils 18 generate magnetic fields corresponding to the coil drive signals supplied from the main body device 20.

[0054] An illumination window (not shown) is provided at the front end 12, which emits illumination light transmitted by a light guide provided inside the insertion part 11 onto the subject. Furthermore, a camera part 110 is provided at the front end 12. Figure 1 (Not shown in the figure), the camera unit 110 is configured to perform an operation corresponding to the camera control signal supplied from the main unit 20, and to output a camera signal by capturing an image of a subject illuminated by illumination light emitted through the illumination window.

[0055] The bending portion 13 is configured to be able to bend according to the control of the bending control unit 242 described later. In addition, the bending portion 13 is configured to be able to bend according to the operation of the angle knob (not shown) provided on the operation unit 16.

[0056] The operating unit 16 is configured to be shaped so that it can be operated by a user such as a doctor. Here, the operating unit 16 is provided with an angle knob, which is configured to bend the bending section 13 in four directions—up, down, left, and right—intersecting the length axis of the insertion section 11. Furthermore, the operating unit 16 is provided with one or more endoscope switches (not shown) that can provide indications corresponding to user input operations.

[0057] like Figure 1 As shown, the main unit 20 is configured to have one or more processors 20P and storage media 20M. Furthermore, the main unit 20 is configured to be detachably connected to the endoscope 10 via a universal cable 17.

[0058] Furthermore, the main unit 20 is configured to be detachably connected to the insertion shape detection device 30, the input device 50, and the display device 60. Furthermore, the main unit 20 is configured to perform actions corresponding to instructions from the input device 50. Furthermore, the main unit 20 is configured to generate an endoscopic image based on the camera signal output from the endoscope 10, and to perform actions to display the generated endoscopic image on the display device 60.

[0059] In this embodiment, the main body device 20 is configured to generate and output various control signals for controlling the operation of the endoscope 10. Furthermore, the main body device 20 functions as an endoscope control device and is configured to control the insertion operation of the insertion section 11 using insertion shape information (described later) output from the insertion shape detection device 30.

[0060] Furthermore, the main unit 20 is configured to generate an insertion shape image corresponding to the insertion shape information output from the insertion shape detection device 30, and to perform an operation to display the generated insertion shape image on the display device 60.

[0061] The insertion shape detection device 30 is configured to detect the magnetic fields emitted from the source coils 18 provided in the insertion section 11, and to obtain the positions of the plurality of source coils 18 based on the strength of the detected magnetic fields. Furthermore, the insertion shape detection device 30 is configured to generate insertion shape information representing the positions of the plurality of source coils 18 obtained as described above, and to output the insertion shape information to the main body device 20 and the external force information acquisition device 40.

[0062] That is, the insertion shape detection device 30 is configured to detect the insertion shape of the insertion part inserted into the test body to obtain insertion shape information, and output the obtained insertion shape information to the main body device 20 and the external force information acquisition device 40.

[0063] The external force information acquisition device 40 stores, for example, data on the curvature (or radius of curvature) and bending angle of a predetermined plurality of positions of the insertion part 11 when no external force is applied, and data on the curvature (or radius of curvature) and bending angle of the predetermined plurality of positions when a predetermined external force is applied to any position of the insertion part 11 from all anticipated directions.

[0064] In this embodiment, the external force information acquisition device 40 is configured to determine the position of each of the plurality of source coils 18 provided in the insertion part 11 based on the insertion shape information output from the insertion shape detection device 30, and obtain the magnitude and direction of the external force at each of the plurality of source coils 18 based on the curvature (or radius of curvature) and bending angle at each of the plurality of source coils 18 and by referring to various pre-stored data.

[0065] Furthermore, the external force information acquisition device 40 is configured to generate and output external force information to the main device 20, which indicates the magnitude and direction of the external force at the respective positions of the plurality of source coils 18 as described above.

[0066] In addition, in this embodiment, as a method for the external force information acquisition device 40 to calculate the external force at the respective positions of the plurality of source coils 18 provided in the insertion part 11, the method disclosed in Japanese Patent No. 5851204 or the method disclosed in Japanese Patent No. 5897092 can be used.

[0067] Furthermore, in this embodiment, for example, if the insertion part 11 is equipped with electronic components such as strain sensors, pressure sensors, acceleration sensors, gyroscope sensors and wireless components, it can also be configured such that the external force information acquisition device 40 calculates the external force at the respective positions of the plurality of source coils 18 based on the signals output from the electronic components.

[0068] The input device 50 is configured to have one or more user-operated input interfaces, such as a mouse, keyboard, and touch panel. Furthermore, the input device 50 is configured to output instructions corresponding to the user's operation to the main unit 20.

[0069] The display device 60 is configured to include, for example, a liquid crystal monitor. Furthermore, the display device 60 is configured to display endoscopic images, etc., output from the main unit 20 on a screen.

[0070] Next, refer to Figure 2 The specific structure of the endoscope system including the endoscope control device of the first embodiment will be described.

[0071] Figure 2 This is a block diagram illustrating the specific structure of the endoscope system according to the first embodiment.

[0072] like Figure 2 As shown, the endoscope 10 is configured to have a source coil 18, an imaging unit 110, an advance / retract mechanism 141, a bending mechanism 142, an AWS mechanism 143, and a rotation mechanism 144. Figure 2 This is a block diagram illustrating the specific structure of the endoscope system of the first embodiment.

[0073] The camera unit 110 is configured, for example, to have an observation window into which reflected light from the illuminated subject is incident; and an image sensor such as a color CCD that captures the reflected light and outputs an image signal.

[0074] The advance / retreat mechanism 141 is configured, for example, to have a pair of rollers positioned opposite each other across the insertion portion 11; and a motor is supplied with a rotational driving force to rotate the pair of rollers. Furthermore, the advance / retreat mechanism 141 is configured, for example, to drive the motor according to an advance / retreat control signal output from the main body device 20, and to rotate the pair of rollers according to the rotational driving force supplied from the motor, thereby enabling selective operation of either advancing the insertion portion 11 or retracting the insertion portion 11.

[0075] The bending mechanism 142 is configured, for example, to have a plurality of bending members provided on the bending section 13, a plurality of lines connected to the plurality of bending members, and a motor supplied with a rotational driving force for pulling the plurality of lines. Furthermore, the bending mechanism 142 is configured, for example, to drive the motor according to a bending control signal output from the main body device 20, and to change the traction amount of each of the plurality of lines according to the rotational driving force supplied from the motor, thereby enabling the bending section 13 to bend in four directions: up, down, left, and right.

[0076] The AWS (Air feeding, Water feeding, and Suction) mechanism 143 is configured, for example, to have two pipelines: an air feeding and water feeding pipeline and a suction pipeline located inside the endoscope 10 (insertion section 11, operation section 16, and universal cable 17); and a solenoid valve that opens one of the two pipelines and closes the other.

[0077] Furthermore, in this embodiment, the AWS mechanism 143 is configured such that, when the solenoid valve performs an operation to open the air and water supply pipeline according to the AWS control signal output from the main body device 20, a fluid containing at least one of water and air supplied from the main body device 20 can flow in the air and water supply pipeline, and the fluid can be discharged from the outlet formed at the front end 12.

[0078] Furthermore, the AWS mechanism 143 is configured, for example, to allow the suction force generated in the main body device 20 to act on the suction line when the AWS control signal output from the main body device 20 is used to open the suction line in the solenoid valve, and to allow the suction force to draw in objects present near the suction port formed at the front end 12.

[0079] The rotating mechanism 144 is configured, for example, to have a gripping member that holds the insertion part 11 at the base end of the flexible tube 14, and a motor that supplies a rotational driving force for rotating the gripping member. Furthermore, the rotating mechanism 144 is configured, for example, to drive the motor according to a rotational control signal output from the main body device 20, and to rotate the gripping member according to the rotational driving force supplied from the motor, thereby enabling the insertion part 11 to rotate about the insertion axis (length axis).

[0080] <Details of Main Unit 20>

[0081] like Figure 2 As shown, the main device 20 is configured to include a light source unit 210, an image processing unit 220, a coil drive signal generation unit 230, an endoscope function control unit 240, a display control unit 250, and a system control unit 260.

[0082] The light source unit 210 is configured, for example, to have one or more LEDs or one or more lamps as a light source. Furthermore, the light source unit 210 is configured to generate illumination light for illuminating the specimen into which the insertion unit 11 is inserted, and to supply this illumination light to the endoscope 10. In addition, the light source unit 210 is configured to vary the amount of illumination light according to a system control signal supplied from the system control unit 260.

[0083] The image processing unit 220 is configured, for example, to have an image processing circuit. Furthermore, the image processing unit 220 is configured to generate an endoscope image by performing predetermined processing on the camera signal output from the endoscope 10, and output the generated endoscope image to the display control unit 250 and the system control unit 260.

[0084] The coil drive signal generation unit 230 is configured, for example, to have a drive circuit. Furthermore, the coil drive signal generation unit 230 is configured to generate and output a coil drive signal for driving the source coil 18 based on a system control signal supplied from the system control unit 260.

[0085] The endoscope function control unit 240 is configured to perform actions to control the functions implemented by the endoscope 10 based on the insertion control signal supplied from the system control unit 260. Specifically, the endoscope function control unit 240 is configured to perform actions to control at least one of the following functions: an advance / retract function implemented by the advance / retract mechanism 141, a bend function implemented by the bend mechanism 142, an AWS function implemented by the AWS mechanism 143, and a rotation function implemented by the rotation mechanism 144. Furthermore, the endoscope function control unit 240 is configured to have an advance / retract control unit 241, a bend control unit 242, an AWS control unit 243, and a rotation control unit 244.

[0086] The forward / backward control unit 241 is configured to generate and output forward / backward control signals for controlling the operation of the forward / backward mechanism 141 based on the insertion control signals supplied from the system control unit 260. Specifically, the forward / backward control unit 241 is configured to generate and output forward / backward control signals, for example, for controlling the rotational state of the motor provided on the forward / backward mechanism 141, based on the insertion control signals supplied from the system control unit 260.

[0087] The bending control unit 242 is configured to generate and output a bending control signal for controlling the operation of the bending mechanism 142 based on the insertion control signal supplied from the system control unit 260. Specifically, the bending control unit 242 is configured to generate, for example, a bending control signal for controlling the rotational state of the motor provided in the bending mechanism 142 based on the insertion control signal supplied from the system control unit 260.

[0088] AWS control unit 243 is configured to control a pump (not shown) or the like based on an insertion control signal supplied from system control unit 260, thereby enabling selective operation of either an operation for supplying a fluid containing at least one of water and air to endoscope 10 or an operation for generating a suction force for aspirating an object present near the suction port of the front end 12.

[0089] Furthermore, the AWS control unit 243 is configured to generate and output AWS control signals for controlling the operation of the AWS mechanism 143. Specifically, the AWS control unit 243 is configured to generate and output AWS control signals for controlling the operation state of the solenoid valve provided in the AWS mechanism 143, based on the insertion control signals supplied from the system control unit 260.

[0090] The rotation control unit 244 is configured to generate and output a rotation control signal for controlling the operation of the rotation mechanism 144 based on the insertion control signal supplied from the system control unit 260. Specifically, the rotation control unit 244 is configured to generate, for example, a rotation control signal for controlling the rotation state of the motor provided in the rotation mechanism 144 based on the insertion control signal supplied from the system control unit 260.

[0091] That is, the endoscope function control unit 240 is configured to generate and output control signals corresponding to the following operations based on the insertion control signal supplied from the system control unit 260, as control signals corresponding to the basic operations realized by the function of the endoscope 10: a push operation equivalent to an operation to advance the insertion part 11; a pull operation equivalent to an operation to retract the insertion part 11; an angle operation equivalent to an operation to bend the bending part 13 and orient the front end 12 toward a direction (e.g., one of eight directions) that intersects the insertion axis (length axis) of the insertion part 11; a torsion operation equivalent to an operation to rotate the insertion part 11 about the insertion axis (length axis); a gas delivery operation to expel gas in front of the front end 12; a water delivery operation to expel liquid in front of the front end 12; and a suction operation to aspirate tissue in front of the front end 12, etc.

[0092] The display control unit 250 performs processing to generate a display image including the endoscopic image output from the image processing unit 220, and performs processing to display the generated display image on the display device 60. Furthermore, the display control unit 250 performs processing to display an insertion shape image (described later) output from the system control unit 260 on the display device 60.

[0093] The system control unit 260 generates and outputs system control signals for performing actions corresponding to instructions from the operation unit 16 and the input device 50. Furthermore, the system control unit 260 is configured to include an insertion shape image generation unit 261, an insertion shape classification unit 262, an insertion control unit 263, and a classification result recording unit 264.

[0094] The insertion shape image generation unit 261 generates an insertion shape image that represents the insertion shape of the insertion part 11 inserted into the object in a two-dimensional manner based on the insertion shape information output from the insertion shape detection device 30 (described later). Furthermore, the insertion shape image generation unit 261 outputs the insertion shape image generated as described above to the display control unit 250.

[0095] The insertion shape classification unit 262 performs the following processing, which is used to obtain a classification result based on the insertion shape image generated by the insertion shape image generation unit 261, classifying the type of the insertion shape of the insertion part 11 contained in the insertion shape image into one of a number of specified types.

[0096] <Structure of Insert Shape Classification Section 262>

[0097] Here, a specific example of the structure of the insertion shape classification unit 262 in this embodiment will be described.

[0098] In this embodiment, the insertion shape classification unit 262 is configured, for example, to process the data using a classifier (e.g., a classifier CLP) to obtain a classification result that classifies the type of the insertion shape of the insertion part 11 contained in the insertion shape image generated by the insertion shape image generation unit 261 into one of a predetermined number of types. The classifier is generated by learning the combination coefficients (weights) of a CNN (Convolutional Neural Network), which is equivalent to a multi-layer neural network containing an input layer, one or more convolutional layers, and an output layer, using learning methods such as deep learning.

[0099] When generating the aforementioned classifier CLP, machine learning is performed, for example, using teaching data that includes: an insert shape image identical to the insert shape image generated by the insert shape image generation unit 261; and a label that indicates a classification result of classifying the insert shape of the insert part 11 contained in the insert shape image into one of a number of specified categories.

[0100] Here, the various types specified above are, for example, set to be the type of insertion shape that can be formed during the period from the time point when the insertion part 11 is first inserted into the test body to the time point when the insertion of the insertion part 11 into the test body ends, and has a characteristic shape that helps to determine whether the operation of the insertion of the insertion part 11 is successful when it is performed manually or automatically, and whether it is necessary to change the operation content.

[0101] When generating the aforementioned teaching data, for example, the following task is performed: the label corresponding to the judgment result of a skilled person visually determining which of the specified multiple categories the insertion shape of the insertion part 11 contained in an insertion shape image belongs to is assigned to that insertion shape image.

[0102] Therefore, according to the aforementioned classifier CLP, for example, multidimensional data such as the pixel values ​​of each pixel contained in the inserted shape image generated by the inserted shape image generation unit 261 is obtained, and this multidimensional data is input as input data to the input layer of the neural network. As a result, multiple likelihoods corresponding to each type of the inserted shape that can be classified as the type of the inserted shape of the inserted part 11 contained in the inserted shape image can be obtained as output data output from the output layer of the neural network.

[0103] Furthermore, based on the processing using the aforementioned classifier CLP, for example, it is possible to obtain a type of insertion shape corresponding to the highest likelihood among the multiple likelihoods contained in the output data output from the output layer of the neural network, as the classification result of the insertion shape of the insertion unit 11.

[0104] That is, the insertion shape classification unit 262 is configured to process the data using a classifier CLP to obtain a classification result representing the type of insertion shape of the insertion part 11 inserted into the test body. The classifier CLP is generated by machine learning using teaching data, which includes: an insertion shape image representing the insertion shape of the insertion part 11; and a label representing a classification result that classifies the insertion shape of the insertion part 11 contained in the insertion shape image into one of a number of specified types.

[0105] Here, a specific example of the classification result of the insertion shape of the insertion part 11 obtained by processing using the aforementioned classifier CLP will be described. In addition, an example of a case in which the following classification result is obtained is given, which corresponds to the type of insertion shape that appears immediately before the formation of the α ring and just after the release of the insertion part 11 that can be formed by the insertion part 11 inserted into the test body.

[0106] Insert shape classification section 262, for example, based on... Figure 3 The pixel values ​​of each pixel contained in the inserted shape image SGA shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into category TA. Figure 3 This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0107] The aforementioned type TA is obtained, for example, as a classification result corresponding to the state in which the insertion portion 11 maintains a generally straight shape and the anterior end portion 12 is located in the interval from near the anus to near the entrance of the S-shaped colon.

[0108] Insert shape classification section 262, for example, based on... Figure 4A The inserted shape image SGB1 or shown Figure 4B The pixel values ​​of each pixel contained in the inserted shape image SGB2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into category TB. Figure 4A and Figure 4B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0109] The aforementioned type TB is obtained, for example, as a classification result corresponding to the state in which the anterior end 12 is located inside the S-shaped colon and the insertion portion 11 forms a flexed shape as the basis of the α-ring.

[0110] Insert shape classification section 262, for example, based on... Figure 5A The inserted shape image SGC1 or shown Figure 5BThe pixel values ​​of each pixel contained in the inserted shape image SGC2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into type TC. Figure 5A and Figure 5B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0111] The aforementioned type TC is obtained, for example, as a classification result corresponding to the state from the point where the front end 12 intersects with either the bend 13 or the flexible tube 14 to the point where the front end 12 reaches the vicinity of the upper end of the α ring.

[0112] Insert shape classification section 262, for example, based on... Figure 6A The inserted shape image SGD1 or shown Figure 6B The pixel values ​​of each pixel contained in the inserted shape image SGD2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into type TD. Figure 6A and Figure 6B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0113] The aforementioned type TD is obtained, for example, as a classification result corresponding to the state where the front end 12 reaches a position slightly beyond the upper end of the α ring.

[0114] Insert shape classification section 262, for example, based on... Figure 7A The inserted shape image SGE1 or shown Figure 7B The pixel values ​​of each pixel contained in the inserted shape image SGE2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into category TE. Figure 7A and Figure 7B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0115] The aforementioned type TE is obtained, for example, as a classification result corresponding to any state between the state where the anterior end 12 reaches the vicinity of the spleen flexure and the state where the anterior end 12 reaches a position sufficiently far from the upper end of the α ring.

[0116] Insert shape classification section 262, for example, based on... Figure 8A The inserted shape image SGF1 or shown Figure 8BThe pixel values ​​of each pixel contained in the inserted shape image SGF2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into category TF. Figure 8A and Figure 8B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0117] The aforementioned type TF is obtained, for example, as a classification result corresponding to the state in which the α-ring shrinks as the α-ring formed by the insertion part 11 is released.

[0118] Insert shape classification section 262, for example, based on... Figure 9A The inserted shape image SGG1 or shown Figure 9B The pixel values ​​of each pixel contained in the inserted shape image SGG2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into type TG. Figure 9A and Figure 9B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0119] The aforementioned type TG is obtained, for example, as a classification result corresponding to the state from the point where the α ring formed by the insertion part 11 is released and the α ring is transferred to a shape similar to the N ring, to the state after the α ring has just been completely released.

[0120] Insert shape classification section 262, for example, based on... Figure 10 The pixel values ​​of each pixel contained in the inserted shape image SGH shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into category TH. Figure 10 This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0121] The aforementioned type TH is obtained, for example, as a classification result corresponding to either the state in which the anterior end 12 reaches the vicinity of the entrance of the transverse colon or the state in which the insertion part 11 moves to a generally straight shape after the α ring is released.

[0122] Insert shape classification section 262, for example, based on... Figure 11A The inserted shape image SGI1 or shown Figure 11B The pixel values ​​of each pixel contained in the inserted shape image SGI2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into category TI. Figure 11A and Figure 11B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0123] The aforementioned type TI is obtained, for example, as a classification result corresponding to the state in which the anterior end 12 is located inside the transverse colon.

[0124] Insert shape classification section 262, for example, based on... Figure 12A The inserted shape image SGJ1 or shown Figure 12B The pixel values ​​of each pixel contained in the inserted shape image SGJ2 shown are input into the classifier CLP and the output data is processed to obtain a classification result that classifies the inserted shape of the insertion part 11 into type TJ. Figure 12A and Figure 12B This is a diagram illustrating an example of an insertion shape image generated in the endoscope system of the first embodiment.

[0125] The aforementioned type TJ is obtained, for example, as a classification result corresponding to the state where the anterior end 12 is located in the interval from the ascending colon to the vicinity of the cecum.

[0126] Furthermore, according to this embodiment, for example, when generating the classifier CLP, learning is performed by using an inserted shape image after modifying the label of at least one of the 10 labels corresponding to each of the categories TA to TJ, or by adding an inserted shape image after adding a label of a new category that is different from the 10 labels corresponding to each of the categories TA to TJ, thereby obtaining a classification result corresponding to the category of the inserted shape that appears immediately before the formation of a shape different from the α ring and just after the release is completed.

[0127] Specifically, according to this embodiment, for example, a classification result can be obtained corresponding to the type of insertion shape that appears immediately before the formation of at least one of the shapes selected from the back α-ring, reverse α-ring, N-ring, γ-ring, and bar shape, up to the point just after the release is completed.

[0128] Furthermore, according to this embodiment, for example, by appropriately changing the method of assigning labels to the learning insertion shape image used when generating the classifier CLP, it is possible to obtain a classification result that corresponds to the type of desired insertion shape that can be formed during the period from the start of inserting the insertion part 11 into the test subject to the end of inserting the insertion part 11 into the test subject.

[0129] The insertion control unit 263 is configured to generate an insertion control signal, which includes information for controlling the insertion operation of the insertion unit 11, based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, and the classification result obtained by the insertion shape classification unit 262, and output the insertion control signal to the endoscope function control unit 240.

[0130] Specifically, the insertion control unit 263 is configured to generate an insertion control signal containing information based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, as well as the classification result obtained by the insertion shape classification unit 262, and output the insertion control signal to the endoscope function control unit 240. The information is used to perform control related to at least one of the following: the start of the insertion operation, the continuation of the insertion operation, the interruption of the insertion operation, the restart of the insertion operation, the stop of the insertion operation, and the completion of the insertion operation, as control of the insertion operation of the insertion unit 11.

[0131] Furthermore, the insertion control unit 263 is configured to generate an insertion control signal containing information based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, and the classification result obtained by the insertion shape classification unit 262, and output the insertion control signal to the endoscope function control unit 240, wherein the information is used to control at least one of the operation amount, the operation speed, and the operation force of the insertion operation of the insertion unit 11.

[0132] Here, the insertion control unit 263 of this embodiment is configured, for example, to set control content based on at least one of the following: the type of the current insertion shape of the insertion unit 11 as a classification result obtained by the insertion shape classification unit 262; the endoscope image output from the image processing unit 220; the external force information output from the external force information acquisition device 40; and the insertion shape image generated by the insertion shape image generation unit 261; to generate an insertion control signal containing information for controlling the insertion operation of the insertion unit 11 using the set control content; and to output the insertion control signal to the endoscope function control unit 240.

[0133] Therefore, the insertion control unit 263 can set an operation control group CGA based on at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, according to the type of the current insertion shape of the insertion unit 11 shown as a classification result obtained by the insertion shape classification unit 262. The operation control unit 263 generates and outputs an insertion control signal containing information related to the set operation control group CGA. For example, the operation control group CGA has control content for individually executing a basic operation selected from each basic operation realized by the endoscope 10 to perform the insertion operation of the insertion unit 11.

[0134] Specifically, the Operation Control Group (CGA) includes, for example, control parameters related to the forward advance, forward speed, and operating force during a push operation.

[0135] Furthermore, the insertion control unit 263 can set an operation control group CGB based on at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, according to the type of the current insertion shape of the insertion unit 11 as a classification result obtained by the insertion shape classification unit 262. It generates and outputs an insertion control signal containing information related to the set operation control group CGB. For example, the operation control group CGB has control content that is used to combine and execute multiple basic operations selected from the basic operations implemented by the endoscope 10 to perform the insertion operation of the insertion unit 11.

[0136] Specifically, the operation control group (CGB) includes, for example, control parameters related to the amount of retraction, retraction speed, rotation angle, rotation direction, and operating force when performing pull and twist operations in combination.

[0137] Furthermore, the operation control group CGB is configured to continuously or simultaneously execute control content for multiple basic operations selected from the various basic operations implemented by the endoscope 10. That is, the control content of the operation control group CGB is configured to be more complex than the control content of the operation control group CGA.

[0138] That is, the insertion control unit 263 is configured to perform control based on either the operation control group CGA or the operation control group CGB, as a control corresponding to the type of the current insertion shape of the insertion unit 11 shown as a classification result obtained by the insertion shape classification unit 262. The operation control group CGA has control content for individually executing a basic operation selected from each basic operation implemented by the endoscope 10 to perform the insertion operation of the insertion unit 11, and the operation control group CGB has control content for combining multiple basic operations selected from each basic operation implemented by the endoscope 10 to perform the insertion operation of the insertion unit 11.

[0139] Furthermore, the insertion control unit 263 performs control related to the insertion operation of the insertion unit 11 based on at least one of the following: an image obtained by the endoscope 10 capturing images inside the subject, information indicating the magnitude of the external force applied to the insertion unit 11, and information indicating the insertion shape of the insertion unit 11, as well as the classification result obtained by the insertion shape classification unit 262.

[0140] The classification result recording unit 264 is configured to perform an operation for recording the classification results obtained by the insertion shape classification unit 262 in a time sequence.

[0141] In this embodiment, at least a portion of the functions of the main device 20 can be implemented by the processor 20P. Furthermore, in this embodiment, at least a portion of the main device 20 can be configured as various electronic circuits, or as a circuit module in an integrated circuit such as an FPGA (Field Programmable Gate Array).

[0142] In addition, the structure of this embodiment can be appropriately modified so that, for example, a computer reads a program from a storage medium 20M such as a memory to perform at least a portion of the functions of the main device 20, and performs actions corresponding to the read program.

[0143] like Figure 2 As shown, the insertion shape detection device 30 is configured to have a receiving antenna 310 and an insertion shape information acquisition unit 320.

[0144] The receiving antenna 310 is configured, for example, to have multiple coils for three-dimensionally detecting magnetic fields emitted from multiple source coils 18 respectively. Furthermore, the receiving antenna 310 is configured to detect magnetic fields emitted from the multiple source coils 18 respectively, generate a magnetic field detection signal corresponding to the strength of the detected magnetic field, and output the magnetic field detection signal to the insertion shape information acquisition unit 320.

[0145] The insertion shape information acquisition unit 320 is configured to acquire the positions of each of the plurality of source coils 18 based on the magnetic field detection signal output from the receiving antenna 310. Furthermore, the insertion shape information acquisition unit 320 is configured to generate insertion shape information and output it to the insertion shape image generation unit 261, the insertion shape information representing the positions of each of the plurality of source coils 18 acquired as described above.

[0146] Specifically, the insertion shape information acquisition unit 320 acquires, for example, multiple three-dimensional coordinate values ​​in a spatial coordinate system as the positions of the multiple source coils 18, which is virtually set with the origin or reference point being a predetermined position (such as the anus) of the subject to be inserted by the insertion unit 11. Furthermore, the insertion shape information acquisition unit 320 generates insertion shape information containing the multiple three-dimensional coordinate values ​​acquired as described above, and outputs the insertion shape information to the insertion shape image generation unit 261.

[0147] Then, in this case, for example, the insertion shape image generation unit 261 performs the following processing: processing for obtaining multiple two-dimensional coordinate values, which correspond to each of the multiple three-dimensional coordinate values ​​contained in the insertion shape information output from the insertion shape information acquisition unit 320; processing for interpolating the obtained multiple two-dimensional coordinate values; and processing for generating an insertion shape image corresponding to the interpolated multiple two-dimensional coordinate values.

[0148] In this embodiment, at least a portion of the insertion shape detection device 30 may be configured as an electronic circuit, or as a circuit module in an integrated circuit such as an FPGA (Field Programmable Gate Array). Furthermore, in this embodiment, for example, the insertion shape detection device 30 may be configured to have one or more processors (CPUs, etc.).

[0149] According to this embodiment, for example, when the insertion shape image generation unit 261 generates a three-dimensional insertion shape image that three-dimensionally represents the insertion shape of the insertion part 11 inserted into the test subject, the classifier CLP of the insertion shape classification unit 262 uses multi-dimensional data such as pixel values ​​obtained from the three-dimensional insertion shape image as input data to classify the type of insertion shape of the insertion part 11. Moreover, in this case, for example, a 3D-CNN (3D Convolutional Neural Network) can be used to generate the classifier CLP.

[0150] According to this embodiment, for example, the classifier CLP of the insertion shape classification unit 262 may be configured to classify the types of insertion shapes of the insertion unit 11 using multiple three-dimensional coordinate values ​​contained in the insertion shape information output from the insertion shape detection device 30 as input data. Moreover, in this case, for example, the classifier CLP can be generated using a known linear discriminant function or a known neural network, which uses numerical values ​​as features to classify the types of insertion shapes of the insertion unit 11.

[0151] According to this embodiment, for example, when generating the classifier CLP, the inserted shape image is assigned a label representing the classification result of classifying the inserted shape of the inserted part 11 into one of a specified number of categories, and machine learning is performed using teaching data including the label and multiple three-dimensional coordinate values ​​used when generating the inserted shape image.

[0152] Next, the operation of this embodiment will be explained. Furthermore, the following explanation will focus on the case where control is performed related to the insertion operation of the insertion part 11 into the large intestine via the anus. Additionally, the following explanation will focus on the case where an α-ring is formed by the insertion part 11 inserted into the intestinal tract.

[0153] After connecting the various parts of the endoscope system 1 and turning on the power, the doctor or other user may, for example, configure the insertion part 11 such that the front end 12 is located near the anus or rectum of the subject.

[0154] According to the aforementioned user operation, the illumination light supplied from the light source unit 210 is irradiated onto the subject, the subject irradiated by the illumination light is photographed by the imaging unit 110, and the endoscopic image obtained by photographing the subject is output from the image processing unit 220 to the display control unit 250 and the system control unit 260.

[0155] Furthermore, according to the aforementioned user operation, a coil drive signal is supplied from the coil drive signal generation unit 230, and a magnetic field is generated from the plurality of source coils 18 respectively according to the coil drive signal. The insertion shape information obtained by detecting the magnetic field is output from the insertion shape information acquisition unit 320 to the system control unit 260, and the insertion shape image corresponding to the insertion shape information is generated by the insertion shape image generation unit 261.

[0156] Furthermore, based on the aforementioned user operation, the external force information acquisition device 40 outputs external force information, representing the magnitude and direction of the external force at the respective positions of the multiple source coils 18, to the system control unit 260.

[0157] With the insertion unit 11 configured as described above, the user can, for example, instruct the main unit 20 to begin inserting the insertion unit 11 by turning on the automatic insertion switch (not shown) of the input device 50.

[0158] When the classification result recording unit 264 detects an instruction to perform insertion control for the insertion start unit 11, it starts, for example, an operation to record the classification results obtained by the insertion shape classification unit 262 in a time sequence and at fixed intervals.

[0159] The insertion control unit 263 sets control content based on at least one of the following: the type of the current insertion shape of the insertion unit 11 shown as a classification result obtained by the insertion shape classification unit 262; the endoscope image output from the image processing unit 220; the external force information output from the external force information acquisition device 40; and the insertion shape image generated by the insertion shape image generation unit 261.

[0160] Specifically, if the insertion control unit 263 detects that the current insertion shape of the insertion unit 11 shown as a classification result obtained by the insertion shape classification unit 262 is any of the types TA, TH, TI and TJ, it generates and outputs an insertion control signal containing information related to the operation control group CGA with set control content, based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40 and the insertion shape image generated by the insertion shape image generation unit 261.

[0161] Furthermore, if the insertion control unit 263 detects that the current insertion shape of the insertion unit 11, as a classification result obtained by the insertion shape classification unit 262, is any of the types TB, TC, TD, TE, TF, and TG, it generates and outputs an insertion control signal containing information related to the operation control group CGB with set control content, based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261.

[0162] That is, according to the specific example described above, when the insertion control unit 263 detects that the type of the current insertion shape of the insertion unit 11 does not conform to the type of insertion shape that appeared immediately before the formation of the α ring until just after its release, it generates and outputs an insertion control signal containing information related to the operation control group CGA.

[0163] Furthermore, according to the specific example described above, when the insertion control unit 263 detects that the type of the current insertion shape of the insertion unit 11 matches the type of insertion shape that appeared immediately before the formation of the α ring until just after its release, it generates and outputs an insertion control signal containing information related to the operation control group CGB, wherein the operation control group CGB has more complex control content than the control content of the operation control group CGA.

[0164] In addition, in this embodiment, for example, when the insertion control unit 263 sets control content based on the endoscopic image output from the image processing unit 220, it can use the classifier CLQ described later for processing.

[0165] Furthermore, in this embodiment, the insertion control unit 263 sets control content for advancing the insertion unit 11 by a relatively large advance amount based on the processing result image PRG obtained by processing using the classifier CLQ, for example, if there is a lumen region in the central part of the processing result image PRG, and if there is a lumen region in the peripheral part of the processing result image PRG, it sets control content for advancing the insertion unit 11 by a relatively small advance amount.

[0166] For example, after the user confirms, based on the insertion shape image displayed on the display device 60, that the insertion shape of the insertion part 11 inserted into the subject no longer changes, they can instruct the main device 20 to stop the insertion control of the insertion part 11 by turning off the automatic insertion switch of the input device 50.

[0167] When the classification result recording unit 264 detects an instruction to stop the insertion control of the insertion unit 11, it stops the operation of recording the classification results obtained by the insertion shape classification unit 262 in a time sequence and at fixed intervals.

[0168] Furthermore, when an endoscope is inserted into the large intestine for examination, various situations can arise, corresponding to combinations such as the direction of the large intestine, the shape of the insertion, and the length of the insertion. Moreover, when a skilled physician manually inserts the endoscope, they can appropriately determine the magnitude of the force applied to the insertion and the type of manipulation performed based on an assessment of the current situation.

[0169] Furthermore, existing solutions that link the automation of the insertion of the endoscope insertion unit present the following problem: it is very difficult to obtain a judgment result equivalent to that of a skilled physician's subjective judgment and to perform control accordingly.

[0170] In this embodiment, the insertion shape classification unit 262 performs the following processing: based on the viewpoint that is roughly equivalent to that of a skilled person when making a subjective judgment or evaluation of the success or failure of the insertion operation of the insertion unit 11, the types of insertion shapes of the insertion unit 11 contained in the insertion shape image generated by the insertion shape image generation unit 261 are classified to obtain a classification result.

[0171] Furthermore, according to this embodiment, insertion control is performed in the insertion control unit 263 according to the type of insertion shape of the insertion part 11 shown as a classification result obtained by the insertion shape classification unit 262. Therefore, according to this embodiment, for example, appropriate insertion control can be performed corresponding to the insertion status of the insertion part, such as individual differences in the internal state of the subject to which the insertion part is inserted and changes in the insertion shape of the insertion part inside the subject over time.

[0172] Furthermore, with existing endoscopic observation using a device having the same function as the insertion shape detection device 30, information related to the insertion shape of the endoscope insertion portion can be recorded during observation within the subject. However, there is a problem: this information is not intended to be reused after the observation within the subject has ended. Therefore, with existing endoscopic observation using a device having the same function as the insertion shape detection device 30, issues arise, for example, corresponding to the aforementioned problems, such as the difficulty in evaluating and analyzing the shift in the insertion shape of the endoscope insertion portion during observation within the subject, after the observation within the subject has ended.

[0173] In this regard, according to the processing of this embodiment described above, during the period from when the automatic insertion switch of the input device 50 is turned on to when it is turned off, the classification results obtained by the insertion shape classification unit 262 are recorded in the classification result recording unit 264 in a time sequence. Therefore, according to this embodiment, by using the information recorded in the classification result recording unit 264, it is possible to evaluate and analyze the shift of the insertion shape of the insertion part 11 during the observation of the subject after the observation of the subject has been completed.

[0174] Specifically, for example, the display control unit 250 performs processing for visualizing the information recorded in the classification result recording unit 264, thereby enabling the display device 60 to display information including... Figures 13A-13C The graph shown represents the passage of time of the types of insertion shapes of the insertion section 11 obtained as a classification result of the insertion shape classification section 262. Figures 13A-13C This is a diagram illustrating an example of how, over time, the type of insertion shape of the insertion section can be visualized using the recorded information in the endoscope system of the first embodiment.

[0175] Figure 13A The curve GRA is generated as follows, which represents the time progression of the type of insertion shape of the insertion part 11 when the α ring formed by the insertion part 11 is released and the anterior end 12 reaches the ascending colon.

[0176] according to Figure 13A The curve GRA shows that during the period PKA, which corresponds to the time from the start of insertion control of insertion section 11 to time NA, the type of insertion shape of insertion section 11 is maintained at type TA. Therefore, according to Figure 13A The curve graph GRA, for example in the period PKA, can confirm that the anterior end 12 reaches the vicinity of the entrance of the S-shaped colon while the insertion part 11 maintains a roughly straight shape.

[0177] according to Figure 13A The curve GRA shows that during the period PKB, which corresponds to the time interval from time NA to time NC, the type of insertion shape of the insertion part 11 changes from type TB to type TC. Therefore, according to Figure 13A The curve graph GRA, for example in the period PKB, can confirm that the insertion part 11 begins to form an α ring.

[0178] according to Figure 13A The curve GRA shows that during the period PKC, which corresponds to the time interval from time NC to time NE, the type of insertion shape of the insertion part 11 changes from type TD to type TG. Furthermore, according to... Figure 13A The curve GRA shows that, during the PKC period, the type of insertion shape of the insertion part 11 vibrates and changes arbitrarily between type TE and type TF. Therefore, according to Figure 13A The curve graph GRA, for example, during the PKC period, can confirm that the α-ring formed by the insertion part 11 is attempting to shrink and release the α-ring.

[0179] according to Figure 13A The curve GRA shows that during the period PKD, which corresponds to the time interval from time NE to time NG, the type of the insertion shape of the insertion part 11 changes from type TF to type TG, and then oscillates in a manner that becomes any type between type TG and type TH. Therefore, according to Figure 13A The curve graph GRA, for example, during the period PKD, can confirm that the release of the α ring formed by the insertion part 11 is being completed.

[0180] according to Figure 13AThe curve GRA shows that during the period PKE, which corresponds to the time interval from time NG to time NI, the type of the insertion shape of the insertion part 11 changes from type TH through type TI to type TJ. Therefore, according to Figure 13A The curve graph GRA, for example, during PKE, can confirm that the anterior end 12 reaches the ascending colon via the transverse colon.

[0181] Figure 13B The GRB curve is generated as follows, which represents the time progression of the types of insertion shapes of the insertion portion 11 in cases where it is difficult to remove the α-ring due to individual differences in the shape of the S-shaped colon, etc. Furthermore, in Figure 13B In the diagram, for ease of illustration, the scale of the horizontal axis is aligned with... Figure 13A and Figure 13C different.

[0182] according to Figure 13B The curve GRB shows that, during the period PKF, which corresponds to the time from the start of insertion control of insertion section 11 at time NY to time NK, the type of insertion shape of insertion section 11 oscillates and changes arbitrarily among type TD, type TE, and type TF. Therefore, according to Figure 16 The curve graph GRB of B, for example, in the period PKF, can confirm that although attempts were made to reduce the α ring formed by the insertion part 11, they were unsuccessful.

[0183] according to Figure 13B The curve GRB shows that during the period PKG, which corresponds to the time interval from time NK to time NM, the type of insertion shape of the insertion part 11 oscillates and changes erratically, becoming any one of type TB, type TC, type TD, type TE, and type TF. Therefore, according to Figure 13B The curve GRB, for example, during the PKG period, can confirm that the α-ring is being reformed based on the insertion part 11 while the shape of the intestinal tube is adjusted in a way that does not hinder the insertion of the insertion part 11 as much as possible.

[0184] according to Figure 13B The curve GRB shows that during the period PKH, which corresponds to the time interval from time NM to time NP, the type of insertion shape of the insertion part 11 oscillates and changes erratically in a manner that becomes any one of type TC, type TD, type TE, type TF, and type TG. Therefore, according to Figure 13B The curve graph GRB, for example, during the PKH period, can confirm that an attempt is being made to undo the α-ring that has been re-formed through the insertion part 11.

[0185] according to Figure 13BThe curve GRB shows that during the period PKI, corresponding to the time interval from time NQ after time NP to time NR, the type of the insertion shape of the insertion part 11 changes from type TF through type TG to type TH. Therefore, according to Figure 13B The curve GRB, for example, during the PKI period, can confirm the successful release of the α-ring re-formed by the insertion part 11.

[0186] Figure 13C The curve GRC is generated as follows, which represents the time progression of the type of insertion shape of the insertion part 11 when the anterior end 12 reaches the ascending colon without forming an α ring based on the insertion part 11.

[0187] according to Figure 13C The curve GRC shows that after the insertion control of the insertion section 11 begins at time NZ, the types of insertion shapes of the insertion section 11 change in the order of type TA, type TH, type TI, and type TJ. Therefore, according to... Figure 13C The curve graph GRC, for example, can confirm that no problems that hinder the insertion of the insertion part 11 occur throughout the entire region of the large intestine.

[0188] According to this embodiment, the operation of the classification result recording unit 264 in recording the classification results obtained by the insertion shape classification unit 262 in a time sequence at fixed intervals is not limited to the case where the insertion unit 11 is automatically inserted under the control of the insertion control unit 263, but can also be configured to be performed when the insertion unit 11 is manually inserted by the user. Furthermore, when the operation of the classification result recording unit 264 is performed during the manual insertion of the insertion unit 11, it is possible to generate a classification result recording unit 264 that is consistent with the classification result recording unit 262. Figures 13A-13C The illustrated graph is similar to a graph that represents the passage of time in terms of the types of insertion shapes of the insertion part 11 that accompany the user's operation.

[0189] Furthermore, when the aforementioned classification result recording unit 264 is activated during manual insertion of the insertion unit 11, data can be obtained, for example, for quantitative evaluation and / or analysis of the insertion operation of the insertion unit 11 performed by the user.

[0190] According to this embodiment, the operation of the classification result recording unit 264 for recording the classification results obtained by the insertion shape classification unit 262 in a time sequence and at fixed intervals is not limited to being performed when the insertion unit 11 is inserted into the subject body, but can also be configured to be performed when the insertion unit 11 inserted into the subject body is pulled out.

[0191] According to this embodiment, the information recorded in the classification result recording unit 264 can also be used to generate Figures 13A-13C Uses other than the illustrated graph.

[0192] Specifically, the information recorded in the classification result recording unit 264 can be used, for example, as metadata in analytical methods such as data mining and statistical analysis. Furthermore, the information recorded in the classification result recording unit 264 can be used, for example, to evaluate the user's proficiency in cases where the insertion unit 11 is manually inserted via user operation. Additionally, the information recorded in the classification result recording unit 264 can be used, for example, to estimate the insertion difficulty when inserting the insertion unit 11 into a specified subject.

[0193] According to this embodiment, the classification result recording unit 264 may also be configured to perform, for example, an operation to record the desired information obtained by the operation of the endoscope system 1, such as endoscopic images, in association with the classification result obtained by the insertion shape classification unit 262.

[0194] Alternatively, the insertion control unit 263 of this embodiment may be configured to, for example, set control content based on at least one of the following: an endoscope image output from the image processing unit 220, external force information output from the external force information acquisition device 40, and an insertion shape image generated by the insertion shape image generation unit 261, based on a detection result indicating whether the type of insertion shape of the insertion unit 11 shown as a classification result obtained by the insertion shape classification unit 262 has changed; use the set control content to generate an insertion control signal containing information for performing control related to the insertion operation of the insertion unit 11; and output the insertion control signal to the endoscope function control unit 240.

[0195] Specifically, the insertion control unit 263 may be configured, for example, to perform... Figure 14 The control shown is described below. An overview of such control will be provided below. Furthermore, for convenience, the following example will be used: Multiple insertion control messages, generated as information containing control content, are pre-stored in the storage medium 20M. This control content corresponds to multiple categories of insertion shapes classified by the insertion shape classification unit 262. From these multiple insertion control messages, one insertion control message corresponding to the classification result obtained by the insertion shape classification unit 262 is selected. It is also detected whether the type of insertion shape of the insertion unit 11 shown as the classification result has changed each time control is performed based on the insertion control unit 263. Figure 14 This is a flowchart illustrating the outline of the control performed in the endoscopic system of the modified example of the first embodiment.

[0196] The insertion control unit 263 performs the following processing: based on the classification result obtained by the insertion shape classification unit 262, it selects and reads from a plurality of insertion control information pre-stored in the storage medium 20M an insertion control information corresponding to a type of insertion shape shown as the classification result. Figure 14 Step S1).

[0197] The aforementioned plurality of insertion control information each includes any one of the following: information related to the method for making the insertion section 11 advanceable and information related to the method for releasing the predetermined insertion shape formed by the insertion section 11. Furthermore, the aforementioned plurality of insertion control information each includes information representing a single control content (control quantity, etc.) corresponding to the action of at least one of the control units included in the endoscope function control unit 240, i.e., at least one of the basic operations implemented by the function of the endoscope 10.

[0198] The information related to the method for making the insertion section 11 advanceable includes, for example, information indicating the setting conditions when setting the movement destination of the front end 12, such as the frame WG (described later) set in the processed result image PRG. Furthermore, the information related to the method for making the insertion section 11 advanceable includes, for example, at least one of the following: information indicating a basic operation performed individually during the advance of the insertion section 11, one of the basic operations implemented by the endoscope 10; and information indicating a combination of multiple basic operations performed consecutively or simultaneously during the advance of the insertion section 11.

[0199] Information relating to the method for releasing the prescribed insertion shape formed by the insertion part 11 may include, for example, at least one of the following: information indicating that a basic operation is performed individually when releasing the prescribed insertion shape in each of the basic operations implemented by the function of the endoscope 10; and information indicating a combination of multiple basic operations performed consecutively or simultaneously in each of the basic operations when releasing the prescribed insertion shape.

[0200] Insertion control unit 263 passed the test. Figure 14 Does the insertion control information read in step S1 contain information related to the method for making the insertion section 11 ready to move forward? Figure 14 Step S2).

[0201] Insertion control unit 263 after obtaining approval Figure 14 If the detection result of the insertion control information read in step S1 does not contain information related to the method for making the insertion unit 11 ready to move forward (S2: No), then proceed as described later. Figure 14 The processing of step S4.

[0202] Insertion control unit 263 after obtaining approval Figure 14 If the detection result of step S1 reading an insertion control message containing information related to a method for making the insertion section 11 advanceable is obtained (S2: Yes), based on the control content contained in the insertion control message and the endoscope image output from the image processing unit 220, the endoscope function control unit 240 performs control to make the insertion section 11 advanceable. Figure 14 Step S3).

[0203] The insertion control unit 263 generates an insertion control signal for primary control based on external force information output from the external force information acquisition device 40, and outputs it to the endoscope function control unit 240. This primary control corresponds to the insertion of an endoscope through an external force information acquisition device 40. Figure 14 Step S1 reads the control content contained in an insert control message and, as described later... Figure 14 Step S6 sets any one of the modified control contents ( Figure 14 (Step S4). Furthermore, a specific example of the aforementioned single-stage control will be discussed later.

[0204] Insertion control unit 263 is in operation Figure 14 The processing time of step S1 is determined by the classification result obtained by the insertion shape classification unit 262 and the time just performed. Figure 14 The classification result obtained by the insertion shape classification unit 262 is compared with the result of the first control in step S4 to detect whether the type of insertion shape of the insertion unit 11 has changed according to the first control. Figure 14 Step S5).

[0205] If the insertion control unit 263 detects a change in the type of insertion shape of the insertion unit 11 (S5: Yes), it will proceed as described later. Figure 14 The processing in step S8. Furthermore, if the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has not changed (S5: No), it determines whether a change is needed. Figure 14 Control content during the first control operation in step S4 ( Figure 14 Step S6).

[0206] Insertion control unit 263 determines that no change is needed. Figure 14 If the result of the determination of the control content during the first control in step S4 is (S6: No), the control content is maintained and the aforementioned process is performed. Figure 14 The control after step S2. Furthermore, the insertion control unit 263, upon receiving the required changes... Figure 14If the result of the determination of the control content when performing control once in step S4 (S6: Yes), then the processing for setting the control content after the change is performed. Figure 14 After step S7), the aforementioned procedure is performed. Figure 14 Control after step S2.

[0207] Insertion control unit 263 is based on the just performed Figure 14 At the moment after the first control in step S4, the classification result obtained by the insertion shape classification unit 262 is used to detect whether the insertion shape of the insertion unit 11 has changed to a specified type. Figure 14 Step S8).

[0208] If the insertion control unit 263 receives a detection result indicating that the insertion shape of the insertion unit 11 has not changed to the specified type (S8: No), the aforementioned process is performed. Figure 14 The control of step S1. Furthermore, if the insertion control unit 263 receives a detection result that the insertion shape change of the insertion unit 11 is of a specified type (S8: Yes), the series of controls for the endoscope function control unit 240 ends.

[0209] That is, according to Figure 14 The insertion control unit 263 is configured to set control content corresponding to a type of insertion shape shown as a classification result obtained by the insertion shape classification unit 262, and to perform insertion control related to the insertion operation of the insertion unit 11 one by one based on the set control content. Each time the insertion control is performed, the control content of the insertion control is determined by referring to the classification result obtained by the insertion shape classification unit 262.

[0210] In addition, according to Figure 14 The insertion control unit 263 is configured to select insertion control information CJX from a plurality of insertion control information pre-stored in the storage medium 20M. The insertion control information CJX contains control content corresponding to the insertion shape of the type TX shown as a classification result obtained by the insertion shape classification unit 262.

[0211] Furthermore, according to Figure 14 The insertion control unit 263 is configured to perform insertion control one step at a time based on the control content contained in the insertion control information CJX, through a series of processes. Furthermore, according to... Figure 14The insertion control unit 263 is configured to, when it detects that the type of the insertion shape of the insertion unit 11 shown as the classification result obtained by the insertion shape classification unit 262 after the aforementioned insertion control has just been performed has changed from type TX to type TY, select from a plurality of insertion control information pre-stored in the storage medium 20M an insertion control information containing control content corresponding to the insertion shape of type TY.

[0212] Furthermore, according to Figure 14 The insertion control unit 263 is configured to determine whether it is necessary to change the control content contained in the insertion control information CJX if it detects that the type of the insertion shape of the insertion unit 11 shown as the classification result obtained by the insertion shape classification unit 262 after the aforementioned insertion control has just been performed has not changed from the type TX.

[0213] In addition, Figure 14 In a series of processes, for example, the insertion control unit 263, through... Figure 14 If the processing in step S5 obtains a detection result indicating that the type of insertion shape of the insertion unit 11 has changed, and detects that the insertion control information corresponding to the type of insertion shape before the change can continue to be used for the type of insertion shape after the change, then skips the process. Figure 14 The process is carried out in step S1. Figure 14 The processing after step S2.

[0214] Next, the function of this modified example will be explained. Furthermore, thereafter, the aforementioned method will be applied to the insertion part 11 inserted into the intestinal tract from the anus into the large intestine. Figure 14 The control situation will be explained as a specific example. In addition, the control contents (control values, etc.) included in the various insertion control information described below show an example of inserting the insertion part 11 into the intestinal tract of the large intestine, and can therefore be appropriately changed according to the application site of the endoscope 10, etc.

[0215] After connecting the various parts of the endoscope system 1 and turning on the power, the doctor or other user may, for example, configure the insertion part 11 such that the front end 12 is located near the anus or rectum of the subject.

[0216] According to the aforementioned user operation, the illumination light supplied from the light source unit 210 is irradiated onto the subject, the subject irradiated by the illumination light is photographed by the imaging unit 110, and the endoscopic image obtained by photographing the subject is output from the image processing unit 220 to the display control unit 250 and the system control unit 260.

[0217] Furthermore, according to the aforementioned user operation, a coil drive signal is supplied from the coil drive signal generation unit 230, and a magnetic field is generated from the plurality of source coils 18 respectively according to the coil drive signal. The insertion shape information obtained by detecting the magnetic field is output from the insertion shape information acquisition unit 320 to the system control unit 260, and the insertion shape image corresponding to the insertion shape information is generated by the insertion shape image generation unit 261.

[0218] Furthermore, based on the aforementioned user operation, the external force information acquisition device 40 outputs external force information, representing the magnitude and direction of the external force at the respective positions of the multiple source coils 18, to the system control unit 260.

[0219] With the insertion unit 11 configured as described above, the user can, for example, instruct the main unit 20 to begin inserting the insertion unit 11 by turning on the automatic insertion switch of the input device 50.

[0220] When the classification result recording unit 264 detects an instruction to initiate insertion control for the insertion unit 11, it begins to record the classification results as the basis for the insertion control in a time sequence each time the insertion control unit 263 performs insertion control related to the insertion operation of the insertion unit 11 on the endoscope function control unit 240.

[0221] Furthermore, when the classification result recording unit 264 operates in this manner, a graph showing the time progression of the insertion shape type of the insertion unit 11 accompanying the control of the insertion control unit 263 can be generated, for example, to show the... Figures 13A-13C The graph shown has its horizontal axis changed from "time" to "number of control cycles".

[0222] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 3 In the case of the inserted shape image SGA shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TA.

[0223] When the insertion control unit 263 detects that the type of the insertion shape of the insertion unit 11 is type TA based on the classification result obtained by the insertion shape classification unit 262, it performs processing to select and read the insertion control information CJA corresponding to type TA from multiple insertion control information pre-stored in the storage medium 20M (equivalent to...). Figure 14 Step S1).

[0224] The aforementioned insertion control information CJA includes information related to the method for making the insertion section 11 capable of advancing. Furthermore, the aforementioned insertion control information CJA includes, for example, information indicating the control content for one operation, such as advancing the insertion section 11 under conditions of an advance amount of 50 mm, an advance speed of 30 mm per second, and a thrust of 2.0 N or less.

[0225] When the insertion control unit 263 detects that the insertion control information CJA contains information related to the method for making the insertion unit 11 ready to move forward (equivalent to S2: Yes), it performs processing to detect the position of the lumen region in the endoscope image based on the endoscope image output from the image processing unit 220.

[0226] Specifically, the insertion control unit 263, for example, inputs a classifier CLQ that has been learned with FCN (Fully Convolutional Neural Network). Figure 15A The endoscopic image EG shown is used to obtain... Figure 15B The processing result shown is the processing of the PRG image. Figure 15A This is a diagram illustrating an example of an endoscopic image generated in an endoscopic system according to an embodiment. Figure 15B It shows the... Figure 15A An example of a processed image obtained when an endoscopic image has undergone processing to detect the location of a lumen region.

[0227] When generating the aforementioned classifier CLQ, machine learning is performed, for example, using teaching data containing the same endoscope image as the endoscope image generated by the image processing unit 220; and labels indicating which part of the endoscope image, such as the edge, lumen, or other parts, each pixel in the endoscope image belongs to.

[0228] Therefore, based on the aforementioned classifier CLQ, multidimensional data such as the pixel values ​​of each pixel contained in the endoscopic image generated by the image processing unit 220 is obtained, and this multidimensional data is input to the input layer of the neural network. As a result, a processed image PRG that can determine the positions of edge regions and lumen regions in the endoscopic image can be obtained as output data. That is, the processed image obtained by using the aforementioned classifier CLQ contains region segmentation results equivalent to semantic segmentation.

[0229] Alternatively, according to this modified example, if the insertion control unit 263 determines that it is difficult to determine the lumen region based on the processed result image PRG, it may generate an insertion control signal and output it to the endoscope function control unit 240. This insertion control signal is used to cause the AWS control unit 243 to operate to perform air delivery, water delivery and / or suction based on the AWS mechanism 143.

[0230] The insertion control unit 263 generates an insertion control signal based on the control content contained in the insertion control information CJA and outputs it to the endoscope function control unit 240. This insertion control signal is used to perform an action that causes the lumen region detected from the endoscope image output from the image processing unit 220 to enter a predetermined area including the central portion of the endoscope image (equivalent to...). Figure 14 Step S3).

[0231] Specifically, the insertion control unit 263 generates an insertion control signal based on the control content contained in the insertion control information CJA and outputs it to the endoscope function control unit 240. This insertion control signal is used for, for example, Figure 15C As shown Figure 15B The processed result image PRG is divided into a 9×9 region, and the orientation of the front end 12 and / or the rotation angle of the insertion part 11 are adjusted so that the entire or substantially the entire lumen region contained in the processed result image PRG is incorporated into the 5×5 region containing the central part of the processed result image PRG. Figure 15C (The inside of the box WG in the middle).

[0232] Then, with the control of the insertion control unit 263, at least one of the following is controlled: the bending control unit 242 bends the bending portion 13 via the bending mechanism 142, and the rotation control unit 244 rotates the insertion portion 11 via the rotation mechanism 144. Furthermore, with the control of the aforementioned insertion control unit 263, the insertion portion 11 becomes a forward-moving state. Figure 15C It is used to explain in order to obtain Figure 15B The control diagram is shown in the case of the processed image.

[0233] During the period from when the insertion control unit 263 enters the designated area including the central portion of the endoscopic image output from the image processing unit 220, the endoscope function control unit 240 is repeatedly controlled.

[0234] Furthermore, when the insertion control unit 263 detects that the lumen region has entered a predetermined area including the central portion of the endoscopic image output from the image processing unit 220, it generates an insertion control signal based on the external force information output from the external force information acquisition device 40, for performing a single control corresponding to the control content contained in the insertion control information CJA, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Step S4). Then, under the control of such insertion control unit 263, the advance control unit 241 controls the insertion unit 11 to advance via the advance mechanism 141.

[0235] Based on the classification result obtained by the insertion shape classification unit 262, the insertion control unit 263 detects whether the type of insertion shape of the insertion unit 11 has changed from type TA due to a single control based on the control content contained in the insertion control information CJA (equivalent to...). Figure 14 Step S5).

[0236] If the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has not changed from type TA (equivalent to S5: No), it then determines whether it is necessary to change the control content contained in the insertion control information CJA (equivalent to...). Figure 14 Step S6).

[0237] For example, if the insertion control unit 263 detects that the external force applied when the insertion unit 11 moves forward is less than 2.0N based on the external force information output from the external force information acquisition device 40, it obtains a determination result that the control content contained in the insertion control information CJA does not need to be changed (equivalent to S6: No).

[0238] Furthermore, when the insertion control unit 263 receives a determination result indicating that the control content contained in the insertion control information CJA does not need to be changed, after performing control to make the insertion unit 11 ready to advance, it generates an insertion control signal for performing a single control corresponding to the control content contained in the insertion control information CJA, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2, S3 and S4).

[0239] For example, if the insertion control unit 263 detects that the external force applied when the insertion unit 11 moves forward exceeds 2.0N based on the external force information output from the external force information acquisition device 40, it obtains a determination result (equivalent to S6: Yes) that the control content contained in the insertion control information CJA needs to be changed.

[0240] Furthermore, when the insertion control unit 263 receives a determination result indicating that the control content contained in the insertion control information CJA needs to be changed, it sets the control content containing the insertion control information CJA with added jiggling as the changed control content (equivalent to...). Figure 14 Step S7). Then, after performing control to make the insertion unit 11 ready to move forward, the insertion control unit 263 generates an insertion control signal for performing a single control corresponding to the aforementioned (including jitter) changed control content, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2, S3 and S4).

[0241] Furthermore, when manually inserting the insertion part 11, the aforementioned shaking is performed by moving the insertion part 11, inserted into the intestinal tract, back and forth little by little. In addition, the aforementioned shaking is performed for example to remove or reduce phenomena that hinder manual insertion of the insertion part 11 into the large intestine, such as deflection of the insertion part 11, friction between the intestinal tract and the insertion part 11, and hooking of the insertion part 11 within the intestinal tract. Therefore, for example, the advance / retreat control unit 241 generates an advance / retreat control signal and outputs it to the advance / retreat mechanism 141, thereby enabling an operation equivalent to the aforementioned shaking, wherein the advance / retreat control signal is used to control the insertion part 11 to repeatedly advance and retreat a predetermined number of times.

[0242] Then, when jitter is added to the control content contained in the insertion control information CJA, as a first control, the insertion control unit 263 controls the insertion unit 11 to jitter and advance according to the control content contained in the insertion control information CJA.

[0243] Furthermore, according to this modified example, for example, if it is detected that even if a predetermined number of controls are performed while jittering is added to the control content included in the insertion control information CJA, and the type of insertion shape of the insertion part 11 does not change from type TA, the insertion control unit 263 can control the insertion part 11 to jitter and move forward after retracting the insertion part 11 by a fixed amount.

[0244] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 4A The inserted shape image SGB1 or shown Figure 4B In the case of the inserted shape image SGB2 shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TB.

[0245] When the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has changed from type TA to type TB (equivalent to S5: Yes and S8: No), it performs processing to select and read the insertion control information CJB corresponding to type TB from a plurality of insertion control information pre-stored in the storage medium 20M (equivalent to... Figure 14 Step S1).

[0246] The aforementioned insertion control information CJB includes information related to the method for making the insertion part 11 capable of advancing. Furthermore, the aforementioned insertion control information CJB includes, for example, information indicating the content of a single control operation, stating that after detecting that the external force applied when advancing the insertion part 11 by jitter is 3.0 N or less, the insertion part 11 is advanced under the conditions of an advance amount of 20 mm, an advance speed of 10 mm per second, and a propulsion force of 3.0 N or less.

[0247] When the insertion control unit 263 detects that the insertion control information CJB contains information related to the method for making the insertion unit 11 advanceable (equivalent to S2: Yes), it performs processing to detect the position of the lumen region in the endoscope image based on the endoscope image output from the image processing unit 220.

[0248] Furthermore, the insertion control unit 263 generates an insertion control signal based on the control content contained in the insertion control information CJB and outputs it to the endoscope function control unit 240. This insertion control signal is used to perform an action that causes the lumen region detected by the aforementioned processing to enter a predetermined area including the central portion of the endoscope image (equivalent to...). Figure 14 Step S3).

[0249] Specifically, the insertion control unit 263 generates an insertion control signal based on the control content contained in the insertion control information CJB and outputs it to the endoscope function control unit 240. This insertion control signal is used to obtain... Figure 15B The processed result image PRG illustrated in the figure is divided into a 9×9 region, and the orientation of the front end 12 and / or the rotation angle of the insertion part 11 are adjusted so that the entire or substantially the entire region of the lumen region contained in the processed result image PRG enters the 7×7 region containing the central part of the processed result image PRG.

[0250] Then, with the control of the insertion control unit 263, at least one of the following is controlled: the bending control unit 242 bends the bending portion 13 via the bending mechanism 142, and the rotation control unit 244 rotates the insertion portion 11 via the rotation mechanism 144. Furthermore, with the control of the aforementioned insertion control unit 263, the insertion portion 11 becomes a forward-moving state.

[0251] During the period from when the lumen region enters the predetermined area including the central portion of the endoscopic image output from the image processing unit 220, the insertion control unit 263 repeatedly controls the endoscope function control unit 240. Furthermore, when the insertion control unit 263 detects that the lumen region has entered the predetermined area including the central portion of the endoscopic image output from the image processing unit 220, it generates an insertion control signal based on the external force information output from the external force information acquisition device 40, for performing a single control corresponding to the control content contained in the insertion control information CJB, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Step S4).

[0252] Then, under the control of the insertion control unit 263, the advance control unit 241 sequentially performs control for shaking the insertion part 11 via the advance mechanism 141 and control for advancing the insertion part 11 via the advance mechanism 141.

[0253] Based on the classification result obtained by the insertion shape classification unit 262, the insertion control unit 263 detects whether the type of insertion shape of the insertion unit 11 has changed from type TB due to a single control based on the control content contained in the insertion control information CJB (equivalent to...). Figure 14 Step S5).

[0254] If the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has not changed from type TB (equivalent to S5: No), it then determines whether it is necessary to change the control content contained in the insertion control information CJB (equivalent to...). Figure 14 Step S6).

[0255] During the period when the insertion control unit 263 is performing control corresponding to the insertion control information CJB read from the storage medium 20M, it obtains a determination result that the control content contained in the insertion control information CJB does not need to be changed (equivalent to S6: No).

[0256] Furthermore, when the insertion control unit 263 determines that it does not need to change the control content contained in the insertion control information CJB, after performing control to make the insertion unit 11 ready to move forward, it generates an insertion control signal for performing a single control corresponding to the control content contained in the insertion control information CJB, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2, S3 and S4).

[0257] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 5A The inserted shape image SGC1 or shown Figure 5B In the case of the inserted shape image SGC2 shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TC.

[0258] When the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has changed from type TB to type TC (equivalent to S5: Yes and S8: No), it again performs control corresponding to the insertion control information CJB read from the storage medium 20M. That is, when the insertion control unit 263 detects that the type of insertion shape of the insertion unit 11 has changed from type TB to type TC, it determines that the insertion control information CJB corresponding to type TB can continue to be used, and skips... Figure 14 The process is carried out in step S1. Figure 14 The processing after step S2.

[0259] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 6A The inserted shape image SGD1 or shown Figure 6B In the case of the inserted shape image SGD2 shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TD.

[0260] When the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has changed from type TC to type TD (equivalent to S5: Yes and S8: No), it performs processing to select and read the insertion control information CJD corresponding to the type TD from a plurality of insertion control information pre-stored in the storage medium 20M (equivalent to... Figure 14 Step S1).

[0261] The aforementioned insertion control information CJD includes information related to the method for making the insertion part 11 capable of advancing. Furthermore, the aforementioned insertion control information CJD includes, for example, information indicating the content of one control operation, stating that after detecting that the external force applied when advancing the insertion part 11 by jitter is 2.5N or less, the insertion part 11 is advanced under the conditions of an advance amount of 20mm, an advance speed of 20mm per second, and a propulsion force of 2.5N or less.

[0262] When the insertion control unit 263 detects that the insertion control information CJD contains information related to the method for making the insertion unit 11 advanceable (equivalent to S2: Yes), it performs processing to detect the position of the lumen region in the endoscope image based on the endoscope image output from the image processing unit 220.

[0263] Furthermore, based on the control content contained in the insertion control information CJD, the insertion control unit 263 generates an insertion control signal and outputs it to the endoscope function control unit 240. This insertion control signal is used to perform an action that causes the lumen region detected by the aforementioned processing to enter a predetermined area including the central portion of the endoscope image (equivalent to...). Figure 14 Step S3).

[0264] Specifically, the insertion control unit 263 generates an insertion control signal based on the control content contained in the insertion control information CJD and outputs it to the endoscope function control unit 240. This insertion control signal is used to obtain... Figure 15B The illustrated processing result image PRG is divided into a 9×9 region, and the orientation of the front end 12 and / or the rotation angle of the insertion part 11 are adjusted so that the entire or substantially the entire region of the lumen region contained in the processing result image PRG enters the 5×5 region containing the central part of the processing result image PRG.

[0265] Furthermore, if the insertion control unit 263 detects that even when the bending angle of the bending portion 13 reaches its maximum value, the insertion portion 11 is rotated but the lumen region still does not enter the aforementioned 5×5 area, the insertion control unit 263 generates an insertion control signal and outputs it to the endoscope function control unit 240. This insertion control signal is used to adjust the orientation of the front end portion 12 and / or the rotation angle of the insertion portion 11, so that the entire or substantially the entire lumen region contained in the processed result image PRG enters the 7×7 area containing the central portion of the processed result image PRG.

[0266] Furthermore, with the control of the insertion control unit 263, at least one of the following is controlled: the bending control unit 242 bends the bending portion 13 via the bending mechanism 142, and the rotation control unit 244 rotates the insertion portion 11 via the rotation mechanism 144. Additionally, with the control of the aforementioned insertion control unit 263, the insertion portion 11 becomes capable of advancing.

[0267] During the period when the insertion control unit 263 enters the lumen region within a predetermined area including the central portion of the endoscopic image output from the image processing unit 220, the endoscope function control unit 240 is repeatedly controlled.

[0268] Furthermore, when the insertion control unit 263 detects that the lumen region has entered a predetermined area including the central portion of the endoscopic image output from the image processing unit 220, it generates an insertion control signal based on the external force information output from the external force information acquisition device 40. This signal is used to perform a single control corresponding to the control content contained in the insertion control information CJD and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Step S4).

[0269] Then, under the control of the insertion control unit 263, the advance control unit 241 sequentially performs control for shaking the insertion part 11 via the advance mechanism 141 and control for advancing the insertion part 11 via the advance mechanism 141.

[0270] Based on the classification result obtained by the insertion shape classification unit 262, the insertion control unit 263 detects whether the type of insertion shape of the insertion unit 11 has changed from type TD due to a single control based on the control content contained in the insertion control information CJD (equivalent to...). Figure 14 Step S5).

[0271] If the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has not changed from the type TD (equivalent to S5: No), it further determines whether it is necessary to change the control content contained in the insertion control information CJD (equivalent to Figure 14 Step S6).

[0272] During the period when the insertion control unit 263 is performing control corresponding to the insertion control information CJD read from the storage medium 20M, it obtains a determination result that the control content contained in the insertion control information CJD does not need to be changed (equivalent to S6: No).

[0273] Furthermore, when the insertion control unit 263 determines that it does not need to change the control content contained in the insertion control information CJD, after performing control to make the insertion unit 11 ready to move forward, it generates an insertion control signal for performing a single control corresponding to the control content contained in the insertion control information CJD, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2, S3 and S4).

[0274] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 7A The inserted shape image SGE1 or shown Figure 7B In the case of the insertion shape image SGE2 shown, a classification result is obtained in which the insertion shape of the insertion part 11 is classified into category TE.

[0275] When the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has changed from type TD to type TE (equivalent to S5: Yes and S8: No), it performs processing to select and read the insertion control information CJE corresponding to type TE from a plurality of insertion control information pre-stored in the storage medium 20M (equivalent to... Figure 14 Step S1).

[0276] The aforementioned insertion control information CJE includes information related to the method for releasing the ring shape formed by the insertion part 11. Furthermore, the aforementioned insertion control information CJE includes, for example, information indicating the amount of retraction BLA when the insertion part 11 retracts, and information indicating the retraction speed BVA when the insertion part 11 retracts.

[0277] When the insertion control unit 263 receives information in the insertion control information CJE related to a method for releasing the ring shape formed by the insertion unit 11 (equivalent to S2: no), it generates an insertion control signal for performing a single control corresponding to the retraction amount BLA and retraction speed BVA contained in the insertion control information CJE, and outputs it to the endoscope function control unit 240 (equivalent to S2: no). Figure 14 Step S4). Then, under the control of such insertion control unit 263, the advance and retreat control unit 241 performs control to retract the insertion unit 11 via the advance and retreat mechanism 141.

[0278] Based on the classification result obtained by the insertion shape classification unit 262, the insertion control unit 263 detects whether the type of insertion shape of the insertion unit 11 has changed from type TE due to a single control based on the control content contained in the insertion control information CJE (equivalent to...). Figure 14 Step S5).

[0279] If the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has not changed from type TE (equivalent to S5: No), it further determines whether it is necessary to change the control content contained in the insertion control information CJE (equivalent to...). Figure 14 Step S6).

[0280] For example, if the insertion control unit 263 detects that the release (shrinkage) of the α ring is progressing based on the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, it obtains a determination result that the control content contained in the insertion control information CJE does not need to be changed (equivalent to S6: No).

[0281] Furthermore, when the insertion control unit 263 receives a determination result indicating that the control content contained in the insertion control information CJE does not need to be changed, it generates an insertion control signal for performing a single control corresponding to the control content contained in the insertion control information CJE, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2 and S4).

[0282] For example, if the insertion control unit 263 detects that the release (shrinkage) of the α ring has not progressed based on the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, it obtains a determination result (equivalent to S6: Yes) that the control content contained in the insertion control information CJE needs to be changed.

[0283] Furthermore, when the insertion control unit 263 receives a determination result indicating that the control content contained in the insertion control information CJE needs to be changed, for example, it sets the control content after changing at least one of the parameters of the retraction amount BLA and the retraction speed BVA contained in the insertion control information CJE as the changed control content (equivalent to...). Figure 14 Step S7).

[0284] Alternatively, if the insertion control unit 263 receives a determination result indicating that the control content contained in the insertion control information CJE needs to be changed, for example, it sets the control content containing other types of control content related to the insertion operation of the insertion unit 11 after adding them to the control content contained in the insertion control information CJE as the changed control content (equivalent to...). Figure 14 Step S7).

[0285] Then, the insertion control unit 263 generates an insertion control signal for performing a single control corresponding to the aforementioned modified control content, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2 and S4).

[0286] Furthermore, the insertion control unit 263 of this modified example is not limited to starting control corresponding to the insertion control information CJE immediately after receiving the detection result that the type of insertion shape of the insertion unit 11 changes from type TD to type TE. For example, it may start control corresponding to the insertion control information CJE when the detection result that the type of insertion shape of the insertion unit 11 changes from type TD to type TE is received within a continuous period of 1 minute.

[0287] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 8A The inserted shape image SGF1 or shown Figure 8B In the case of the inserted shape image SGF2 shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TF.

[0288] When the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has changed from type TE to type TF (equivalent to S5: Yes and S8: No), it performs processing to select and read the insertion control information CJF corresponding to the type TF from a plurality of insertion control information pre-stored in the storage medium 20M (equivalent to... Figure 14 Step S1).

[0289] The aforementioned insertion control information CJF includes information related to the method for releasing the ring shape formed by the insertion part 11. Furthermore, the aforementioned insertion control information CJF includes, for example, information indicating the control content for one operation, information indicating the rotation angle BAA when the insertion part 11 is rotated to the right about the insertion axis (length axis).

[0290] When the insertion control unit 263 detects that the insertion control information CJF contains information related to a method for releasing the ring shape formed by the insertion unit 11 (equivalent to S2: No), it generates an insertion control signal for performing a single control corresponding to the rotation angle BAA contained in the insertion control information CJF, and outputs it to the endoscope function control unit 240 (equivalent to S2: No). Figure 14 Step S4). Then, under the control of the insertion control unit 263, the rotation control unit 244 controls the insertion part 11 to rotate to the right about the insertion axis (length axis) via the rotation mechanism 144.

[0291] Based on the classification result obtained by the insertion shape classification unit 262, the insertion control unit 263 detects whether the type of insertion shape of the insertion unit 11 has changed from type TF due to a single control based on the control content contained in the insertion control information CJF (equivalent to...). Figure 14 Step S5).

[0292] If the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has not changed from type TF (equivalent to S5: No), it further determines whether it is necessary to change the control content contained in the insertion control information CJF (equivalent to...). Figure 14 Step S6).

[0293] For example, if the insertion control unit 263 detects that the release (shrinkage) of the α ring is progressing based on the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, it obtains a determination result that the control content contained in the insertion control information CJF does not need to be changed (equivalent to S6: No).

[0294] Furthermore, when the insertion control unit 263 determines that it does not need to change the control content contained in the insertion control information CJF, it generates an insertion control signal for performing a single control corresponding to the control content contained in the insertion control information CJF, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2 and S4).

[0295] For example, if the insertion control unit 263 detects that the release (shrinkage) of the α ring has not progressed based on the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, it obtains a determination result (equivalent to S6: Yes) that the control content contained in the insertion control information CJF needs to be changed.

[0296] Furthermore, when the insertion control unit 263 receives a determination result indicating that the control content contained in the insertion control information CJF needs to be changed, for example, it sets the control content that was added after the action of retracting the insertion unit 11 by a predetermined amount before rotating the insertion unit 11 by the rotation angle BAA as the changed control content (equivalent to...). Figure 14 Step S7).

[0297] Then, the insertion control unit 263 generates an insertion control signal for performing a single control corresponding to the aforementioned modified control content, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2 and S4).

[0298] Furthermore, according to this modified example, for example, if it is detected that even after a predetermined number of control operations are performed while the control content contained in the insertion control information CJF has been changed, the release (shrinkage) of the α ring has not progressed, the insertion control unit 263 can also control the insertion unit 11 to advance while maintaining the formed α ring.

[0299] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 9A The inserted shape image SGG1 or shown Figure 9B In the case of the inserted shape image SGG2 shown, a classification result is obtained in which the inserted shape of the inserted part 11 is separated into the category TG.

[0300] When the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has changed from type TF to type TG (equivalent to S5: Yes and S8: No), it performs processing to select and read the insertion control information CJG corresponding to type TG from a plurality of insertion control information pre-stored in the storage medium 20M (equivalent to... Figure 14 Step S1).

[0301] The aforementioned insertion control information CJG includes information related to the method for releasing the ring shape formed by the insertion part 11. Furthermore, the aforementioned insertion control information CJG includes, for example, information indicating the control action, information such as the retraction amount BLB when the insertion part 11 retracts, the retraction speed BVB when the insertion part 11 retracts, and the rotation angle BAB when the insertion part 11 rotates to the right around the insertion axis (length axis). The retraction speed BVB can be set, for example, to a speed of approximately 15 mm per second.

[0302] When the insertion control unit 263 detects that the insertion control information CJG contains information related to a method for releasing the ring shape formed by the insertion unit 11 (equivalent to S2: No), it generates an insertion control signal for performing a single control corresponding to the retraction amount BLB, retraction speed BVB, and rotation angle BAB contained in the insertion control information CJG, and outputs it to the endoscope function control unit 240 (equivalent to S2: No). Figure 14 Step S4).

[0303] Then, according to the control of the insertion control unit 263, the control for the advance control unit 241 to retract the insertion part 11 via the advance mechanism 141 is performed simultaneously, and the control for the rotation control unit 244 to rotate the insertion part 11 to the right around the insertion axis (length axis) via the rotation mechanism 144 is performed simultaneously.

[0304] Based on the classification result obtained by the insertion shape classification unit 262, the insertion control unit 263 detects whether the type of insertion shape of the insertion unit 11 has changed from type TG due to a single control based on the control content contained in the insertion control information CJG (equivalent to...). Figure 14 Step S5).

[0305] If the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has not changed from type TG (equivalent to S5: No), it further determines whether it is necessary to change the control content contained in the insertion control information CJG (equivalent to Figure 14 Step S6).

[0306] During the period when the insertion control unit 263 is performing control corresponding to the insertion control information CJG read from the storage medium 20M, it obtains a determination result that the control content contained in the insertion control information CJG does not need to be changed (equivalent to S6: No).

[0307] Furthermore, when the insertion control unit 263 determines that it does not need to change the control content contained in the insertion control information CJG, it generates an insertion control signal for performing a single control corresponding to the control content contained in the insertion control information CJG, and outputs it to the endoscope function control unit 240 (equivalent to...). Figure 14 Steps S2 and S4).

[0308] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 10 In the case of the inserted shape image SGH shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TH.

[0309] When the insertion control unit 263 receives a detection result indicating that the type of insertion shape of the insertion unit 11 has changed from type TG to type TH (equivalent to S5: Yes and S8: No), it performs processing to select and read the insertion control information CJA corresponding to type TH from a plurality of insertion control information pre-stored in the storage medium 20M (equivalent to... Figure 14 Step S1). Furthermore, regarding the control performed by the insertion control unit 263 when the type of insertion shape in the insertion unit 11 is type TH, since it can be used for the control already described corresponding to the control content contained in the insertion control information CJA, a detailed explanation is omitted.

[0310] In this embodiment, if the insertion part 11 passes through the S-shaped colon without forming an α-ring, the insertion shape type of the insertion part 11 is maintained as type TA based on the classification result of the insertion shape classification unit 262. The insertion control unit 263 then continues to perform control corresponding to the control content contained in the insertion control information CJA corresponding to that type TA. Therefore, if the insertion control unit 263 receives a detection result indicating that the insertion shape type of the insertion part 11 has changed from type TA to type TH (equivalent to S5: Yes and S8: No), it again performs control corresponding to the insertion control information CJA read from the storage medium 20M.

[0311] That is, when the insertion control unit 263 detects that the type of the insertion shape of the insertion unit 11 changes from type TA to type TH, it determines that the insertion control information CJA corresponding to type TA can continue to be used, and skips the insertion control unit 263. Figure 14 The process is carried out in step S1. Figure 14 The processing after step S2.

[0312] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 11A The inserted shape image SGI1 or shown Figure 11BIn the case of the inserted shape image SGI2 shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TI.

[0313] When the insertion control unit 263 receives a detection result that the type of insertion shape of the insertion unit 11 changes from type TH to type TI (equivalent to S5: yes and S8: no), it performs control again corresponding to the insertion control information CJA read from the storage medium 20M.

[0314] That is, when the insertion control unit 263 detects that the type of the insertion shape of the insertion unit 11 changes from type TH to type TI, it determines that the insertion control information CJA corresponding to type TH can continue to be used, and skips the insertion control unit 263. Figure 14 The process is carried out in step S1. Figure 14 The processing after step S2.

[0315] The insertion shape classification unit 262, for example, generates a shape image through the insertion shape image generation unit 261. Figure 12A The inserted shape image SGJ1 or shown Figure 12B In the case of the inserted shape image SGJ2 shown, a classification result is obtained in which the inserted shape of the inserted part 11 is classified into category TJ.

[0316] When the insertion control unit 263 receives a detection result that the type of insertion shape of the insertion unit 11 changes from type TI to type TJ (equivalent to S5: Yes and S8: Yes), it ends a series of controls on the endoscope function control unit 240.

[0317] For example, after the user confirms, based on the insertion shape image displayed on the display device 60, that the insertion shape of the insertion part 11 inserted inside the subject no longer changes, the user turns off the automatic insertion switch of the input device 50, thereby giving an instruction to stop the main device 20 from controlling the insertion of the insertion part 11.

[0318] When the classification result recording unit 264 detects an instruction to stop the insertion control of the insertion stop unit 11, it stops the operation of recording the classification results of the insertion shape classification unit 262 in a time sequence.

[0319] As described above, according to this modified example, the insertion shape classification unit 262 performs the following processing: based on a viewpoint roughly equivalent to that of a skilled person making a subjective judgment or evaluation of the success or failure of the insertion operation of the insertion unit 11, the types of insertion shapes of the insertion unit 11 contained in the insertion shape image generated by the insertion shape image generation unit 261 are classified to obtain a classification result. Furthermore, as described above, according to this modified example, the insertion control unit 263 performs insertion control based on insertion control information corresponding to the types of insertion shapes of the insertion unit 11 shown as the classification result obtained by the insertion shape classification unit 262.

[0320] Furthermore, as described above, according to this modified example, each time insertion control is performed based on insertion control information corresponding to the type of insertion shape of the insertion part 11, the insertion control unit 263 performs an operation to detect whether the type of insertion shape of the insertion part 11 has changed. Therefore, according to this modified example, for example, appropriate insertion control can be performed corresponding to the insertion status of the insertion part, such as individual differences in the internal state of the subject to which the insertion part is inserted and changes in the insertion shape of the insertion part inside the subject over time.

[0321] Furthermore, this modified example allows for the application of controls related to the release of the back α ring and the reverse α ring by replacing the rotation angles contained in the insertion control information CJF and CJG with the angles at which the insertion part 11 rotates to the left about the insertion axis (length axis).

[0322] Furthermore, this modified example changes... Figure 14 This is part of a series of controls, such as the ability to apply controls related to the removal of various insertion shapes that obstruct the insertion of the insertion part 11 into the large intestine, such as rods and γ-rings.

[0323] (Second Implementation)

[0324] Figures 16 to 17D This is a diagram of the second embodiment.

[0325] Furthermore, in this embodiment, detailed descriptions of parts having the same structure as in the first embodiment are appropriately omitted, and descriptions are mainly given of parts having a structure different from that in the first embodiment.

[0326] For example, Figure 16 As shown, the endoscope system 1A is configured to include an endoscope 10, a main body device 20A, an insertion shape detection device 30, an external force information acquisition device 40, an input device 50, and a display device 60. Figure 16 This is a block diagram illustrating the specific structure of the endoscope system according to the second embodiment.

[0327] The main unit 20A is configured to have one or more processors 20P and storage media 20M. Furthermore, as... Figure 16 As shown, the main device 20A is configured to include a light source unit 210, an image processing unit 220, a coil drive signal generation unit 230, an endoscope function control unit 240, a display control unit 250, and a system control unit 270.

[0328] The system control unit 270 is configured to generate and output system control signals for performing actions corresponding to instructions from the operation unit 16 and the input device 50. Furthermore, the system control unit 270 is configured to include an insertion shape image generation unit 261, an insertion shape element extraction unit 272, an insertion control unit 273, and an extraction result recording unit 274.

[0329] The insertion shape element extraction unit 272 is configured to perform the following process: extract one or more structural elements associated with the insertion shape of the insertion unit 11 from the insertion shape image generated by the insertion shape image generation unit 261, and obtain the extraction result.

[0330] <Specific example of the structure of the Insert Shape Element Extraction Unit 272>

[0331] Here, a specific example of the structure of the insertion shape element extraction unit 272 in this embodiment will be described.

[0332] The insertion shape feature extraction unit 272 is configured to process the image using a fully learned classifier (e.g., classifier CLR) with an FCN (Fully Convolutional Neural Network) to obtain an extraction result of one or more structural features associated with the insertion shape of the insertion unit 11 extracted from the insertion shape image generated by the insertion shape image generation unit 261.

[0333] Here, when generating the aforementioned classifier CLR, machine learning is performed, for example, using teaching data. This teaching data includes the same insertion shape image as the insertion shape image generated by the insertion shape image generation unit 261, and a label indicating which structural element each pixel in the insertion shape image belongs to among the following: the endoscope tip (hereinafter referred to as structural element E1), a relatively large closed loop (hereinafter referred to as structural element E2), a relatively small closed loop (hereinafter referred to as structural element E3), an open loop (hereinafter referred to as structural element E4), a cross in a closed loop (hereinafter referred to as structural element E5), a bend on the base side of an N-ring (hereinafter referred to as structural element E6), a bend on the front side of an N-ring (hereinafter referred to as structural element E7), the interior of a closed loop (hereinafter referred to as structural element E8), the portion of the endoscope insertion part other than structural elements E1 to E8 (hereinafter referred to as structural element E9), and the background (hereinafter referred to as structural element E10).

[0334] Here, for example, the presence or absence of the aforementioned structural elements E1 to E10 is determined by the judgment of a skilled person who has visually confirmed the inserted shape image used as teaching data.

[0335] Furthermore, regarding the closed loop corresponding to the aforementioned structural element E3, it can be represented, for example, as a loop of such size that a skilled person can attempt to release the loop by twisting the insertion part 11.

[0336] Furthermore, the closed loop corresponding to the aforementioned structural element E2 is represented as a loop with a larger size than the aforementioned structural element E3.

[0337] Furthermore, the aforementioned structural elements E1 to E8 are set as follows: the structural elements are used to extract local regions from an insertion shape image containing the insertion shape of the insertion part 11. These local regions help to determine whether the insertion operation of the insertion part 11 is successful when performed manually or automatically, and whether the operation content needs to be changed.

[0338] Therefore, according to the aforementioned classifier CLR, for example, multidimensional data such as the pixel values ​​of each pixel contained in the inserted shape image generated by the inserted shape image generation unit 261 is obtained, and this multidimensional data is input as input data to the input layer of the neural network. Thus, a processing result image representing the classification result of the pixels contained in the inserted shape image into any one of the aforementioned structural elements E1 to E10 can be obtained as output data. That is, the processing result image obtained by using the aforementioned classifier CLR contains a region segmentation result equivalent to semantic segmentation.

[0339] For example, if the insertion shape element extraction unit 272 generates an insertion shape image containing insertion shapes classified as type TB by the insertion shape image generation unit 261, it processes the insertion shape image by inputting it into the classifier CLR to obtain... Figure 17A The processed image PBG is shown. Figure 17A This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment.

[0340] Figure 17AThe processed image PBG is generated as an image containing the following region segmentation results, which indicate that the inserted shape image generated by the inserted shape image generation unit 261 is segmented into four regions: region EA1 with a pixel group classified as structural element E1, region EA4 with a pixel group classified as structural element E4, region EA9 with a pixel group classified as structural element E9, and region EA10 with a pixel group classified as structural element E10.

[0341] Right now, Figure 17A The processing result image PBG is obtained as an image representing the following extraction result: three structural elements corresponding to structural elements E1, E4 and E9 are extracted from the insertion shape image generated by the insertion shape image generation unit 261 and used as structural elements associated with the insertion shape of the insertion unit 11.

[0342] For example, if the insertion shape element extraction unit 272 generates an insertion shape image containing insertion shapes classified as type TE by the insertion shape image generation unit 261, the insertion shape image is input to the classifier CLR and processed to obtain... Figure 17B The processed result image shown is PEG. Figure 17B This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment.

[0343] Figure 17B The resulting image PEG is generated as an image containing the following region segmentation results, which indicate that the inserted shape image generated by the inserted shape image generation unit 261 is segmented into six regions: region EA1 with a pixel group classified as structural element E1, region EA2 with a pixel group classified as structural element E2, region EA5 with a pixel group classified as structural element E5, region EA8 with a pixel group classified as structural element E8, region EA9 with a pixel group classified as structural element E9, and region EA10 with a pixel group classified as structural element E10.

[0344] Right now, Figure 17B The processing result image PEG is obtained as an image representing the following extraction result, which is that five structural elements corresponding to structural elements E1, E2, E5, E8 and E9 are extracted from the insertion shape image generated by the insertion shape image generation unit 261 as structural elements associated with the insertion shape of the insertion unit 11.

[0345] For example, if the insertion shape image generation unit 261 generates an insertion shape image containing insertion shapes classified as type TF by the insertion shape classification unit 262, the insertion shape element extraction unit 272 inputs the insertion shape image to the classifier CLR and processes it to obtain... Figure 17C The processed result image PFG is shown. Figure 17C This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment.

[0346] Figure 17C The processed image PFG is generated as an image containing the following region segmentation results, which indicate that the inserted shape image generated by the inserted shape image generation unit 261 is segmented into six regions: region EA1 with a pixel group classified as structural element E1, region EA3 with a pixel group classified as structural element E3, region EA5 with a pixel group classified as structural element E5, region EA8 with a pixel group classified as structural element E8, region EA9 with a pixel group classified as structural element E9, and region EA10 with a pixel group classified as structural element E10.

[0347] Right now, Figure 17C The processing result image PFG is obtained as an image representing the following extraction result: five structural elements corresponding to structural elements E1, E3, E5, E8 and E9 are extracted from the insertion shape image generated by the insertion shape image generation unit 261 and used as structural elements associated with the insertion shape of the insertion unit 11.

[0348] For example, if the insertion shape image generation unit 261 generates an insertion shape image containing insertion shapes classified as type TG by the insertion shape classification unit 262, the insertion shape element extraction unit 272 inputs the insertion shape image to the classifier CLR and processes it to obtain... Figure 17D The processed image shown is PGG. Figure 17D This is an example of an image showing the extraction result obtained by extracting structural elements associated with the insertion shape of the insertion part from an insertion shape image generated by the endoscope system of the second embodiment.

[0349] Figure 17DThe processed image PGG is generated as an image containing the following region segmentation results, which indicate that the inserted shape image generated by the inserted shape image generation unit 261 is segmented into five regions: region EA1 with a pixel group classified as structural element E1, region EA6 with a pixel group classified as structural element E6, region EA7 with a pixel group classified as structural element E7, region EA9 with a pixel group classified as structural element E9, and region EA10 with a pixel group classified as structural element E10.

[0350] Right now, Figure 17D The processing result image PGG is obtained as an image representing the following extraction result: four structural elements corresponding to structural elements E1, E6, E7 and E9 are extracted from the insertion shape image generated by the insertion shape image generation unit 261 and used as structural elements associated with the insertion shape of the insertion unit 11.

[0351] That is, the insertion shape element extraction unit 272 is configured to perform the following processing: from the insertion shape image generated by the insertion shape image generation unit 261, at least one of the following as structural elements related to the insertion shape of the insertion part 11 is extracted: the endoscope front end corresponding to the front end 12, the ring portion corresponding to the ring-shaped portion in the insertion part 11, and the zigzag portion corresponding to the zigzag portion in the insertion part 11, and the extraction result is obtained.

[0352] Furthermore, the insertion shape feature extraction unit 272 uses a classifier CLR generated by machine learning using teaching data to process and extract one or more structural features related to the insertion shape of the insertion part 11 inserted into the test body, thereby obtaining an extraction result. The teaching data includes an insertion shape image representing the insertion shape of the insertion part 11, and a label representing the classification result of each pixel contained in the insertion shape image as one of a number of prescribed structural features.

[0353] The insertion control unit 273 is configured to generate an insertion control signal containing information for performing the insertion operation of the insertion unit 11 based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, and the extraction result obtained by the insertion shape element extraction unit 272, and output it to the endoscope function control unit 240.

[0354] Specifically, the insertion control unit 273 generates an insertion control signal containing information based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, as well as the extraction result obtained by the insertion shape element extraction unit 272. This information is used to perform at least one of the following controls related to the insertion operation: the start of the insertion operation, the continuation of the insertion operation, the interruption of the insertion operation, the restart of the insertion operation, the stop of the insertion operation, and the completion of the insertion operation, as control related to the insertion operation of the insertion unit 11.

[0355] Furthermore, the insertion control unit 273 is configured to generate an insertion control signal containing information and output it to the endoscope function control unit 240 based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, as well as the extraction result obtained by the insertion shape element extraction unit 272. This information is used to control at least one of the operation amount, the operation speed, and the operation force of the insertion operation of the insertion unit 11.

[0356] Here, the insertion control unit 273 of this embodiment is configured to, for example, set control content based on at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, based on structural elements related to the current insertion shape of the insertion unit 11 as the extraction result obtained by the insertion shape element extraction unit 272; using the set control content, generate an insertion control signal containing information for performing control related to the insertion operation of the insertion unit 11, and output it to the endoscope function control unit 240.

[0357] Therefore, the insertion control unit 273 sets control content based on at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, based on the structural elements related to the current insertion shape of the insertion unit 11 as the extraction result obtained by the insertion shape element extraction unit 272. Thus, for example, it is possible to set an operation control group CGC, generate and output an insertion control signal containing information related to the set operation control group CGC, wherein the operation control group CGC has control content for individually performing a basic operation selected from each basic operation implemented by the function of the endoscope 10 to perform the insertion operation of the insertion unit 11.

[0358] Furthermore, the insertion control unit 273 sets control content based on at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, based on the structural elements related to the current insertion shape of the insertion unit 11 as the extraction result obtained by the insertion shape element extraction unit 272. Thus, for example, it is possible to set an operation control group CGD, generate and output an insertion control signal containing information related to the set operation control group CGD, which has control content for combining and executing multiple basic operations selected from the basic operations implemented by the endoscope 10 to perform the insertion operation of the insertion unit 11.

[0359] Furthermore, the operation control group CGD is configured to continuously or simultaneously execute multiple basic operations selected from the basic operations implemented by the endoscope 10. That is, the control content of the operation control group CGD is configured to be more complex than the control content of the operation control group CGC.

[0360] That is, the insertion control unit 263 is configured to perform control based on either the operation control group CGC or the operation control group CGD as a control corresponding to the structural elements related to the current insertion shape of the insertion unit 11 shown as the extraction result obtained by the insertion shape element extraction unit 272. The operation control group CGC has control content for individually executing a basic operation selected from each basic operation implemented by the endoscope 10 to perform the insertion operation of the insertion unit 11, and the operation control group CGD has control content for combining and executing multiple basic operations selected from each basic operation implemented by the endoscope 10 to perform the insertion operation of the insertion unit 11.

[0361] Furthermore, the insertion control unit 273 is configured to perform control related to the insertion operation of the insertion unit based on at least one of the following: an image obtained by taking pictures of the body under examination through the endoscope 10, information indicating the magnitude of the external force applied to the insertion unit 11, and information indicating the insertion shape of the insertion unit 11, as well as the extraction result obtained by the insertion shape element extraction unit 272.

[0362] Furthermore, the insertion control unit 273 is configured to change the control content based on at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, according to the change of at least one structural element contained in the extraction result obtained by the insertion shape element extraction unit 272 over time.

[0363] The extraction result recording unit 274 is configured to perform an operation for recording the extraction results obtained by the insertion shape element extraction unit 272 in a time sequence.

[0364] In this embodiment, at least a portion of the functions of the main device 20A can be implemented by the processor 20P. Furthermore, in this embodiment, at least a portion of the main device 20 can be configured as various electronic circuits, or as circuit modules in an integrated circuit such as an FPGA (Field Programmable Gate Array).

[0365] In addition, by appropriately modifying the structure of this embodiment, for example, a computer can read a program from a storage medium 20M such as a memory to perform at least a portion of the functions of the main device 20A, and perform actions corresponding to the read program.

[0366] Next, the function of this embodiment will be explained.

[0367] After connecting the various parts of the endoscope system 1A and turning on the power, the doctor or other user may, for example, configure the insertion part 11 such that the front end 12 is located near the anus or rectum of the subject.

[0368] According to the aforementioned user operation, illumination light supplied from the light source unit 210 is irradiated onto the subject, and the camera unit 110 captures an image of the subject illuminated by the illumination light. The resulting endoscopic image of the subject is then output from the image processing unit 220 to the display control unit 250 and the system control unit 270. Furthermore, according to the aforementioned user operation, a coil drive signal is supplied from the coil drive signal generation unit 230, and magnetic fields are generated from multiple source coils 18 based on this coil drive signal. Insertion shape information obtained by detecting the magnetic field is output from the insertion shape information acquisition unit 320 to the system control unit 270, and an insertion shape image corresponding to the insertion shape information is generated by the insertion shape image generation unit 261.

[0369] Furthermore, based on the aforementioned user operation, the external force information acquisition device 40 outputs external force information, representing the magnitude and direction of the external force at the respective positions of the multiple source coils 18, to the system control unit 270.

[0370] With the insertion unit 11 configured as described above, the user can, for example, instruct the main unit 20A to begin inserting the insertion unit 11 by turning on the automatic insertion switch of the input device 50.

[0371] When the extraction result recording unit 274 detects an instruction for insertion control for the insertion start unit 11, for example, it begins to record the extraction results obtained by the insertion shape element extraction unit 272 in a time sequence and at fixed intervals.

[0372] The insertion control unit 273 sets control content based on at least one of the following: the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261, based on the structural elements related to the current insertion shape of the insertion unit 11 as the extraction result obtained by the insertion shape element extraction unit 272.

[0373] Specifically, the insertion control unit 273, for example, if it detects that the extraction result obtained by the insertion shape element extraction unit 272 contains structural element E2 or E4, generates and outputs an insertion control signal containing information related to the operation control group CGC. The operation control group CGC is an operation control group whose control content is set based on at least one of the endoscopic image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261.

[0374] Furthermore, if the insertion control unit 273 detects that the extraction result obtained by the insertion shape element extraction unit 272 contains structural element E3, it generates and outputs an insertion control signal containing information related to the operation control group CGD. The operation control group CGD is an operation control group whose control content is set based on at least one of the endoscope image output from the image processing unit 220, the external force information output from the external force information acquisition device 40, and the insertion shape image generated by the insertion shape image generation unit 261.

[0375] The insertion control unit 273 changes the control content based on at least one of the following: the change of at least one structural element contained in the extraction result obtained by the insertion shape element extraction unit 272 over time; the endoscope image output from the image processing unit 220; the external force information output from the external force information acquisition device 40; and the insertion shape image generated by the insertion shape image generation unit 261.

[0376] Specifically, as a process for detecting the change in the position of the region EA1 contained in the processed result image obtained by the insertion shape feature extraction unit 272 over time, the insertion control unit 273 performs, for example, a process of binarizing the processed result image obtained by the insertion shape feature extraction unit 272 to generate a binarized image, a process for determining the centroid position of the region EA1 contained in the binarized image, and a process for detecting the change in the centroid position over time.

[0377] Furthermore, as a process for detecting the change in the area of ​​region EA8 contained in the processed result image obtained by the insertion shape feature extraction unit 272 over time, the insertion control unit 273 performs, for example, a process of binarizing the processed result image obtained by the insertion shape feature extraction unit 272 to generate a binarized image and a process for detecting the change in the number of pixels of region EA8 contained in the binarized image over time.

[0378] Furthermore, as a process for detecting changes in the shape of the region EA8 contained in the processed result image obtained by the insertion shape feature extraction unit 272 over time, the insertion control unit 273 performs, for example, the process of binarizing the processed result image obtained by the insertion shape feature extraction unit 272 to generate a binarized image and the process of detecting changes in the roundness of the region EA8 contained in the binarized image over time.

[0379] Furthermore, as a process for detecting the change in the area of ​​region EA3 contained in the processed result image obtained by the insertion shape feature extraction unit 272 over time, the insertion control unit 273 performs, for example, the process of binarizing the processed result image obtained by the insertion shape feature extraction unit 272 to generate a binarized image and the process of detecting the change in the number of pixels of region EA3 contained in the binarized image over time.

[0380] Furthermore, as a process for detecting changes in the length of the region EA3 contained in the processed result image obtained by the insertion shape feature extraction unit 272 over time, the insertion control unit 273 performs, for example, a process of binarizing the processed result image obtained by the insertion shape feature extraction unit 272 to generate a binarized image, a process of thinning the region EA3 contained in the binarized image to generate a line segment, and a process of detecting changes in the number of pixels of the line segment over time.

[0381] Furthermore, the insertion control unit 273 may change the control content based on at least one of the following changes over time: the position of region EA1, the area of ​​region EA8, the shape of region EA8, the area of ​​region EA3, and the length of region EA3 detected by the processing result image obtained by the insertion shape element extraction unit 272; the endoscope image output from the image processing unit 220; the external force information output from the external force information acquisition device 40; and the insertion shape image generated by the insertion shape image generation unit 261.

[0382] For example, after confirming that the insertion shape of the insertion part 11 inserted into the subject no longer changes based on the insertion shape image displayed on the display device 60, the user instructs the main device 20A to stop the insertion control of the insertion part 11 by turning off the automatic insertion switch of the input device 50.

[0383] When the extraction result recording unit 274 detects an instruction to stop the insertion control of the insertion stop unit 11, it stops the operation of recording the extraction results obtained by the insertion shape element extraction unit 272 in a time sequence and at fixed intervals.

[0384] As described above, according to this embodiment, the following processing is performed in the insertion shape element extraction unit 272: based on a viewpoint that is roughly equivalent to that of a skilled person when making a subjective judgment or evaluation of the success or failure of the insertion operation in the insertion unit 11, one or more structural elements contained in the insertion shape image generated by the insertion shape image generation unit 261 are extracted to obtain the extraction result.

[0385] Furthermore, according to this embodiment, insertion control is performed in the insertion control unit 273 corresponding to one or more structural elements included in the extraction result obtained by the insertion shape element extraction unit 272. Therefore, according to this embodiment, for example, appropriate insertion control can be performed corresponding to the insertion condition of the insertion part, such as individual differences in the internal state of the subject to which the insertion part is inserted and changes in the insertion shape of the insertion part inside the subject over time.

[0386] Furthermore, the insertion control unit 273 of this embodiment may, for example, be configured to use the classification result obtained by the insertion shape classification unit 262 and the extraction result obtained by the insertion shape element extraction unit 272 to perform the control described in the variation of the first embodiment. Specific examples of processing that can be implemented in this case are listed below.

[0387] For example, when the insertion control unit 273 performs jitter-related control on the endoscope function control unit 240 according to the control content contained in the insertion control information CJB, it obtains auxiliary information HJA that can be used to determine whether the jitter is successful or not by detecting the change in the position of region EA1 contained in the processed result image obtained by the insertion shape element extraction unit 272 over time. The aforementioned auxiliary information HJA, for example, can be used to determine whether friction occurs between the insertion part 11 and the intestinal tract, and whether deflection occurs in the insertion part 11, when performing control corresponding to the control content contained in the insertion control information CJB.

[0388] For example, when the insertion control unit 273 performs control corresponding to the control content contained in the insertion control information CJD on the endoscope function control unit 240, if it detects that the area of ​​region EA8 contained in the processed result image obtained by the insertion shape element extraction unit 272 exceeds a predetermined value, it performs control to advance the insertion unit 11 after retracting it by a predetermined amount. Moreover, according to this control, the α ring can be slightly reduced and the insertion unit 11 can be advanced.

[0389] For example, when the insertion control unit 273 performs control corresponding to the control content contained in the insertion control information CJD on the endoscope function control unit 240, if it detects that the area or length of the region EA3 contained in the processed result image obtained by the insertion shape element extraction unit 272 exceeds a predetermined value, it performs control to advance the insertion unit 11 after retracting it by a predetermined amount. Moreover, according to this control, the α ring can be slightly reduced and the insertion unit 11 can be advanced.

[0390] For example, if the insertion control unit 273 detects that region EA4 in the processed result image obtained by the insertion shape element extraction unit 272 has changed to EA2 and a new region EA5 has appeared in the processed result image, the insertion control unit 273 obtains the detection result that the insertion shape of the insertion unit 11 has changed from type TB to type TC.

[0391] For example, if the insertion control unit 273 detects that region EA2 in the processed result image obtained by the insertion shape element extraction unit 272 has changed to region EA3, it obtains the detection result that the insertion shape of the insertion unit 11 has changed from type TE to type TF.

[0392] For example, when the insertion control unit 273 controls the release of the α ring formed by the insertion unit 11 in relation to the endoscope function control unit 240, it detects which type of insertion shape of the insertion unit 11 is TE, TF, or TG based on the area of ​​the region EA8 contained in the processed result image obtained by the insertion shape element extraction unit 272.

[0393] For example, when the insertion control unit 273 performs control on the endoscope function control unit 240 corresponding to the control content contained in the insertion control information CJE, it detects the change in the position of region EA1 in the processed result image obtained by the insertion shape element extraction unit 272 over time, and obtains auxiliary information HJB corresponding to the retraction state of the insertion unit 11. The aforementioned auxiliary information HJB can be used, for example, to determine whether to change the retraction amount BLA and retraction speed BVA contained in the insertion control information CJE.

[0394] For example, when the insertion control unit 273 performs control on the endoscope function control unit 240 corresponding to the control content contained in the insertion control information CJG, it detects the change in the position of region EA1 in the processed result image obtained by the insertion shape element extraction unit 272 over time, and obtains auxiliary information HJC corresponding to the retraction state of the insertion unit 11. The aforementioned auxiliary information HJC can be used, for example, to determine whether to change the retraction amount BLB, retraction speed BVB, and rotation angle BAB contained in the insertion control information CJG.

[0395] For example, the insertion control unit 273 controls the endoscope function control unit 240 according to the control content contained in the insertion control information CJF. When it detects that regions EA3 and EA8 in the processed result image obtained by the insertion shape element extraction unit 272 have disappeared and regions EA6 and EA7 have appeared in the processed result image, it obtains the detection result that the insertion shape of the insertion unit 11 has changed from type TF to type TG.

[0396] For example, when the insertion control unit 273 performs control on the endoscope function control unit 240 corresponding to the control content contained in the insertion control information CJG, it acquires auxiliary information HJD corresponding to the positional relationship between regions EA6 and EA7 contained in the processed result image obtained by the insertion shape element extraction unit 272. The aforementioned auxiliary information HJD can be used, for example, to determine whether to change the backlash amount BLB, backlash speed BVB, and rotation angle BAB contained in the insertion control information CJG.

[0397] For example, the insertion control unit 273 controls the endoscope function control unit 240 according to the control content contained in the insertion control information CJG. When it detects that the regions EA6 and EA7 in the processing result image obtained by the insertion shape element extraction unit 272 have disappeared, it obtains the detection result that the insertion shape of the insertion unit 11 has changed from type TG to type TH.

[0398] For example, the insertion control unit 273 controls the endoscope function control unit 240 according to the control content contained in the insertion control information CJF. When it is detected that the processed result image obtained by the insertion shape element extraction unit 272 does not contain any region from region TH2 to region TH8, the insertion shape of the insertion unit 11 changes from type TG to type TH.

[0399] For example, if the endoscope function control unit 240 performs control corresponding to the control content contained in the insertion control information CJA, and the region EA4 still does not appear in the processed result image obtained by the insertion shape element extraction unit 272, the insertion control unit 273 obtains the detection result that the insertion shape of the insertion unit 11 changes from type TA to type TH by tracking the position of the region EA1 contained in the processed result image.

[0400] The present invention is not limited to the above-described embodiments and variations, and various changes and applications can be made without departing from the spirit of the invention.

Claims

1. An information processing apparatus that classifies a type of an insertion shape of an insertion portion of an endoscope inserted into a subject using information about the insertion shape of the insertion portion, characterized by comprising: has: an insertion shape classification section that obtains a classification result that classifies a kind of an insertion shape of the endoscope insertion section inserted into the subject into one of a plurality of kinds prescribed as time changes, based on a viewpoint equivalent to a viewpoint when a skilled person subjectively judges or evaluates success or failure of an operation in an insertion operation of the endoscope insertion section; an output section that outputs the classification result; and a control section that performs control related to the insertion operation of the endoscope insertion section based on the kind of the insertion shape of the endoscope insertion section indicated by the classification result.

2. The information processing apparatus according to claim 1, wherein the control section performs control based on either one of a first operation control group set to separately execute a control content of a basic operation selected from among basic operations related to the endoscope insertion section and a second operation control group set to combine and execute control contents of a plurality of basic operations selected from among the respective basic operations implemented by a function of an endoscope.

3. The information processing apparatus according to claim 2, wherein the second operation control group is set to successively or simultaneously execute the control contents of the plurality of basic operations.

4. The information processing apparatus according to claim 1, wherein the control section performs control related to the insertion operation of the endoscope insertion section based on the classification result, as the control related to the insertion operation of the endoscope insertion section, in at least one of a start, a continuation, an interruption, a restart, a stop, and a completion of the insertion operation of the endoscope insertion section.

5. The information processing apparatus according to claim 1, wherein the control section controls at least one of an operation amount, an operation speed, and an operation force of the insertion operation of the endoscope insertion section based on the classification result.

6. The information processing apparatus according to claim 1, wherein the control section performs control related to the insertion operation of the endoscope insertion section based on at least one of an image obtained by photographing the inside of the subject into which the endoscope insertion section is inserted, information indicating a magnitude of an external force applied to the endoscope insertion section, and information indicating an insertion shape of the endoscope insertion section, and the classification result.

7. The information processing apparatus according to claim 1, wherein the insertion shape classification section performs processing using a classifier generated by machine learning using teaching data containing an insertion shape image indicating an insertion shape of the endoscope insertion section and a label indicating a classification result that classifies the insertion shape of the endoscope insertion section included in the insertion shape image into one of the plurality of kinds prescribed.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus further has a classification result recording section that performs an action for recording the classification result in chronological order. has:

9. An endoscope control device that uses information relating to the insertion shape of an endoscope insertion portion inserted into a patient to perform control related to the insertion operation of the endoscope insertion portion, characterized in that, ​ an insertion shape factor extraction section that extracts one or more structure factors related to an insertion shape of the insertion section of the endoscope inserted into the subject based on a viewpoint equivalent to a viewpoint at which a person skilled in the art subjectively judges or evaluates success or failure of an operation in an insertion operation of the insertion section of the endoscope, and obtains an extraction result; and a control section that performs control related to the insertion operation of the insertion section of the endoscope based on a change over time of the one or more structure factors included in the extraction result.

10. The endoscope control device according to claim 9, wherein the control section performs control based on either of a first operation control group set to separately execute a control content of a basic operation selected from basic operations related to the insertion section of the endoscope and a second operation control group set to combine and execute control contents of a plurality of basic operations selected from the basic operations related to the insertion section of the endoscope.

11. The endoscope control device according to claim 10, wherein the second operation control group is set to successively or simultaneously execute the control contents of the plurality of basic operations.

12. The endoscope control device according to claim 9, wherein the control section performs control related to at least one of a start, continuation, interruption, restart, stop, and completion of the insertion operation of the insertion section of the endoscope based on the extraction result, as the control related to the insertion operation of the insertion section of the endoscope.

13. The endoscope control device according to claim 9, wherein the control section is configured to control at least one of an operation amount, an operation speed, and an operation force of the insertion operation of the insertion section of the endoscope based on the extraction result.

14. The endoscope control device according to claim 9, wherein the insertion shape factor extraction section performs processing for extracting at least one of a distal end portion, a loop portion, and a bending portion of the insertion section of the endoscope and obtaining the extraction result.

15. The endoscope control device according to claim 9, wherein the control section performs the control related to the insertion operation of the insertion section of the endoscope based on at least one of an image obtained by photographing the subject into which the insertion section of the endoscope is inserted, information indicating a magnitude of an external force applied to the insertion section of the endoscope, and information indicating an insertion shape of the insertion section of the endoscope, and the extraction result.

16. The endoscope control device according to claim 9, wherein the insertion shape factor extraction section performs processing using a classifier generated by machine learning using teaching data including an insertion shape image indicating an insertion shape of the insertion section of the endoscope and a label indicating a classification result of classifying each pixel included in the insertion shape image into one of a plurality of structure factors defined in advance.

17. A computer program product including a computer program for causing a computer to execute: a process for classifying a kind of an insertion shape of an endoscope insertion section into one of a plurality of kinds based on a viewpoint equivalent to a viewpoint of a skilled person making a subjective judgment or evaluation on success or failure of an operation in an insertion operation of the endoscope insertion section; a process of outputting the classification result; and a process of performing control related to the insertion operation of the endoscope insertion section based on the kind of the insertion shape of the endoscope insertion section indicated by the classification result.

18. The computer program product according to claim 17, wherein the computer program product further causes the computer to perform an action for recording the classification result in a time series.

19. A computer program product including a computer program for causing a computer to perform: a process for extracting one or more structural elements related to an insertion shape of an endoscope insertion section based on a viewpoint equivalent to a viewpoint of a skilled person making a subjective judgment or evaluation on success or failure of an operation in an insertion operation of the endoscope insertion section and obtaining an extraction result; and control related to the insertion operation of the endoscope insertion section based on a change in the one or more structural elements included in the extraction result over time.

20. A storage medium storing a program for causing a computer to perform: a process for obtaining a classification result of classifying a kind of an insertion shape of an endoscope insertion section inserted into a subject into one of a plurality of kinds based on a viewpoint equivalent to a viewpoint of a skilled person making a subjective judgment or evaluation on success or failure of an operation in an insertion operation of the endoscope insertion section; a process of outputting the classification result; and a process of performing control related to the insertion operation of the endoscope insertion section based on the kind of the insertion shape of the endoscope insertion section indicated by the classification result.

21. The storage medium according to claim 20, wherein the storage medium stores a program for causing the computer to further perform a process for recording the classification result in a time series.

22. A storage medium storing a program for causing a computer to perform: a process for extracting one or more structural elements related to an insertion shape of an endoscope insertion section inserted into a subject based on a viewpoint equivalent to a viewpoint of a skilled person making a subjective judgment or evaluation on success or failure of an operation in an insertion operation of the endoscope insertion section and obtaining an extraction result; and control related to the insertion operation of the endoscope insertion section based on a change in the one or more structural elements included in the extraction result over time.

Images