ENDOSCOPE INSERTION SUPPORT SYSTEM, OPERATION METHOD OF ENDOSCOPE INSERTION SUPPORT SYSTEM, AND STORAGE MEDIUM

Abstract

An endoscope image acquisition unit 32 acquires shape information on an inserted endoscope portion during an endoscopic examination. In response to prescribed conditions, the present invention executes at least one of: the processing in which an insertion state estimation unit 33 inputs the shape information into an insertion state training model 33a and acquires, from the insertion state training model 33a, an insertion state category obtained by categorizing the insertion state of the inserted endoscope portion; and the processing in which a shape determination unit 34 applies the shape information to a prescribed shape category determination logic 34a to acquire an insertion shape category obtained by categorizing the insertion shape of the inserted endoscope portion.

Description

[Technical field]

[0001] The present disclosure relates to an endoscope insertion support system that supports the insertion of an endoscope, System Operation The present invention relates to a method and a storage medium. [Background technology]

[0002] 2. Description of the Related Art In a colonoscopic examination, an endoscope control device is known that uses AI to classify the shape of the insertion portion of a colonoscope and controls the insertion of the colonoscope in accordance with the classification results (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO 21 / 038871 Summary of the Invention [Problem to be solved by the invention]

[0004] When using the above-mentioned endoscope control device to control insertion or display the insertion guide, if the AI ​​misrecognizes the shape of the insertion part, the insertion control or insertion guide display may not be optimal, resulting in cases where the endoscope cannot be inserted smoothly.

[0005] The present disclosure has been made in consideration of these circumstances, and has an object to provide a technique that can recognize the insertion status of an endoscope with high accuracy. [Means for solving the problem]

[0006] In order to solve the above problems, an endoscope insertion support system according to an embodiment of the present disclosure is an endoscope insertion support system that includes one or more processors having hardware. The processor executes at least one of the following: acquiring shape information of an endoscope insertion portion during endoscopic examination, inputting the shape information to an insertion situation learning model according to a predetermined condition, and acquiring an insertion situation category that categorizes an insertion situation of the endoscope insertion portion from the insertion situation learning model; and applying the shape information to a predetermined shape category determination logic, and acquiring an insertion shape category that categorizes an insertion shape of the endoscope insertion portion.

[0007] Another aspect of the present invention is an endoscope insertion assistance method comprising: acquiring shape information of an endoscope insertion portion during endoscopic examination; and, in accordance with a predetermined condition, inputting the shape information into an insertion situation learning model to acquire insertion situation categories that categorize insertion situations of the endoscope insertion portion from the insertion situation learning model; and applying the shape information to a predetermined shape category determination logic to acquire insertion shape categories that categorize insertion shapes of the endoscope insertion portion.

[0008] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc. are also valid aspects of the present disclosure. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an overall system configuration related to a colonoscopy examination according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an endoscope used in the present embodiment. [Diagram 3] 1 is a diagram showing a configuration example of an endoscope insertion support system according to a first embodiment. [Figure 4] FIG. 1 is a diagram showing a colonoscopy examination. [Diagram 5] FIG. 13 is a diagram showing an example of a screen displayed on a display device during an endoscopic examination. [Figure 6]6(a) and 6(b) are diagrams for explaining an example of the determination logic used in the shape determination unit. [Figure 7] 7(a) and (b) are diagrams for explaining correction example 1 of the insertion status category. [Figure 8] 8(a) and (b) are diagrams for explaining a second correction example of the insertion status category. [Figure 9] 5 is a flowchart showing an operation example of the endoscope insertion support system according to the first embodiment. [Figure 10] FIG. 11 is a diagram illustrating a configuration example of an endoscope insertion support system according to a second embodiment. [Figure 11] 11(a) and 11(b) are diagrams showing an example of the shape of an endoscope before and after changing a body position. [Figure 12] 10 is a flowchart showing an example of the operation of the endoscope insertion support system according to the second embodiment. [Figure 13] FIG. 11 is a diagram illustrating a configuration example of an endoscope insertion support system according to a third embodiment. [Figure 14] 13 is a flowchart showing an example of the operation of the endoscope insertion support system according to the third embodiment. [Figure 15] FIG. 13 is a diagram illustrating a configuration example of an endoscope insertion support system according to a fourth embodiment. [Figure 16] 13 is a flowchart showing an example of the operation of the endoscope insertion support system according to the fourth embodiment. [Figure 17] FIG. 13 is a diagram illustrating a configuration example of an endoscope insertion support system according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] This embodiment relates to a colonoscopic examination. In order to perform a high-quality colonoscopic examination with less stress on the patient, highly advanced technology is required. In order to realize a high-quality colonoscopic examination, a system has been developed that uses AI to recognize the insertion status of the endoscope, i.e., the shape of the insertion part of the endoscope inside the body, and determines the optimal operation method for that insertion status. If the AI ​​in this system misrecognizes the insertion status of the endoscope, it may lead to presenting an incorrect operation method to the doctor or sending incorrect operation information to the control system of the automatic insertion device.

[0011] In this embodiment, in order to reduce such risks, rather than simply adopting the judgment results of one AI, a mechanism is introduced to reduce erroneous recognition of the insertion status by combining the shape recognition results based on logic with the judgment results of other AIs.

[0012] 1 is a diagram showing an overall system configuration related to a colonoscopic examination according to an embodiment. In this embodiment, an endoscope system 10, an endoscope 11, a light source device 15, an endoscope insertion shape observation device (UPD: Endoscope Position Detecting Unit) 20, an endoscope insertion support system 30, a display device 41, an input device 42, and a storage device 43 are used. The endoscope 11 according to this embodiment is a colonoscope that is inserted into the large intestine of a subject (patient).

[0013] The endoscope 11 includes a lens and a solid-state image sensor (for example, a CMOS image sensor, a CCD image sensor, or a CMD image sensor). The solid-state image sensor converts light collected by the lens into an electrical signal and outputs it to the endoscope system 10 as an endoscopic image (electrical signal). The endoscope 11 includes a forceps channel. An operator (doctor) can perform various procedures during an endoscopic examination by passing a treatment tool through the forceps channel.

[0014] The light source device 15 includes a light source such as a xenon lamp, and supplies observation light (white light, narrowband light, fluorescent light, near-infrared light, etc.) to the tip of the endoscope 11. The light source device 15 also includes a pump that sends water or air to the endoscope 11.

[0015] The endoscope system 10 controls the light source device 15 and processes an endoscopic image input from the endoscope 11. The endoscope system 10 is equipped with functions such as narrow band imaging (NBI), red dichromatic imaging (RDI), texture and color enhancement imaging (TXI), and extended depth of field (EDOF), for example.

[0016] In narrowband light observation, endoscopic images can be obtained that emphasize the capillaries and fine structures of the mucosal surface by irradiating light of specific wavelengths of purple (415 nm) and green (540 nm), which are strongly absorbed by hemoglobin in blood. In red light observation, endoscopic images can be obtained that emphasize the contrast of deep tissues by irradiating light of specific wavelengths of three colors (green, amber, and red). In structural color enhancement, endoscopic images are generated that optimize the three elements of the "structure," "color tone," and "brightness" of the mucosal surface under normal light observation. In extended depth of field, endoscopic images with a wide focal range can be obtained by synthesizing two images focused on close and far distances.

[0017] The endoscope system 10 outputs an endoscope image obtained by processing an endoscope image input from the endoscope 11, or outputs the endoscope image input from the endoscope 11 as is, to the endoscope insertion support system 30.

[0018] The endoscope insertion shape observation device 20 is a device for observing the three-dimensional shape of the endoscope 11 inserted into the lumen of the subject. A receiving antenna 20a is connected to the endoscope insertion shape observation device 20. The receiving antenna 20a is an antenna for detecting magnetic fields generated by multiple magnetic coils built into the endoscope 11.

[0019] 2 is a diagram showing an example of an endoscope 11 used in this embodiment. The endoscope 11 has an elongated tubular insertion section 11a made of a flexible member, and an operation section 11e connected to a base end of the insertion section 11a. The insertion section 11a has, from the tip side to the base end side, a tip hard section 11b, a bending section 11c, and a flexible tube section 11d. The base end of the tip hard section 11b is connected to the tip end of the bending section 11c, and the base end of the bending section 11c is connected to the base end of the flexible tube section 11d.

[0020] The operation section 11e has a main body section 11f from which the flexible tube section 11d extends, and a grip section 11g connected to the base end of the main body section 11f. The grip section 11g is held by the surgeon. A universal cord including an imaging electric cable, a light guide, and the like extends from the insertion section 11a from the operation section 11e, and is connected to the endoscope system 10 and the light source device 15.

[0021] The tip hard portion 11b is the tip of the insertion portion 11a and also the tip of the endoscope 11. The tip hard portion 11b has built-in components such as a solid-state imaging element, an illumination optical system, and an observation optical system. Illumination light emitted from the light source device 15 is propagated along the light guide to the tip surface of the tip hard portion 11b, and is irradiated from the tip surface of the tip hard portion 11b toward an observation target in a lumen.

[0022] The bending section 11c is configured by connecting the joint rings along the longitudinal axis direction of the insertion section 11a. The bending section 11c bends in a desired direction in response to an operation of the surgeon inputted to the operation section 11e, and the position and orientation of the distal end rigid section 11b change in response to the bending.

[0023] The flexible tube portion 11d is a tubular member extending from the main body portion 11f of the operation portion 11e, has a desired flexibility, and is bent by an external force. The surgeon inserts the insertion portion 11a into the large intestine of the subject while bending the bending portion 11c and twisting the flexible tube portion 11d.

[0024] A plurality of magnetic coils 12 are arranged at predetermined intervals (e.g., 10 cm intervals) inside the insertion portion 11a along the longitudinal direction. Each magnetic coil 12 generates a magnetic field when a current is supplied to it. The magnetic coils 12 function as position sensors for detecting each position of the insertion portion 11a.

[0025] Returning to Fig. 1, the receiving antenna 20a receives magnetic fields emitted from the multiple magnetic coils 12 built in the insertion section 11a of the endoscope 11, and outputs the magnetic fields to the endoscope insertion shape observation device 20. The endoscope insertion shape observation device 20 applies the magnetic field strength of each of the multiple magnetic coils 12 received by the receiving antenna 20a to a predetermined position detection algorithm to estimate the three-dimensional position of each of the multiple magnetic coils 12. The endoscope insertion shape observation device 20 generates a three-dimensional endoscope shape of the insertion section 11a of the endoscope 11 by curve-interpolating the estimated three-dimensional positions of the multiple magnetic coils 12.

[0026] The reference plate 20b is attached to the subject (e.g., the abdomen of the subject). A posture sensor for detecting the posture of the subject is disposed on the reference plate 20b. For example, a three-axis acceleration sensor or a gyro sensor can be used as the posture sensor. In FIG. 1, the reference plate 20b is connected to the endoscope insertion shape observation device 20 via a cable, and the reference plate 20b outputs three-dimensional posture information indicating the posture of the reference plate 20b (i.e., the posture of the subject) to the endoscope insertion shape observation device 20.

[0027] Note that multiple magnetic coils similar to the multiple magnetic coils 12 built in the insertion section 11a of the endoscope 11 may be used as the body position sensor arranged on the reference plate 20b. In this case, the receiving antenna 20a receives magnetic fields emitted from the multiple magnetic coils arranged on the reference plate 20b and outputs them to the endoscope insertion shape observation device 20. The endoscope insertion shape observation device 20 applies the magnetic field strengths of the multiple magnetic coils received by the receiving antenna 20a to a predetermined posture detection algorithm to generate three-dimensional posture information indicating the posture of the reference plate 20b (i.e., the posture of the subject).

[0028] The endoscope insertion shape observation device 20 changes the generated three-dimensional endoscope shape so as to follow the change in the three-dimensional posture information. Specifically, the endoscope insertion shape observation device 20 changes the three-dimensional endoscope shape so as to cancel the change in the three-dimensional posture information. This makes it possible to always recognize the endoscope shape from a specific viewpoint (for example, a viewpoint that views the abdomen of the subject perpendicularly from the front side of the abdomen) even if the subject's posture is changed during endoscopic examination.

[0029] The endoscope insertion shape observation device 20 can acquire an insertion length indicating the length of the portion of the endoscope 11 inserted into the large intestine, and the time that has elapsed since the endoscope 11 was inserted into the large intestine (hereinafter referred to as insertion time). For example, the endoscope insertion shape observation device 20 measures the insertion length with the position of the timing at which the surgeon inputs an operation to start the examination into the input device 42 as the base point, and measures the insertion time with the timing as the base point. The endoscope insertion shape observation device 20 may estimate the position of the anus from the generated three-dimensional endoscope shape and the difference in magnetic field strength between a magnetic coil inside the body and a magnetic coil outside the body, and may use the estimated position of the anus as the base point of the insertion length.

[0030] In order to measure the insertion length with high accuracy, an encoder may be installed near the subject's anus. The endoscope insertion shape observation device 20 detects the insertion length from the position of the anus based on the signal from the encoder.

[0031] The endoscope insertion shape observation device 20 adds the insertion length and insertion time to the three-dimensional endoscope shape after posture correction based on the three-dimensional posture information, and outputs the result to the endoscope insertion support system 30.

[0032] The endoscope insertion support system 30 generates insertion support information for the endoscope 11 based on the endoscope image input from the endoscope system 10 and the endoscope shape input from the endoscope insertion shape observation device 20, and presents it to the surgeon. In addition, the endoscope insertion support system 30 generates endoscopic examination history information based on the endoscope image input from the endoscope system 10 and the endoscope shape input from the endoscope insertion shape observation device 20, and records it in the storage device 43.

[0033] The display device 41 includes a liquid crystal monitor or an organic electroluminescence monitor, and displays an image input from the endoscope insertion support system 30. The input device 42 includes a mouse, a keyboard, a touch panel, and the like, and outputs operation information input by an operator or the like to the endoscope insertion support system 30. The storage device 43 includes a storage medium such as an HDD or SSD, and stores the endoscopic examination history information generated by the endoscope insertion support system 30. The storage device 43 may be a dedicated storage device associated with the endoscope system 10, or may be a database in an in-hospital server connected via an in-hospital network, or may be a database in a cloud server.

[0034] 3 shows a configuration example of an endoscope insertion support system 30 according to the first embodiment. The endoscope insertion support system 30 may be constructed with a processing device dedicated to endoscope insertion support, or may be constructed with a general-purpose server (which may be a cloud server). The endoscope insertion support system 30 may also be constructed with any combination of a processing device dedicated to endoscope insertion support, a general-purpose server (which may be a cloud server), and a dedicated image diagnostic device. The endoscope insertion support system 30 may also be constructed integrally with the endoscope system 10.

[0035] The endoscope insertion support system 30 includes an endoscope shape acquisition unit 31, an endoscope image acquisition unit 32, an insertion status estimation unit 33, a shape determination unit 34, a site estimation unit 35, and an insertion support information generation unit 36. These components can be realized in terms of hardware using at least one arbitrary processor (e.g., CPU, GPU), memory (e.g., DRAM), or other LSI (e.g., FPGA, ASIC), and in terms of software using a program loaded into memory, but here, functional blocks realized by cooperation between them are depicted. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof.

[0036] The endoscope shape acquisition unit 31 acquires shape information of the insertion portion 11a of the endoscope 11 during endoscopic examination (hereinafter simply referred to as the endoscope insertion portion) from the endoscope insertion shape observation device 20. The endoscope image acquisition unit 32 acquires an endoscopic image during endoscopic examination from the endoscope system 10.

[0037] The insertion status estimation unit 33 has an insertion status learning model 33a. The insertion status estimation unit 33 inputs shape information of the endoscope insertion part acquired by the endoscope shape acquisition unit 31 to the insertion status learning model 33a, and acquires an insertion status category that categorizes the insertion status of the endoscope insertion part from the insertion status learning model 33a. The insertion status of the endoscope insertion part indicates the arrangement status of the entire endoscope insertion part inside the body of the subject. The insertion status category of the endoscope insertion part is a classification into a plurality of deformed shapes by dividing the entire endoscope insertion part into several insertion stages, based on where and what shape the endoscope insertion part is arranged inside the body of the subject.

[0038] The insertion status learning model 33a is generated by machine learning using a supervised data set of shape information of a large number of endoscope insertion parts to which the insertion status categories of the endoscope insertion parts are annotated. The annotations are provided by annotators with specialized knowledge, such as doctors. For the machine learning, CNN, RNN, LSTM, etc., which are types of deep learning, can be used.

[0039] The data set may be a combination of a large number of endoscope images and a large number of pieces of shape information of the endoscope insertion part, to which an insertion status category of the endoscope insertion part is added as an annotation. In this case, the insertion status estimation unit 33 inputs the shape information of the endoscope insertion part and the corresponding endoscope image to the insertion status learning model 33a, and acquires the insertion status category from the insertion status learning model 33a.

[0040] Part estimation unit 35 has part learning model 35a. Part estimation unit 35 inputs the endoscopic image acquired by endoscopic image acquisition unit 32 to part learning model 35a, and acquires the part of the subject shown in the endoscopic image from part learning model 35a. Part learning model 35a is generated by machine learning using a large number of endoscopic images, to which parts of the large intestine are annotated, as a supervised dataset.

[0041] The data set may be a combination of many endoscopic images and many pieces of shape information of the endoscope insertion part, to which the parts of the large intestine are annotated. In this case, part estimation unit 35 inputs the endoscopic images and the corresponding shape information of the endoscope insertion part to part learning model 35a, and acquires the parts of the large intestine from part learning model 35a.

[0042] The region to be acquired may be the region where the tip hard portion 11b of the endoscope 11 (hereinafter simply referred to as the endoscope tip portion) is located, or may be the region where each of the multiple parts of the endoscope insertion portion is located. Each of the multiple parts of the endoscope insertion portion may be a portion where the magnetic coil 12 is installed. The regions of the large intestine may be classified into five regions, the sigmoid colon, descending colon, transverse colon, ascending colon, and cecum, or may be classified into two regions, the sigmoid colon or the part after the sigmoid colon, or may be classified into any other unit.

[0043] When the part of the subject detected by part learning model 35a is not detected continuously for a predetermined time, part estimation unit 35 may invalidate the detected part of the subject. That is, part estimation unit 35 may provide a "delay" in the part estimation result. The "delay" here refers to a process of determining that a recognition error has occurred when the same estimation result is not returned for a predetermined number of frames when switching parts.

[0044] For example, when the frame rate of the endoscopic image is 60 Hz and a one-second "Delay" is provided, if the same estimation result is not returned for 60 frames of endoscopic images that are consecutive in time series, the part estimation unit 35 determines that there has been an erroneous recognition. Also, when the frame rate of the endoscopic image is 60 Hz, the acquisition rate of shape information of the endoscope insertion part is 5 Hz, and a one-second "Delay" is provided, if the same estimation result is not returned for a combination of 60 frames of endoscopic images that are consecutive in time series and the corresponding five pieces of shape information of the endoscope insertion part, the part estimation unit 35 determines that there has been an erroneous recognition.

[0045] The shape determination unit 34 has a shape category determination logic 34a. The shape determination unit 34 applies the shape information of the endoscope insertion part acquired by the endoscope shape acquisition unit 31 to the shape category determination logic 34a to acquire an insertion shape category that categorizes the insertion shape of the endoscope insertion part. The shape category determination logic 34a is a determination logic (defined by geometric conditions, for example) for geometrically classifying the shape of the endoscope insertion part, and is determined in advance by a designer. For example, the shape determination unit 34 applies the shape information of the endoscope insertion part to the shape category determination logic 34a to determine various loops (inverted X, N, α (front and back), inverted α (front and back), γ (front and back), re-loop) of the endoscope tip part and the presence or absence of a hook in the center of the transverse colon.

[0046] The following is a comparison of the characteristics of the estimation of the insertion status of the endoscope insertion part by the insertion status estimation unit 33 and the determination of the insertion shape of the endoscope insertion part by the shape determination unit 34. The estimation by the insertion status estimation unit 33 has a wide range of recognition of the insertion status. That is, even if it does not completely match any of the models of the insertion status categories, it will be assigned to the insertion status category with the highest similarity. In addition, it is possible to estimate the insertion status of the endoscope insertion part including the part. In addition, time series information can also be used. The parts of the large intestine progress from the anus side in the order of sigmoid colon, descending colon, transverse colon, ascending colon, and cecum. In the estimation by the insertion status estimation unit 33, the part can also be specified using time series information. However, the estimation accuracy by the insertion status estimation unit 33 depends on the data set to be learned. There is variation in the estimation accuracy depending on the insertion status category.

[0047] On the other hand, in the judgment by the shape judgment unit 34, the shape set by the judgment logic can be recognized with high accuracy. However, in the judgment by the shape judgment unit 34, there is no flexibility in recognizing the insertion shape. In other words, even a slight deviation from the set conditions makes it impossible to recognize. In the judgment by the shape judgment unit 34, since a learning data set is not used, there is a limit to the conditions that can be set.

[0048] By combining the estimation result of the insertion status estimation unit 33 and the determination result of the shape determination unit 34, it is possible to reduce erroneous recognition due to variations in accuracy of the learning data set while enabling broad recognition including parts based only on shape information of the endoscope insertion part. For example, the estimation accuracy can be improved by correcting the estimation result by the insertion status estimation unit 33 for an insertion status category with a small learning data set with the determination result by the shape determination unit 34.

[0049] However, when the insertion status is estimated by the insertion status estimation unit 33 and the insertion shape is determined by the shape determination unit 34, erroneous recognition is likely to occur in cases where the insertion shapes are similar but the parts are different. It is possible to provide a "delay" in the estimation of the insertion status by the insertion status estimation unit 33 as in the part estimation unit 35, but it is difficult to provide a long "delay" in the estimation of the insertion status by the insertion status estimation unit 33. If a long "delay" is provided, the insertion shape will change and the same result will not be returned. Therefore, providing a "delay" in the estimation by the insertion status estimation unit 33 has little effect in preventing erroneous recognition.

[0050] Therefore, by combining the result of part estimation by part estimation unit 35, which has a "Delay" provided to reduce the occurrence of erroneous recognition, with the result of insertion status estimation by insertion status estimation unit 33 and the result of insertion shape determination by shape determination unit 34, it is possible to more accurately determine which insertion shape is at which part. In other words, by having the insertion status estimation unit 33 estimate an insertion status category including the part, while having the shape determination unit 34 complement the correctness of the insertion shape at parts that the insertion status estimation unit 33 is weak at, and having the part estimation unit 35 complement the correctness of parts at parts that the insertion status estimation unit 33 is weak at, it becomes possible to recognize almost all insertion statuses that can be estimated by the insertion status estimation unit 33 with high accuracy.

[0051] The insertion support information generating unit 36 ​​generates insertion support information based on the insertion status category. The insertion support information generating unit 36 ​​outputs the generated insertion support information to the display device 41 for display. The insertion support information includes operation support information for the surgeon to operate the endoscope 11. The insertion support information is previously associated with each insertion status category.

[0052] 4 is a diagram showing a state of a colonoscopic examination. The surgeon inserts the endoscope 11 into the anus of the subject, and performs the endoscopic examination while viewing the endoscopic image captured by the endoscope 11 and the insertion support information displayed on the display device 41.

[0053] 5 is a diagram showing an example of a screen displayed on the display device 41 during an endoscopic examination. An endoscopic image A1 captured by the endoscope 11 is displayed on the right side of the screen, and an endoscope shape B1 is displayed on the left side. A guide C3 indicating the traveling direction of the endoscope 11 is displayed in the upper right corner of the endoscopic image A1, and a guide C1 recommending a pull operation and a guide C2 recommending a right torque operation are displayed as operation support information below the endoscopic image A1.

[0054] 6(a)-(b) are diagrams for explaining an example of the judgment logic used by the shape judgment unit 34. The example shown in FIG. 6(a)-(b) shows the judgment logic for judging the endoscope shape B2 curved to the right or left. FIG. 6(a) shows the parameters of the endoscope shape B2, and FIG. 6(b) shows the conditional expression of the judgment logic. The shape judgment unit 34 compares the inner angle θ of the curved portion of the endoscope shape B2 with a threshold value (S341a). If the inner angle θ of the curved portion is equal to or greater than the threshold value (N in S341a), the shape judgment unit 34 judges that the endoscope shape B2 does not correspond to the judgment logic.

[0055] If the inner angle θ of the bending portion is less than the threshold value (Y in S341a), the shape determination unit 34 compares the x-axis coordinate xa of the apex of the bending portion of the endoscope shape B2 with the x-axis coordinate xb of the tip of the bending portion (S341b). If the x-axis coordinate xa of the apex of the bending portion is smaller than the x-axis coordinate xb of the tip of the bending portion (Y in S341b), the shape determination unit 34 determines that the endoscope shape is curved to the right. If the x-axis coordinate xa of the apex of the bending portion is greater than the x-axis coordinate xb of the tip of the bending portion (N in S341b), the shape determination unit 34 determines that the endoscope shape is curved to the left.

[0056] If the insertion shape category included in the insertion status category output from the insertion status estimation unit 33 does not match the insertion shape category output from the shape determination unit 34, the insertion support information generation unit 36 ​​corrects the insertion shape category included in the insertion status category output from the insertion status estimation unit 33 to the insertion shape category output from the shape determination unit 34.

[0057] When the part of the subject included in the insertion status category output from the insertion status estimation unit 33 does not match the part of the subject output from the part estimation unit 35, the insertion assistance information generation unit 36 ​​corrects the part of the subject included in the insertion status category output from the insertion status estimation unit 33 to the part of the subject output from the part estimation unit 35.

[0058] 7(a)-(b) are diagrams for explaining correction example 1 of the insertion status category. Fig. 7(a) shows the actual insertion status of the endoscope insertion part, and Fig. 7(b) shows the insertion status of the endoscope insertion part estimated by the insertion status estimation unit 33. As shown in Fig. 7(a), in the actual insertion status (correct answer), the insertion shape B11a of the endoscope insertion part is the "front α loop", and the part d11a where the endoscope tip is located is the "sigmoid colon". As shown in Fig. 7(b), in the insertion status (incorrect answer) estimated by the insertion status estimation unit 33, the insertion shape B11b of the endoscope insertion part is the "reverse α loop", and the part d11b where the endoscope tip is located is the "sigmoid colon".

[0059] The location where the endoscope tip is located estimated by location estimation unit 35 is the "sigmoid colon." The insertion shape of the endoscope insertion part determined by shape determination unit 34 is the "front alpha loop." The location where the endoscope tip is located estimated by insertion status estimation unit 33 and the location where the endoscope tip is located estimated by location estimation unit 35 are the "sigmoid colon," which are the same. Therefore, insertion support information generation unit 36 ​​determines that the location where the endoscope tip is located is the "sigmoid colon."

[0060] The "reverse α loop", which is the insertion shape of the endoscope insertion part estimated by the insertion status estimation unit 33, differs from the "front α loop", which is the insertion shape of the endoscope insertion part determined by the shape determination unit 34. Since the shape determination unit 34 has higher accuracy in recognizing the loop shape, the insertion support information generation unit 36 ​​determines that the "reverse α loop" estimated by the insertion status estimation unit 33 is an erroneous recognition, and adopts the "front α loop" determined by the shape determination unit 34.

[0061] That is, the insertion support information generating unit 36 ​​corrects the insertion shape category defined by "sigmoid colon" and "rear alpha loop" estimated by the insertion situation estimating unit 33 to the insertion shape category defined by "sigmoid colon" and "front alpha loop". The insertion support information generating unit 36 ​​acquires insertion support information linked to the corrected insertion situation category, and displays on the display device 41 an operation guide based on the acquired insertion support information.

[0062] 8(a)-(b) are diagrams for explaining a correction example 2 of the insertion situation category. FIG. 8(a) shows the actual insertion situation of the endoscope insertion part, and FIG. 8(b) shows the insertion situation of the endoscope insertion part estimated by the insertion situation estimation unit 33. As shown in FIG. 8(a), in the actual insertion situation (correct answer), the insertion shape B12a of the endoscope insertion part is "no loop, tip curved to the left", and the region d12a where the endoscope tip is located is "sigmoid colon". As shown in FIG. 8(b), in the insertion situation (incorrect answer) estimated by the insertion situation estimation unit 33, the insertion shape B12b of the endoscope insertion part is "no loop, tip curved to the left", and the region d12b where the endoscope tip is located is "transverse colon (more specifically, "splenic flexure")".

[0063] The location where the endoscope tip is located as estimated by location estimation unit 35 is the "sigmoid colon." The insertion shape of the endoscope insertion part as determined by shape determination unit 34 is "no loop, tip curved to the left." The location where the endoscope tip is located as estimated by insertion status estimation unit 33 is the "transverse colon," which is different from the location where the endoscope tip is located as estimated by location estimation unit 35, which is the "sigmoid colon." Since location recognition is more accurate with location estimation unit 35, insertion support information generation unit 36 ​​determines that the "transverse colon" estimated by insertion status estimation unit 33 is an erroneous recognition, and adopts the "sigmoid colon" estimated by location estimation unit 35.

[0064] The insertion shape of the endoscope insertion part estimated by the insertion status estimation unit 33 and the insertion shape of the endoscope insertion part determined by the shape determination unit 34 are "no loop, tip curved to the left", which are consistent. Therefore, the insertion support information generation unit 36 ​​determines that the insertion shape of the endoscope insertion part is "no loop, tip curved to the left".

[0065] That is, the insertion assistance information generating unit 36 ​​corrects the insertion shape category defined by "transverse colon" and "no loop, tip curved to the left" estimated by the insertion status estimating unit 33 to an insertion shape category defined by "sigmoid colon" and "no loop, tip curved to the left".

[0066] 9 is a flowchart showing an example of the operation of the endoscope insertion support system 30 according to embodiment 1. The endoscope image acquisition unit 32 acquires an endoscope image from the endoscope system 10, and the endoscope shape acquisition unit 31 acquires shape information of the endoscope insertion part from the endoscope insertion shape observation device 20 (S10).

[0067] The insertion status estimation unit 33 inputs the shape information of the endoscope insertion portion and the endoscope image to the insertion status learning model 33a, and acquires the insertion status category of the endoscope insertion portion from the insertion status learning model 33a (S20). The part estimation unit 35 inputs the endoscope image and the shape information of the endoscope insertion portion to the part learning model 35a, and acquires the part where the endoscope tip is located from the part learning model 35a (S30). The shape determination unit 34 applies the shape information of the endoscope insertion portion to the shape category determination logic 34a, and acquires the insertion shape category from the shape category determination logic 34a (S40). The processes of steps S20, S30, and S40 may be executed in chronological order or simultaneously in parallel.

[0068] The insertion support information generation unit 36 ​​compares the insertion shape category included in the insertion situation category estimated by the insertion situation estimation unit 33 with the insertion shape category determined by the shape determination unit 34 (S50). If the two do not match (N in S50), the insertion support information generation unit 36 ​​corrects the insertion shape category included in the insertion situation category estimated by the insertion situation estimation unit 33 to the insertion shape category determined by the shape determination unit 34 (S60). If the two match (Y in S50), the processing of step S60 is skipped.

[0069] The insertion support information generating unit 36 ​​compares the location of the endoscope tip included in the insertion status category estimated by the insertion status estimating unit 33 with the location of the endoscope tip estimated by the location estimating unit 35 (S70). If they do not match (N in S70), the insertion support information generating unit 36 ​​corrects the location of the endoscope tip included in the insertion status category estimated by the insertion status estimating unit 33 to the location of the endoscope tip estimated by the location estimating unit 35 (S80). If they match (Y in S70), the process of step S80 is skipped. The insertion support information generating unit 36 ​​acquires insertion support information linked to the finally determined insertion status category, and displays the acquired insertion support information on the display device 41 (S90).

[0070] In addition, priority information specifying whether to give priority to the insertion shape category included in the insertion situation category estimated by the insertion situation estimation unit 33 or the insertion shape category determined by the shape determination unit 34 may be pre-assigned to each of the multiple insertion situation categories prepared as estimation results of the insertion situation learning model 33a.

[0071] In addition, priority information specifying whether to give priority to the insertion shape category included in the insertion status category estimated by the insertion status estimation unit 33 or the insertion shape category determined by the shape determination unit 34 may be pre-assigned to each of the multiple insertion shape categories prepared as the determination result of the shape category determination logic 34a.

[0072] The insertion support information generation unit 36 ​​determines whether or not to correct the insertion shape category included in the insertion situation category obtained from the insertion situation learning model 33a to the insertion shape category obtained from the shape category determination logic 34a, based on priority information assigned to at least one of the insertion shape category included in the insertion situation category estimated by the insertion situation estimation unit 33 and the insertion shape category determined by the shape determination unit 34.

[0073] When the priority of the insertion shape category included in the insertion situation category estimated by the insertion situation estimation unit 33 is higher, the insertion support information generation unit 36 ​​does not correct the insertion shape category included in the insertion situation category acquired from the insertion situation learning model 33a. Conversely, when the priority of the insertion shape category determined by the shape determination unit 34 is higher, the insertion support information generation unit 36 ​​corrects the insertion shape category included in the insertion situation category acquired from the insertion situation learning model 33a to the insertion shape category acquired from the shape category determination logic 34a.

[0074] When the insertion status estimation unit 33 inputs shape information of the endoscope insertion part (or a combination of shape information and an endoscopic image) into the insertion status learning model 33a to obtain an insertion status category, it can simultaneously obtain a reliability based on the similarity between the input shape information (or a combination of shape information and an endoscopic image) and the classified insertion status category.

[0075] The insertion support information generating unit 36 ​​can determine whether or not to correct the insertion shape category included in the insertion situation category acquired from the insertion situation learning model 33a to the insertion shape category acquired from the shape category determination logic 34a based on the reliability of the insertion situation category acquired from the insertion situation learning model 33a. If the reliability is higher than the threshold, the insertion support information generating unit 36 ​​does not correct the insertion shape category included in the insertion situation category acquired from the insertion situation learning model 33a. On the other hand, if the reliability is lower than the threshold, the insertion support information generating unit 36 ​​corrects the insertion shape category included in the insertion situation category acquired from the insertion situation learning model 33a to the insertion shape category acquired from the shape category determination logic 34a.

[0076] As described above, there are cases where the shape information of the endoscope insertion part does not satisfy any of the conditions of the shape category determination logic 34a in the determination by the shape determination unit 34. When the shape information of the endoscope insertion part does not satisfy any of the conditions of the shape category determination logic 34a and the reliability of the insertion situation category acquired from the insertion situation learning model 33a is less than a threshold, the insertion support information generation unit 36 ​​determines that the insertion shape of the endoscope insertion part cannot be estimated.

[0077] During the period when the insertion shape of the endoscope insertion part cannot be estimated, the insertion support information generating unit 36 ​​does not display operation guide based on the insertion support information on the display device 41. The insertion support information generating unit 36 ​​may display a message such as "The insertion status of the endoscope cannot be recognized. Please operate carefully" on the display device 41.

[0078] In the explanation so far, it is assumed that the insertion status estimation process by the insertion status estimation unit 33, the insertion shape determination process by the shape determination unit 34, and the part estimation process by the part estimation unit 35 are all executed. In this regard, it is possible to omit one of the insertion status estimation process by the insertion status estimation unit 33 and the insertion shape determination process by the shape determination unit 34, depending on a predetermined condition. The predetermined condition may be a condition determined depending on the result of a priority determination that determines whether the output of the insertion status estimation unit 33 or the shape determination unit 34 is to be prioritized.

[0079] For example, the insertion status estimation unit 33 may perform the process of estimating the insertion status prior to the process of determining the insertion shape by the shape determination unit 34. When the reliability of the insertion status category acquired from the insertion status learning model 33a is higher than a threshold, the process of determining the insertion shape by the shape determination unit 34 may be omitted. In this case, the insertion support information generation unit 36 ​​adopts the insertion status category acquired from the insertion status learning model 33a as it is.

[0080] For example, the insertion shape determination process by the shape determination unit 34 may be executed prior to the insertion status estimation process by the insertion status estimation unit 33. When the insertion shape category can be acquired from the shape category determination logic 34a, the insertion status estimation process by the insertion status estimation unit 33 may be omitted. In this case, the insertion support information generation unit 36 ​​specifies the insertion shape category acquired from the shape category determination logic 34a and the insertion status category including the part acquired from the part learning model 35a.

[0081] Whether to first execute the insertion status estimation process by the insertion status estimation unit 33 or the insertion shape determination process by the shape determination unit 34 may be determined based on the most recent adoption rates of the insertion shape category included in the insertion status category acquired from the insertion status learning model 33a and the insertion shape category acquired from the shape category determination logic 34a. For example, if the most recent adoption rate of the insertion shape category acquired from the shape category determination logic 34a is higher, the insertion shape determination process by the shape determination unit 34 is executed first.

[0082] In this example, the priority information does not need to be previously assigned to each insertion state category prepared as an estimation result of the insertion state learning model 33a or each insertion shape category prepared as a judgment result of the shape category judgment logic 34a. This can be executed if the shape information of the endoscope insertion portion is input to either the insertion state estimation unit 33 or the shape judgment unit 34.

[0083] A priority determination unit may be provided that determines whether to give priority to the output of the insertion status estimation unit 33 or the shape determination unit 34, based on the shape information of the endoscope insertion portion acquired by the endoscope shape acquisition unit 31. In this case, the insertion support information generation unit 36 ​​may obtain the output of the insertion shape category from the one that is prioritized, between the insertion status estimation unit 33 and the shape determination unit 34. Furthermore, when the shape determination unit 34 is prioritized, the insertion support information generation unit 36 ​​may use a combination of the insertion shape category information from the shape determination unit 34 and the information on the site where the endoscope tip is located, from the site estimation unit 35. In other words, the output from the one that is not prioritized, between the insertion status estimation unit 33 and the shape determination unit 34, is not essential.

[0084] As described above, according to the first embodiment, it is possible to reduce erroneous recognition of the insertion status of the endoscope by combining the estimation result of the insertion status estimation unit 33 with at least one of the determination result of the shape determination unit 34 and the part estimation result of the part estimation unit 35. For example, when an insertion shape is prone to erroneous recognition by AI, it is possible to reduce erroneous recognition of the insertion shape by adopting the insertion shape determined based on the determination logic set by the designer.

[0085] Fig. 10 shows a configuration example of an endoscope insertion support system 30 according to embodiment 2. The endoscope insertion support system 30 according to embodiment 2 has a configuration in which a body position information acquisition unit 37 and an endoscope shape correction unit 38 are added to the configuration of the endoscope insertion support system 30 according to embodiment 1 shown in Fig. 3.

[0086] In the explanation of Fig. 1, an example has been shown in which the reference plate 20b on which the position sensor is arranged is connected to the endoscope insertion shape observation device 20. In the second embodiment, an example is assumed in which the reference plate 20b on which the position sensor is arranged is directly connected to the endoscope insertion support system 30. The shape information used by the insertion status estimation unit 33, the shape determination unit 34, and the site estimation unit 35 is based on the viewpoint from the abdominal side of the subject in the supine position. Therefore, if the subject's position changes during the examination, there is a possibility that the endoscope shape recognized will differ from the actual one.

[0087] The posture information acquiring unit 37 acquires, as posture information, three-dimensional information indicating the posture of the subject from the reference plate 20b. Note that the second embodiment is not limited to an example in which a posture sensor is attached to the subject to measure the subject's posture. For example, a plurality of pressure sensors may be installed on an examination bed, and the subject's posture may be estimated from the detection values ​​of the pressure sensors. Also, a camera may be installed in an examination room, and the subject's movements captured in a moving image captured by the camera may be tracked by image recognition to estimate the subject's posture.

[0088] The endoscope shape correction unit 38 corrects the shape information of the endoscope insertion part acquired by the endoscope shape acquisition unit 31, based on the body position information acquired by the body position information acquisition unit 37. Specifically, the endoscope shape correction unit 38 changes the shape information of the endoscope insertion part so as to cancel out the three-dimensional change in the posture of the subject defined by the body position information.

[0089] 11(a)-(b) are diagrams showing an example of an endoscope shape B3 before and after a change in body position. Fig. 11(a) shows an example of an endoscope shape B3 inserted into a subject in a supine position, viewed from a ceiling viewpoint. Fig. 11(b) shows an example of an endoscope shape B3 inserted into a subject in a left lateral position, viewed from a ceiling viewpoint. When the subject's body position changes, the endoscope shape B3 viewed from the ceiling viewpoint changes significantly.

[0090] 12 is a flowchart showing an example of the operation of the endoscope insertion support system 30 according to embodiment 2. The endoscope image acquisition unit 32 acquires an endoscope image from the endoscope system 10, the endoscope shape acquisition unit 31 acquires shape information of the endoscope insertion part from the endoscope insertion shape observation device 20, and the body position information acquisition unit 37 acquires body position information from the body position sensor (S10a).

[0091] The endoscope shape correction unit 38 corrects the direction of the acquired shape information based on the body position information. For example, the endoscope shape correction unit 38 corrects the acquired shape information so that it is always shape information from the viewpoint from the abdominal side of the subject (S15). The processing from step S20 onwards is the same as the processing in the flowchart shown in FIG.

[0092] As described above, in addition to the effects of the first embodiment, the second embodiment further provides the following effects. In the second embodiment, by acquiring body position information, it is possible to know the current body position of the subject. By correcting the shape information of the endoscope insertion part based on the body position information, it is possible to always handle shape information from the viewpoint of the abdominal side of the subject.

[0093] Fig. 13 is a diagram showing a configuration example of an endoscope insertion support system 30 according to embodiment 3. The endoscope insertion support system 30 according to embodiment 3 has a configuration in which an intestinal tract state estimating unit 39 is added to the configuration of the endoscope insertion support system 30 according to embodiment 1 shown in Fig. 3.

[0094] The intestinal tract condition estimation unit 39 inputs the endoscopic image acquired by the endoscopic image acquisition unit 32 to the intestinal tract condition learning model 39a, and acquires the condition relating to the distance between the tip of the endoscope and the intestinal wall of the subject (hereinafter referred to as the intestinal wall distance condition) from the intestinal tract condition learning model 39a. For example, the intestinal tract condition estimation unit 39 can acquire any one of "optimum distance," "clay pipe-like," and "red ball" as the intestinal wall distance condition. "Optimum distance" refers to a condition in which the distance between the tip of the endoscope and the intestinal wall is within a predetermined range. "Clay pipe-like" refers to a condition in which the lumen extends straight and visibility is good. "Red ball" refers to a condition in which the tip of the endoscope is in contact with or close to the intestinal wall and appears bright red.

[0095] The intestinal tract state estimation unit 39 can also input the endoscopic image acquired by the endoscopic image acquisition unit 32 to the intestinal tract state learning model 39a and acquire the state related to residue or water from the intestinal tract state learning model 39a. That is, the intestinal tract state estimation unit 39 can identify an endoscopic image showing residue or water. Furthermore, the intestinal tract state estimation unit 39 may be able to identify an endoscopic image showing cleaning fluid, bleeding, diverticulum, or a lesion (such as a polyp) in addition to residue or water.

[0096] The intestinal state learning model 39a is generated by machine learning using a supervised dataset of a large number of endoscopic images annotated with information on the intestinal wall distance state and whether or not residue, water, washing solution, bleeding, diverticula, or lesions are present.

[0097] The insertion support information generating unit 36 ​​generates insertion support information based on the insertion status category and the intestinal wall distance status, and displays it on the display device 41. By referring to the intestinal wall distance status, the insertion support information generating unit 36 ​​can grasp the insertion status in more detail, and can present more detailed insertion support information suitable for the insertion status.

[0098] For example, when the location where the endoscope tip is located is "sigmoid colon (more specifically, near the first curve on the rectal side of the sigmoid colon)" and the insertion shape of the endoscope insertion part is "no loop, tip curved to the right" in the insertion situation category, if the intestinal wall distance state has not been acquired, the insertion support information generating unit 36 ​​generates, as the insertion support information, a guide recommending a right torque operation. Similarly, when the intestinal wall distance state has been acquired and the intestinal wall distance state is "optimum distance", the insertion support information generating unit 36 ​​generates, as the insertion support information, a guide recommending a right torque operation. On the other hand, when the intestinal wall distance state is "clay pipe-like", the insertion support information generating unit 36 ​​generates, as the insertion support information, a guide recommending an operation of bringing the endoscope tip closer to the intestinal wall by a pull operation or suction.

[0099] The insertion support information generating unit 36 ​​can also generate insertion support information based on the insertion status category and the state related to residue or water. When residue or water is reflected in the endoscopic image, the insertion support information generating unit 36 ​​generates a guide recommending, for example, washing or suction.

[0100] Fig. 14 is a flowchart showing an example of the operation of the endoscope insertion support system 30 according to the third embodiment. The processes of steps S10 to S40 are similar to those of the flowchart shown in Fig. 9. The intestinal tract state estimation unit 39 inputs an endoscopic image acquired by the endoscopic image acquisition unit 32 to the intestinal tract state learning model 39a, and acquires an intestinal wall distance state from the intestinal tract state learning model 39a (S42). The processes of steps S50 to S80 are similar to those of the flowchart shown in Fig. 9. The insertion support information generation unit 36 ​​acquires insertion support information linked to the finally determined combination of the insertion status category and the intestinal wall distance state, and displays the acquired insertion support information on the display device 41 (S90a).

[0101] The insertion situation learning model 33a may be generated by machine learning using a combination of a large number of endoscope images (with intestinal wall distance states) and a large number of shape information pieces of the endoscope insertion portion, to which the insertion situation category of the endoscope insertion portion is annotated, as a supervised data set. In this case, the insertion situation estimation unit 33 inputs the shape information of the endoscope insertion portion, the corresponding endoscope images, and the corresponding intestinal wall distance states to the insertion situation learning model 33a, and acquires the insertion situation category from the insertion situation learning model 33a.

[0102] As described above, the third embodiment has the following advantages in addition to the advantages of the first embodiment: In the third embodiment, by referring to the intestinal tract state, more detailed insertion support information can be presented.

[0103] Fig. 15 shows a configuration example of an endoscope insertion support system 30 according to embodiment 4. The endoscope insertion support system 30 according to embodiment 4 has a configuration in which a pain estimation unit 310 is added to the configuration of the endoscope insertion support system 30 according to embodiment 1 shown in Fig. 3.

[0104] The pain estimation unit 310 generates pain information indicating pain experienced by the subject based on the insertion shape of the endoscope insertion part and the location where the endoscope tip is located. The pain information includes at least the presence or absence of pain. It may also include the degree of pain. A table is prepared in advance in which pain information is defined for each combination of the insertion shape of the endoscope insertion part and the location where the endoscope tip is located. The location where the endoscope tip is located estimated by the location estimation unit 35 can be used as the location where the endoscope tip is located.

[0105] The insertion shape of the endoscope insertion part may use the insertion shape category determined by the shape determination unit 34, or may use an insertion shape classified by a category classification different from the category classification of the insertion shape by the shape determination unit 34. In the latter case, a learning model of the insertion shape for pain estimation is prepared, or a determination logic for the insertion shape for pain estimation is prepared separately.

[0106] The insertion support information generating unit 36 ​​generates insertion support information based on the insertion state category and the pain information, and displays the insertion support information on the display device 41. By referring to the pain information, it becomes possible to present insertion support information specific to the occurrence of pain.

[0107] Fig. 16 is a flowchart showing an example of the operation of the endoscope insertion support system 30 according to the fourth embodiment. The processes in steps S10 to S40 are the same as those in the flowchart shown in Fig. 9. The pain estimation unit 310 generates pain information based on the shape information of the endoscope insertion portion acquired by the endoscope shape acquisition unit 31 and the part where the endoscope tip is located estimated by the part estimation unit 35 (S44). The processes in steps S50 to S80 are the same as those in the flowchart shown in Fig. 9. The insertion support information generation unit 36 ​​generates insertion support information according to the finally determined insertion state category and pain information, and displays the generated insertion support information on the display device 41 (S90b).

[0108] As described above, in addition to the effects of embodiment 1, embodiment 4 further provides the following effects: In embodiment 4, by referring to pain information, more detailed insertion support information can be presented.

[0109] Fig. 17 shows a configuration example of an endoscope insertion support system 30 according to embodiment 5. The endoscope insertion support system 30 according to embodiment 5 has a configuration in which the insertion support information generator 36 of the endoscope insertion support system 30 according to embodiment 1 shown in Fig. 3 is replaced with an insertion control information generator 311.

[0110] The insertion control information generator 311 generates insertion control information based on the insertion status category, and transmits the generated insertion control information to the endoscope control device 50. The endoscope control device 50 is, for example, a fully-automatic or semi-automatic endoscope operation robot. Based on the insertion control information received from the endoscope insertion support system 30, the endoscope control device 50 performs the insertion operation of the endoscope insertion portion fully-automatically or semi-automatically.

[0111] As described above, according to the fifth embodiment, the possibility of transmitting an erroneous operation instruction to the endoscope control device 50 can be reduced.

[0112] The present disclosure has been described above based on a number of embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing step, and that such modifications are also within the scope of the present disclosure.

[0113] In the above-mentioned embodiments 2 to 5, similarly to the embodiment 1, it is also possible to omit one of the insertion status estimation process by the insertion status estimation unit 33 and the insertion shape determination process by the shape determination unit 34, depending on a predetermined condition. The predetermined condition may be a condition determined depending on the result of a priority determination that determines whether the output of the insertion status estimation unit 33 or the shape determination unit 34 is to be prioritized.

[0114] In the above embodiment, an example has been described in which the endoscope shape is estimated by incorporating multiple magnetic coils in the endoscope 11. In this regard, the endoscope shape may be estimated by incorporating multiple shape sensors in the endoscope 11. The shape sensor may be, for example, a fiber sensor that detects the bending shape from the curvature of a specific location using an optical fiber. The fiber sensor has, for example, an optical fiber arranged along the longitudinal direction of the insertion portion 11a, and the optical fiber is provided with multiple photodetectors along the longitudinal direction. Detection light is supplied from the detection light emitting device to the optical fiber, and the endoscope shape is estimated based on changes in the amount of light detected by each photodetector when the detection light propagates through the optical fiber. [Industrial Applicability]

[0115] The present disclosure can be used for colonoscopy. [Explanation of symbols]

[0116] 10 endoscope system, 11 endoscope, 11a insertion section, 11b rigid tip section, 11c bending section, 11d flexible tube section, 11e operation section, 11f main body section, 11g grip section, 12 magnetic coil, 15 light source device, 20 endoscope insertion shape observation device, 20a receiving antenna, 20b reference plate, 30 endoscope insertion support system, 31 endoscope shape acquisition section, 32 endoscope image acquisition section, 33 insertion status estimation unit, 33a...insertion situation learning model, 34...shape determination unit, 34a...shape category determination logic, 35...part estimation unit, 35a...part learning model, 36...insertion assistance information generation unit, 37...position information acquisition unit, 38...endoscope shape correction unit, 39...intestinal tract state estimation unit, 39a...intestinal tract state learning model, 310...pain estimation unit, 311...insertion control information generation unit, 41...display unit, 42...input device, 43...storage device, 50...endoscope control device.

Claims

1. An endoscope insertion assistance system, comprising: one or more processors having hardware; The processor, Acquire shape information of the endoscope insertion part during endoscopic examination, According to a priority condition, at least one of inputting the shape information into an insertion situation learning model and acquiring an insertion situation category that categorizes the insertion situation of the endoscope insertion part from the insertion situation learning model, and applying the shape information to a shape category determination logic and acquiring an insertion shape category that categorizes the insertion shape of the endoscope insertion part. Endoscope insertion assistance system.

2. The endoscope insertion support system according to claim 1, The processor, inputting the shape information into the insertion situation learning model, and acquiring, from the insertion situation learning model, an insertion situation category obtained by categorizing an insertion situation of the endoscope insertion portion; applying the shape information to the shape category determination logic to obtain an insertion shape category into which the insertion shape of the endoscope insertion portion is categorized; when the insertion shape category included in the insertion status category does not match the insertion shape category acquired from the shape category determination logic, correcting the insertion shape category included in the insertion status category to the insertion shape category acquired from the shape category determination logic; Endoscope insertion assistance system.

3. The endoscope insertion support system according to claim 1, The processor generates insertion assistance information based on the insertion status category, and displays the insertion assistance information on a monitor. Endoscope insertion assistance system.

4. The endoscope insertion support system according to claim 1, The processor generates insertion control information based on the insertion status category. Endoscope insertion assistance system.

5. The endoscope insertion support system according to claim 1, The processor, Acquire an endoscopic image during the endoscopic examination; inputting the endoscopic image into a region learning model, and acquiring a region of the subject shown in the endoscopic image from the region learning model; Endoscope insertion assistance system.

6. The endoscope insertion support system according to claim 5, The processor, inputting the shape information into the insertion situation learning model, and acquiring, from the insertion situation learning model, an insertion situation category obtained by categorizing an insertion situation of the endoscope insertion portion; An endoscopic image obtained during the endoscopic examination is input to a region learning model, and a region of the subject shown in the endoscopic image is acquired from the region learning model; When the part of the subject included in the insertion state category does not match the part of the subject acquired from the part learning model, the part of the subject included in the insertion state category is corrected to the part of the subject acquired from the part learning model. Endoscope insertion assistance system.

7. The endoscope insertion support system according to claim 2, The processor, Acquire an endoscopic image during the endoscopic examination; inputting the shape information and the endoscope image into the insertion situation learning model, and acquiring the insertion situation category from the insertion situation learning model; Endoscope insertion assistance system.

8. The endoscope insertion support system according to claim 5, The processor inputs the endoscopic image and the shape information into the part learning model, and acquires the part of the subject shown in the endoscopic image from the part learning model. Endoscope insertion assistance system.

9. The endoscope insertion support system according to claim 1, The processor, inputting the shape information into the insertion situation learning model, and acquiring, from the insertion situation learning model, an insertion situation category obtained by categorizing an insertion situation of the endoscope insertion portion; applying the shape information to the shape category determination logic to obtain an insertion shape category into which the insertion shape of the endoscope insertion portion is categorized; determining whether or not to correct an insertion shape category included in the insertion status category to an insertion shape category acquired from the shape category determination logic based on the shape information; Endoscope insertion assistance system.

10. The endoscope insertion support system according to claim 1, The processor, inputting the shape information into the insertion situation learning model, and acquiring, from the insertion situation learning model, an insertion situation category obtained by categorizing an insertion situation of the endoscope insertion portion; applying the shape information to the shape category determination logic to obtain an insertion shape category into which the insertion shape of the endoscope insertion portion is categorized; determining whether or not to correct the insertion shape category included in the insertion status category to the insertion shape category acquired from the shape category determination logic based on priority information assigned to at least one of the insertion status category and the insertion shape category; Endoscope insertion assistance system.

11. The endoscope insertion support system according to claim 1, The processor, inputting the shape information into the insertion situation learning model, and acquiring an insertion situation category obtained by categorizing an insertion situation of the endoscope insertion portion from the insertion situation learning model; applying the shape information to the shape category determination logic to obtain an insertion shape category into which the insertion shape of the endoscope insertion portion is categorized; determining whether or not to correct an insertion shape category included in the insertion situation category to an insertion shape category obtained from the shape category determination logic based on the reliability of the insertion situation category obtained from the insertion situation learning model; Endoscope insertion assistance system.

12. The endoscope insertion support system according to claim 11, The processor, if the shape information does not satisfy any of the conditions of the shape category determination logic and the reliability of the insertion state category is less than a threshold, it is determined that the insertion shape of the endoscope insertion portion cannot be estimated. Endoscope insertion assistance system.

13. The endoscope insertion support system according to claim 1, The shape category determination logic is a determination logic determined by a designer. Endoscope insertion assistance system.

14. The endoscope insertion support system according to claim 6, The processor, acquiring a time series of endoscopic images; When the part of the subject detected by the part learning model is not detected continuously for a predetermined time, the detected part of the subject is invalidated. Endoscope insertion assistance system.

15. The endoscope insertion support system according to claim 1, The processor, Acquire posture information of the subject; correcting the shape information based on the body position information; Endoscope insertion assistance system.

16. The endoscope insertion support system according to claim 1, The processor, Acquire an endoscopic image during the endoscopic examination; determining a state relating to the distance between the tip of the endoscope and the intestinal wall of the subject; generating insertion assistance information or insertion control information based on the insertion status category and a state related to a distance between the distal end portion of the endoscope and an intestinal wall of the subject; Endoscope insertion assistance system.

17. The endoscope insertion support system according to claim 1, The processor, Acquire an endoscopic image during the endoscopic examination; determining a state relating to the distance between the tip of the endoscope and the intestinal wall of the subject; inputting the shape information, the endoscopic image, and a state related to a distance between the distal end portion of the endoscope and an intestinal wall of a subject into the insertion situation learning model, and acquiring the insertion situation category from the insertion situation learning model; Endoscope insertion assistance system.

18. The endoscope insertion support system according to claim 1, The processor, Acquire an endoscopic image during the endoscopic examination; Determine the residue or water condition; generating insertion assistance information or insertion control information based on the insertion status category and a state related to the residue or the water; Endoscope insertion assistance system.

19. The endoscope insertion support system according to claim 1, the processor generates pain information indicating pain caused to the subject based on the insertion shape of the endoscope insertion portion and a location where the endoscope tip portion is located, and displays the pain information on a monitor. Endoscope insertion assistance system.

20. One or more processors having hardware, Obtaining shape information of an endoscope insertion portion during endoscopic examination; According to a priority condition, at least one of inputting the shape information into an insertion situation learning model to obtain an insertion situation category obtained by categorizing the insertion situation of the endoscope insertion part from the insertion situation learning model, and applying the shape information to a shape category determination logic to obtain an insertion shape category obtained by categorizing the insertion shape of the endoscope insertion part; A method for operating an endoscope insertion assist system comprising:

21. A process of acquiring shape information of an endoscope insertion part during an endoscopic examination; A process of executing at least one of the following, according to a priority condition: inputting the shape information into an insertion situation learning model and acquiring an insertion situation category that categorizes the insertion situation of the endoscope insertion part from the insertion situation learning model; and applying the shape information to a shape category determination logic and acquiring an insertion shape category that categorizes the insertion shape of the endoscope insertion part. A storage medium that stores a program that causes a computer to execute the above.

Images