Endoscopic examination support system, method of operation of the endoscopic examination support system, and program

Abstract

To provide an endoscopic examination support system that can predict the risk of unintentional movements occurring. [Solution] The endoscopic examination support system 100 includes a processor 110 that performs inference using a trained model 130. The trained model 130 is trained to output risk information by taking multiple training endoscopic images taken in the gastrointestinal tract as input. The risk information indicates the risk of changes in the position of the endoscope tip occurring at each position of the endoscope when each training endoscopic image was taken, regardless of endoscopic manipulation. The processor 110 inputs multiple endoscopic images taken in the gastrointestinal tract in real time during the endoscopic examination to the trained model 130, causing the trained model 130 to output risk information.

Description

Technical Field

[0005] , ,

[0001] The present invention relates to an endoscopy support system, an operation method of the endoscopy support system, a program, and the like.

Background Art

[0002] Patent Document 1 discloses a technique assuming hand tremors of an endoscope in super-resolution processing. A plurality of low-resolution images with displaced imaging positions are acquired using hand tremors, and the plurality of low-resolution images are used for super-resolution processing. In this case, a marker portion is provided on the endoscope, and by using this marker portion as a reference, the alignment accuracy in super-resolution processing can be improved.

[0003] Patent Document 2 discloses a technique for controlling an input instruction to an electric scope of an endoscope. A sensor for detecting the operator's intention to perform a bending operation is provided at the position of the grip portion of the electric scope. Based on the detected intention to perform a bending operation, the input instruction to the electric scope is controlled.

Prior Art Documents

Patent Documents

[0004] <​​​​​​​​​​​​​​​​​In endoscopes, if the endoscope moves unintended by the operator, the position of the endoscope tip changes suddenly, and the field of view of the endoscope changes abruptly without the operator's intent. Patent Document 1 is a technology related to super-resolution processing using hand tremors, and does not detect or predict the occurrence of hand tremors. Patent Document 2 is a technology that detects the operator's intent and controls input instructions to the motorized endoscope, and does not detect or predict unintended endoscope movements by the operator. [Means for solving the problem]

[0006] One aspect of this disclosure relates to an endoscopic examination support system that includes a processor for performing inference using a trained model, wherein the trained model is trained to take a plurality of training endoscopic images taken in the gastrointestinal tract as input and output risk information for the occurrence of changes in the position of the endoscope tip that are independent of endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken, and the processor inputs a plurality of endoscopic images taken in the gastrointestinal tract in real time during an endoscopic examination to the trained model, thereby causing the trained model to output the risk information.

[0007] Other aspects of this disclosure relate to a method of operating an endoscopy support system, which includes: the endoscopy support system acquiring multiple endoscopic images taken in real time within the gastrointestinal tract during an endoscopy; and the endoscopy support system inputting the multiple endoscopic images taken in the gastrointestinal tract as input to a trained model that has been trained to output risk information regarding the occurrence of changes in the position of the endoscope tip that are independent of endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken, thereby causing the trained model to output the risk information.

[0008] Further aspects of this disclosure relate to a program that causes a computer to perform the following actions: acquire multiple endoscopic images taken in real time within the gastrointestinal tract during an endoscopic examination; and input the multiple endoscopic images taken within the gastrointestinal tract into a trained model that has been trained to output risk information regarding the occurrence of changes in the position of the endoscope tip that are independent of endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken, thereby causing the trained model to output the risk information. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram illustrating the general outline of risk prediction. [Figure 2] The first example of a situation in which unintentional actions occur. [Figure 3] A second example of a situation in which unintentional actions occur. [Figure 4] A third example of a situation in which unintentional actions occur. [Figure 5] The fourth example of a situation in which unintentional actions occur. [Figure 6] The fifth example of a situation in which unintentional actions occur. [Figure 7] An example of a learning system configuration. [Figure 8] An example of an endoscope system configuration. [Figure 9] A diagram illustrating the flow of generating training data. [Figure 10] An example of training data in the first embodiment. [Figure 11] Learning flow in the first embodiment. [Figure 12] The inference flow in the first embodiment. [Figure 13] An example of training data in the second embodiment. [Figure 14] Learning flow in the second embodiment. [Figure 15] The inference flow in the second embodiment. [Figure 16] An example of training data in the third embodiment. [Figure 17]A diagram showing a situation where an operator operates an endoscope to examine a subject lying on an examination table. [Figure 18] The learning flow in the third embodiment. [Figure 19] The inference flow in the third embodiment. [Figure 20] The second configuration example of the endoscope examination support system. [Figure 21] The flow at the time of insertion in the fourth embodiment. [Figure 22] An example of recorded data. [Figure 23] The flow at the time of removal in the fourth embodiment. [Figure 24] The processing flow in the fifth embodiment. [Figure 25] An example of the scope shape obtained by UPD. [Figure 26] The processing flow of the sixth embodiment. [Figure 27] The flow at the time of removal in the eighth embodiment.

Embodiments for Carrying out the Invention

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail. Note that the embodiments described below do not unduly limit the content described in the claims, and not all of the configurations described in these embodiments are essential constituent elements.

[0011] 1. Basic Method Hereinafter, an example of using an endoscope examination support system for a large intestine examination will be described. However, the endoscope examination support system can also be used for examinations of other digestive tracts such as the stomach, duodenum, or small intestine. In addition, although it is assumed that the examination is performed while removing the endoscope in a large intestine examination, the endoscope examination support system can be used for both insertion and removal of the endoscope into the digestive tract.

[0012] Figure 1 is a diagram illustrating the general risk prediction performed by the endoscopic examination support system. The operator inserts the tip of the endoscope 10 into the ileocecal region of the large intestine 1, and then examines the intestinal tract while withdrawing the endoscope 10. In the large intestine 1, there is a position 5 where unintended movements of the endoscope 10 are likely to occur due to the interaction between the endoscope 10 and the digestive tract wall. Hereafter, unintended movements of the endoscope will be referred to as unintentional movements. In Figure 1, position 5 is shown as being near the hepatic flexure, but as will be described later in Figures 2 to 6, unintentional movements may occur in various situations.

[0013] The endoscopic examination support system estimates the risk of unintentional movement of the endoscope 10 based on information such as endoscopic images, and presents this risk information. Specifically, the endoscopic examination support system informs the operator that there is a high risk of unintentional movement when the tip of the endoscope 10 approaches position 5. This allows the operator to know in advance that there is a possibility of unintentional movement of the endoscope 10. For example, when the operator is informed of a high risk, they can operate the endoscope 10 in a way that minimizes the likelihood of unintentional movement.

[0014] The following describes specific examples of unintentional movements. An unintentional movement is a change (movement) in the tip position of the endoscope 10 that occurs even though no corresponding operational input has been applied to the control unit of the endoscope 10. Furthermore, an unintentional movement is a change (movement) in the tip position of the endoscope 10 that occurs due to the interaction between the endoscope 10 and the digestive tract wall, rather than due to operational factors such as hand tremor, misoperation, or dropping the control unit. Figures 2 to 6 show specific examples of unintentional movements.

[0015] Figure 2 shows the first example of a situation in which unintentional movement occurs. The large intestine 1 includes both a wide and a narrow section of the intestinal tract. As shown in the upper diagram, the operator is observing the intestinal wall of the wide section of the intestinal tract by angled the curved tip of the endoscope 10.

[0016] As shown in the middle diagram, as the operator withdraws the endoscope 10, the curved tip of the endoscope 10 enters the narrow part of the intestinal tract. At this time, the curved tip becomes unangled due to the pressure from the intestinal tract, and the camera of the endoscope 10 points in the direction of depth of the intestinal tract. This change in angle is unintended by the operator and constitutes an unintentional action.

[0017] As shown in the lower diagram, as the operator further withdraws the endoscope 10, the curved tip of the endoscope 10 enters the wider part of the intestinal tract. At this time, the curved tip is released from the pressure it receives from the intestinal tract and abruptly returns to its angled position, and the camera of the endoscope 10 points towards the intestinal wall. This abrupt return to the angle is unintended by the operator and constitutes an unintentional action.

[0018] Furthermore, the thicker portion of the intestinal tract described above may be a flexible portion where the intestinal tract is not fixed. In flexible portions, the curved tip of the endoscope 10 is not subjected to pressure from the intestinal tract, so the angle of the curved tip is maintained. Also, similar unintentional movements may occur in parts where the angled curved tip gets caught and stretched, such as bends in the intestinal tract.

[0019] Figure 3 shows a second example of a situation in which unintentional movement occurs. As shown in the left figure, a portion of the insertion part of the endoscope 10 is in contact with a hard part of the intestinal tract of the large intestine 1 and is supported by the reaction force received from the intestinal wall. At this time, the position of the insertion part on the tip side of the supported portion is fixed.

[0020] As shown in the diagram on the right, as the operator withdraws the endoscope 10, the part supported by the intestinal tract moves to the softer part of the intestinal tract. As a result, the insertion part loses support, the movement of the endoscope 10 becomes unstable, and the position of the endoscope tip changes abruptly. Such a change in the position of the endoscope tip is unintended by the operator and constitutes an unintentional movement.

[0021] Figure 4 shows a third example of a situation in which unintentional movement occurs. The endoscope 10 includes an active bending section 11 that bends due to the operator's angle manipulation, and a passive bending section 12 provided on the base end side of the active bending section 11 that bends passively due to external forces, etc. Specifically, the active bending section 11 includes a plurality of sprockets connected in series in the axial direction and pivotably connected to each other. A wire connecting the operating dial and the active bending section 11 passes through the tube of the passive bending section 12. When the operator turns the operating dial, tension is applied to the wire, causing the plurality of sprockets to pivot and the active bending section 11 to bend.

[0022] As shown in the left diagram, the operator is observing the depth of the intestinal tract of the large intestine 1 with the active bending section 11 in a straight position. The active bending section 11 is located beyond the bend in the intestinal tract, and the passive bending section 12 is curved along the bend. As shown in the right diagram, as the operator withdraws the endoscope 10, the active bending section 11 moves towards the bend. Since the active bending section 11 does not bend passively, it tries to maintain its straight position, causing the camera's orientation to change and the camera to point towards the intestinal wall. Such movement of the endoscope 10 is unintended by the operator and is therefore an unintentional action.

[0023] Figure 5 shows a fourth example of a situation in which unintentional movement occurs. The large intestine 1 includes a wide section and a narrow section of the intestinal tract. As shown in the upper diagram, the operator is observing the wide section of the intestinal tract with the active bending section 11 in a straight position. The passive bending section 12 extends from the narrow section to the wide section of the intestinal tract, and the active bending section 11 is located in the wide section of the intestinal tract. The portion of the passive bending section 12 that extends into the wide section of the intestinal tract bends due to gravity, etc. As a result, the active bending section 11 faces the direction of the intestinal wall, and the camera faces the direction of the intestinal wall.

[0024] As shown in the lower diagram, as the operator withdraws the endoscope 10, the active bending section 11 enters the narrow part of the intestinal tract. Because the active bending section 11 is oriented along the direction of the narrow intestinal tract, the camera is oriented in the direction of depth of the intestinal tract. This movement of the endoscope 10 is unintended by the operator and is therefore an unintentional movement.

[0025] Figure 6 shows the fifth example of a situation in which unintentional movement occurs. Assume that subject 2 is lying on examination table 3 in a left lateral decubitus position. As shown in the left figure, when a certain length of the insertion portion from the tip of the endoscope 10 is in the ascending colon, the endoscope 10 is stabilized by the weight of the insertion portion being supported by the intestinal tract.

[0026] As shown in the figure on the right, as the operator withdraws the endoscope 10, the tip of the endoscope 10 approaches the hepatic flexure. At this time, the insertion portion near the tip of the endoscope 10 loses support from the ascending colon and becomes able to move freely within the transverse colon. In the left lateral decubitus position, gravity acts in the direction along the transverse colon, so the insertion portion within the transverse colon moves due to gravity, causing the camera's field of view to change abruptly. Such movement of the endoscope 10 is unintended by the operator and is therefore an unintentional movement. It should be noted that similar unintentional movements may occur depending on the position of the subject 2 and the tip position of the endoscope 10, not limited to the above. Furthermore, the unintentional movement in Figure 6 is not limited to when the endoscope 10 moves within the space of the transverse colon, but may also occur when the transverse colon itself moves due to the weight of the endoscope 10, causing the endoscope 10 to move together with the transverse colon.

[0027] 2. Risk detection using AI The following describes an embodiment in which an endoscopic examination support system uses AI to detect risks. Figures 7 to 12 are explanatory diagrams of the first embodiment.

[0028] Figure 7 shows an example configuration of a learning system 500 that trains a machine learning model used for risk detection. The learning system 500 includes a processor 510 and memory 520. The learning system 500 is, for example, an information processing device such as a PC. Alternatively, the learning system 500 may be a cloud system accessed from a terminal via a network. A processor included in one or more information processing devices that constitute the cloud system corresponds to the processor 510 of the learning system 500.

[0029] Memory 520 stores the learning model 530, training data 550, and program 540. Endoscopic images taken for training are called training endoscopic images. Processor 510 performs training on the learning model 530 using training data 550, which includes image data 570 of the training endoscopic images. Specifically, processor 510 inputs the training endoscopic images to the learning model 530, and the learning model 530 infers risk information indicating the risk of unintentional actions from the training endoscopic images. Processor 510 calculates the error between the inferred risk information and the risk information based on training data 550, and updates the parameters of the learning model 530 to reduce this error. The trained learning model 530 is used as the trained model 130 in Figure 8. The content of this training process is described in program 540, and the training process is realized when processor 510 executes program 540.

[0030] Figure 8 shows an example configuration of the endoscope system 300. The endoscope system 300 includes an endoscope 10, a video processor 250, a display 290, an endoscopy support system 100, and a display 190.

[0031] The endoscope 10 is a flexible endoscope inserted into the digestive tract to image the inside of the digestive tract. The endoscope 10 includes an insertion section inserted into the body cavity, an operating section connected to the proximal end of the insertion section, a universal cord connected to the proximal end of the operating section, and a connector section connected to the proximal end of the universal cord. The tip of the insertion section is provided with a camera for imagery of the inside of the body cavity and an illumination optical system for illuminating the inside of the body cavity. The camera includes an objective optical system and an image sensor that captures the subject imaged by the objective optical system. The connector section detachably connects a transmission cable to the video processor 250. Images captured by the endoscope 10 will be referred to as endoscopic images.

[0032] The video processor 250 is a processing unit that controls the endoscope, processes endoscopic images, and displays endoscopic images. The video processor 250 is composed of a processor such as a CPU, processes the image signals transmitted from the endoscope 10 to generate image data of the endoscope, and outputs the image data to the display 290 and the endoscopic examination support system. The endoscope system 300 includes a light source device (not shown) that generates and controls illumination light. The light source device may be housed in the same enclosure as the video processor 250, or in a separate enclosure. The illumination light emitted by the light source device is guided by a light guide to the illumination optical system of the endoscope 10, and then emitted from the illumination optical system into the body cavity.

[0033] The endoscopic examination support system 100 includes a processor 110 and memory 120. The endoscopic examination support system 100 is, for example, an information processing device such as a PC. Alternatively, the endoscopic examination support system 100 may be a cloud system connected to the video processor 250 via a network. A processor included in one or more information processing devices constituting the cloud system corresponds to the processor 110 of the endoscopic examination support system 100. Alternatively, the endoscopic examination support system 100 may be built into the video processor 250.

[0034] Memory 120 stores the trained model 130 and the program 140. Processor 110 inputs the endoscopic image from the video processor 250 to the trained model 130, which infers risk information indicating the risk of unintentional movement from the endoscopic image and outputs that risk information. Processor 110 provides information to the operator based on the risk information. The risk information is a risk value that represents the risk information in stages or continuously. Processor 110 presents the risk value to the operator, or presents the operator with the result of comparing the risk value with a threshold as the presence or absence of risk. Presentation methods include monitor display, sound, vibration, or lamp. Monitor display may be, for example, a mark, icon, character, or bar display. Processor 110 superimposes these onto the endoscopic image and displays them on display 190. Note that display 190 and display 290 may be shared. The details of the above support process are described in program 140, and the support process is realized when the processor 110 executes program 140.

[0035] Furthermore, a non-temporary information storage medium, which is a computer-readable medium, may store the program 140 or the learned model 130. The information storage medium may be, for example, an optical disc, a memory card, a hard disk drive, or a semiconductor memory. The semiconductor memory may be, for example, ROM or non-volatile memory.

[0036] The hardware configuration of the processor and memory is described below. Here, the processor 510 and memory 520 of the learning system 500 and the processor 110 and memory 120 of the endoscopy support system 100 are referred to as "processor" and "memory" respectively, with their reference numerals omitted. The processor includes hardware. The processor is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microcomputer, or a DSP (Digital Signal Processor). Alternatively, the processor may be an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor may consist of one or more of the CPU, GPU, microcomputer, DSP, ASIC, and FPGA. The memory is, for example, a semiconductor memory which is volatile memory or non-volatile memory. Alternatively, the memory may be a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive.

[0037] An example of a learning model 530 and a trained model 130 is described below. Here, the learning model 530 and the trained model 130 are simply referred to as "models." A model is, for example, a neural network learned by deep learning. In this case, the model includes a program describing the algorithm of the neural network, and weight parameters between the nodes of the neural network. The neural network includes an input layer into which input data is input, an intermediate layer that performs calculations on the data input through the input layer, and an output layer that outputs inference results based on the calculation results output from the intermediate layer. In this embodiment, since inference is performed using the relationships of time-series images, an RNN (Recurrent Neural Network) is assumed as the neural network. However, various models applicable to image processing may be adopted.

[0038] Figure 9 illustrates the flow of training data generation. While this example shows training data being generated automatically, it may also be generated through manual annotation. Furthermore, while this example shows the training data being generated by the learning system 500, a different information processing system may also generate the training data.

[0039] In step S1, the processor 510 acquires the current frame of the training endoscopic image from the image data 570. Here, the current frame refers to the frame being processed at the current stage when processing time-series frames sequentially.

[0040] In step S2, the processor 510 acquires the previous frame's training endoscopic image from the image data 570. The previous frame is a frame that precedes the current frame in time, for example, the frame immediately preceding the current frame.

[0041] In step S3, the processor 510 estimates the scope movement distance between the current frame and the previous frame from the training endoscopic images of the current frame and the previous frame. The scope movement distance is estimated from the amount of movement of the subject within the image, specifically the amount of movement of the feature points in the image. For example, the processor 510 determines the optical flow as the scope movement distance.

[0042] In step S4, the processor 510 calculates the scope position based on the scope movement distance. For example, the processor 510 accumulates the scope movement distance for each frame and uses the accumulated value as the scope position. Alternatively, the processor 510 may estimate the scope position from UPD data. UPD stands for Endoscope Insertion Shape Observation Device.

[0043] In step S5, the processor 510 detects an unintentional action event based on the scope movement distance estimated in step S3. When the scope movement distance is greater than or equal to a predetermined value, the processor 510 considers that an unintentional action has occurred and sets an event flag.

[0044] In step S6, the processor 510 adds the scope position data and an event flag for unintentional movement as metadata to the current frame's training endoscopic image, and records the training endoscopic image with the added metadata as training data 550 in memory 520. In this way, video data with metadata is created for each case. Note that recording is not limited to metadata; it is sufficient if the frame, scope position data, and event flag are associated and recorded in the training data 550.

[0045] Figure 10 shows an example of training data in the first embodiment. As shown in the table in Figure 10, each frame is assigned a frame number 0, 1, 2, ..., 30000, and each frame is assigned metadata including the scope tip position and an event flag. Scope tip position "0" indicates the starting position of the examination. In a colonoscopy, the operator usually withdraws the endoscope after the endoscope tip reaches the ileocecal region. Therefore, scope tip position "0" represents the position of the ileocecal region. The numerical value of the scope tip position may be the same value obtained in processing S4 in Figure 9, or it may be a value converted to a unit such as centimeters. The event flag is a label indicating the presence or absence of unintentional action; for example, "0" indicates "none" and "1" indicates "present".

[0046] Figure 11 shows the learning flow in the first embodiment. The inference unit 530a corresponds to the learning model 530 executed by the processor 510.

[0047] In step S21, the processor 510 obtains the scope tip position (current position) of frame number k and the scope tip position where the event flag is 1 (event occurrence position). k = 0, 1, 2, ..., 30000. The event occurrence position is, for example, the nearest scope tip position with an event flag of 1 that is anal-side from the current position. Alternatively, the event occurrence position is a scope tip position with an event flag of 1 that is located within a predetermined distance anal-side from the current position. From the current position and the event occurrence position, the processor 510 calculates a calculated risk value indicating the risk of unintentional action occurring. The smaller the difference between the current position and the event occurrence position, the larger the calculated risk value. For example, (calculated risk value) = (coefficient) × ((scope tip position) - (current position)), but the method for calculating the calculated risk value is not limited to this.

[0048] The inference unit 530a acquires image data of the training endoscopic image with frame number k and infers a risk value from that image data. In step S22, the processor 510 calculates the error between the inferred value and the calculated risk value and updates the parameters of the training model 530 to minimize this error.

[0049] Alternatively, the calculated risk value may be determined in advance before training and attached as metadata to each frame of the training data 550 video. In that case, step S21 is omitted. In step S22, the processor 110 obtains the calculated risk value from the training data 550 and calculates the error between the calculated risk value and the inferred value.

[0050] Figure 12 shows the inference flow in the first embodiment. The inferencer 130a corresponds to the trained model 130 executed by the processor 110.

[0051] In step S41, the processor 110 acquires frame image data of the motion image data captured in real time by the endoscope as image data of the endoscope. The inference unit 130a infers a risk value indicating the risk of unintentional action from the image data. In step S43, the processor 510 provides a risk notification based on the inferred risk value. The risk presentation method may vary as described above.

[0052] Figures 13 to 15 are explanatory diagrams of the second embodiment. The configuration of the learning system 500 and the endoscopic examination support system 100 is the same as in the first embodiment. The differences from the first embodiment will be described below.

[0053] Figure 13 shows an example of training data in the second embodiment. In the second embodiment, UPD data is further added to each frame as metadata. UPD data is data indicating the insertion shape of the endoscope in each frame and is acquired by UPD. Multiple coils are provided at predetermined intervals along the axial direction in the insertion part of the endoscope, and UPD observes the position of each coil from outside the body. UPD data is data indicating the three-dimensional position of each observed coil. For example, in the UPD data for frame number 0, (1,10,5) indicates the three-dimensional coordinates of the first coil, (5,18,16) indicates the three-dimensional coordinates, and (10,0,4) indicates the three-dimensional coordinates of the last coil.

[0054] Figure 14 shows the learning flow in the second embodiment. The inference unit 530a acquires image data and UPD data of the training endoscope image with frame number k, and infers a risk value from the image data and UPD data.

[0055] Figure 15 shows the inference flow in the second embodiment. In step S42, the processor 110 acquires UPD data from the UPD. The inference unit 530a infers a risk value indicating the risk of unintentional movement from the image data and UPD data. By using the scope shape indicated by the UPD data in the inference, the risk of unintentional movement estimated from the scope shape can be estimated. For example, the risk of unintentional movement associated with the interaction between the scope and the large intestine, such as the reaction force from the large intestine as explained in Figure 3, can be estimated.

[0056] Figures 16 to 19 are explanatory diagrams of the third embodiment. The configuration of the learning system 500 and the endoscopic examination support system 100 is the same as in the first embodiment. Note that the third embodiment and the second embodiment may be combined. The differences from the first embodiment will be described below.

[0057] Figure 16 shows an example of training data in the third embodiment. In the third embodiment, each frame is further accompanied by the subject's body position data as metadata. The body position can vary, but for colonoscopy, for example, left lateral decubitus and supine positions are assumed. There can be various methods for acquiring body position data. As an example, the body position can be detected by an accelerometer that detects the direction of gravity, or by image recognition of an image taken of the subject. Figure 17 shows a situation in which operator 4 operates the endoscope 10 to examine a subject 2 lying on the examination table 3. At this time, a smartphone 6 may be attached to the subject 2, and the orientation of the smartphone 6 may be detected by a sensor built into the smartphone 6, and the detected data may be acquired as body position data.

[0058] Figure 18 shows the learning flow in the third embodiment. The inference unit 530a acquires image data and body position data of the training endoscopic image with frame number k, and infers a risk value from the image data and body position data.

[0059] Figure 19 shows the inference flow in the third embodiment. In step S44, the processor 110 acquires body position data from a smartphone 6 or the like attached to the subject 2. The inference unit 530a infers a risk value indicating the risk of unintentional movement from the image data and body position data. By using the body position of the subject 2 indicated by the body position data in the inference, the risk of unintentional movement estimated from the body position can be estimated. For example, the risk of unintentional movement due to gravity, as explained in Figure 6, can be estimated.

[0060] The second and third embodiments may be combined. Specifically, in the learning phase, the training data 550 includes image data of the endoscopic image, UPD data, and body position data. The inference unit 530a infers a risk value from the image data of the endoscopic image, UPD data, and body position data. The processor 510 updates the parameters of the learning model 530 based on the error between the inferred value and the calculated risk value. In the inference phase, the processor 110 acquires the endoscopic image, UPD data, and body position data, and the inference unit 130a infers a risk value from them. The processor 110 presents risk information to the operator based on the inferred value.

[0061] In this embodiment, the endoscopic examination support system 100 includes a processor 110 that performs inference using a trained model 130. The trained model 130 is trained to output risk information by taking multiple training endoscopic images taken in the gastrointestinal tract as input. The risk information indicates the risk of changes in the position of the endoscope tip occurring at each position of the endoscope when each training endoscopic image was taken, regardless of endoscopic manipulation. The processor 110 inputs multiple endoscopic images taken in the gastrointestinal tract in real time during the endoscopic examination to the trained model 130, causing the trained model 130 to output risk information.

[0062] According to this embodiment, the processor 110 can infer the risk of unintentional movement from endoscopic images captured in real time, using a trained model 130 that has learned locations in the gastrointestinal tract where the risk of unintentional movement is high. This makes it possible to predict the occurrence of unintentional movement before it occurs. For example, by presenting risk information to the operator, the operator can take measures against unintentional movement in advance.

[0063] Furthermore, as explained in Figures 10 and 11, the trained model 130 is trained to output risk information indicating a higher degree of risk the closer the distance is between each position of the endoscope 10 when each training endoscope image was captured and the position where a change in the endoscope tip position occurs independently of endoscope manipulation.

[0064] In this embodiment, the trained model 130 is trained using training data 550 to learn calculated risk information as the correct label. The training data 550 includes multiple training endoscopic images, position information indicating the position of the endoscope when each training endoscopic image was taken, and flag information indicating whether or not there is a change in the endoscope tip position at each position that is independent of endoscopic operation. The calculated risk information is calculated using the training data 550 based on the position information and flag information.

[0065] According to these embodiments, the trained model 130 is trained to output risk information indicating a high degree of risk when the endoscope 10 approaches a position where unintentional movements are likely to occur. By using such a trained model 130, it becomes possible to predict unintentional movements in advance.

[0066] Furthermore, as explained in S21 of Figure 11, the calculated risk information is calculated based on the distance between the position of the endoscope at the time each learning endoscope image was captured and the position of the endoscope when there is a change in the endoscope tip position that is not due to endoscope manipulation.

[0067] Furthermore, in this embodiment, the calculated risk information indicates a higher risk level the closer the distance.

[0068] According to these embodiments, when the endoscope 10 approaches a position where unintentional movements are likely to occur, it is possible to calculate calculated risk information indicating a high degree of risk. By training with such calculated risk values ​​as the correct label, the trained model 130 is trained to output risk information indicating a high degree of risk when the endoscope 10 approaches a position where unintentional movements are likely to occur.

[0069] As explained in S5 of Figure 9, the flag information is a label generated when a change is detected when the amount of movement between predetermined frames in multiple training endoscopic images is greater than a predetermined amount. The trained model 130 outputs risk information for the occurrence of a situation where the amount of movement between predetermined frames exceeds a predetermined amount due to a change in the position of the endoscope tip that is not dependent on endoscopic manipulation.

[0070] According to this embodiment, the trained model 130 can infer risk information regarding the occurrence of a situation where the amount of movement between predetermined frames exceeds a predetermined amount, as risk information regarding the occurrence of unintentional actions.

[0071] Furthermore, as explained in Figures 2 to 6, the interaction between the endoscope 10 and the digestive tract wall can cause changes in the position of the endoscope tip that are not dependent on endoscopic manipulation. Risk information indicates the degree of risk of such changes occurring.

[0072] In this embodiment, changes in the position of the endoscope tip that are not due to endoscopic manipulation occur when the endoscope tip passes through a place where the diameter of the digestive tract changes, when the endoscope tip passes through a bend in the digestive tract, when the endoscope 10 is supported by a reaction force from the digestive tract wall and the part supported by the reaction force moves from a hard place to a soft place on the digestive tract wall, or when the endoscope, which is not supported by the digestive tract wall, moves due to gravity.

[0073] In endoscopic examinations, regardless of whether or not the endoscope is manipulated or what is being done, unintentional movements like those described above may occur due to the interaction between the endoscope 10 and the digestive tract wall. According to this embodiment, the risk of such unintentional movements can be predicted.

[0074] As explained in Figures 13 to 15, the trained model 130 also receives shape information (e.g., UPD data) indicating the shape of the endoscope 10 when each training endoscope image was captured as input. The trained model 130 is trained to output risk information indicating a higher degree of risk the closer the distance is, and the closer the shape of the endoscope 10 when each training endoscope image was captured is to the shape of the endoscope 10 that causes changes in the endoscope tip position independent of endoscope operation. The processor 110 inputs the shape information (e.g., UPD data) to the trained model 130, causing the trained model 130 to output risk information.

[0075] In this embodiment, the training data 550 further includes shape information indicating the shape of the endoscope when each training endoscope image was captured. The trained model 130 is trained to output risk information by taking the shape information as further input. The processor 110 inputs the shape information further into the trained model 130, causing the trained model 130 to output risk information.

[0076] According to these embodiments, the risk of unintentional movement can be predicted not only based on the position of the endoscope but also on the insertion shape of the endoscope 10. This is expected to improve prediction accuracy compared to prediction based on position alone.

[0077] As explained in Figures 16 to 19, the trained model 130 also receives positional information indicating the position of the subject 2 when each training endoscopic image was acquired. The trained model 130 is trained to output risk information indicating a higher degree of risk when the aforementioned distance is close, and when the position when each training endoscopic image was acquired is the same as the position in which a change in the endoscope tip position occurs regardless of endoscopic manipulation. The processor 110 inputs the positional information further into the trained model 130, causing the trained model 130 to output risk information.

[0078] In this embodiment, the training data 550 further includes positional information indicating the body position of the endoscopic patient when each training endoscopic image was captured. The trained model 130 is trained to output risk information by further inputting the positional information. The processor 110 inputs the positional information further into the trained model 130, causing the trained model 130 to output risk information.

[0079] According to these embodiments, the risk of unintentional movement can be predicted not only based on the position of the endoscope but also on the posture of the subject 2. This is expected to improve prediction accuracy compared to prediction based on position alone.

[0080] The above embodiments may be implemented as methods for operating the endoscopic examination support system 100. The operating method includes the endoscopic examination support system 100 acquiring multiple endoscopic images taken in the gastrointestinal tract in real time during an endoscopic examination. The operating method also includes the endoscopic examination support system 100 inputting multiple endoscopic images into a trained model 130, causing the trained model 130 to output risk information. The trained model 130 is trained to output risk information by taking multiple training endoscopic images taken in the gastrointestinal tract as input. The risk information indicates the risk of changes in the position of the endoscope tip that are not dependent on endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken.

[0081] Furthermore, the above embodiments may be implemented as a program. The program causes a computer to acquire multiple endoscopic images taken in real time within the digestive tract during an endoscopic examination. The program causes a computer to input multiple endoscopic images into a trained model 130, thereby causing the trained model 130 to output risk information. The trained model 130 is trained to output risk information by taking multiple training endoscopic images taken within the digestive tract as input. The risk information indicates the risk of changes in the position of the endoscope tip that are not dependent on endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken.

[0082] 3. Risk detection using rule-based processing The following describes an embodiment in which an endoscopic examination support system performs risk detection using rule-based processing. Although rule-based processing is used for the risk estimation itself, AI may be used in the stage of acquiring the data used for risk estimation. Figures 20 to 23 are explanatory diagrams of the fourth embodiment.

[0083] Figure 20 shows a second configuration example of the endoscopic examination support system 100. The differences from the configuration example of the endoscopic examination support system 100 described in Figure 8 will be explained. Memory 120 stores recording data 160. As described later, recording data 160 is data recorded when the endoscope 10 is inserted from the anus to the ileocecal region. During an examination performed while withdrawing the endoscope 10, the processor 110 estimates the risk of unintentional operation by referring to the recording data 160. Note that in embodiments from the fifth embodiment onward, where recording data 160 is not used, recording during insertion may be omitted.

[0084] Figure 21 shows the insertion flow in the fourth embodiment. This flow is performed when the operator inserts the endoscope 10 from the anus to the ileocecal region. In step S101, the processor 110 acquires frame image data of the motion image data captured in real time by the endoscope as image data of the endoscope image.

[0085] In step S102, the processor 110 infers the diameter of the intestine from the endoscopic image of the current frame. For example, the processor 110 inputs the image data of the current frame into a machine learning model and has the machine learning model infer the diameter of the intestine. The machine learning model is pre-trained to infer and output the diameter of the intestine from the input endoscopic image.

[0086] In step S103, the processor 110 infers the scope movement amount (scope movement distance) between the current frame and the previous frame from the endoscopic images of the current frame and the previous frame. For example, the processor 110 determines the optical flow as the scope movement amount.

[0087] In step S104, the processor 110 accumulates the scope movement amount obtained in step S103 to determine the scope tip position, associates the scope tip position with the diameter of the intestinal tract obtained in step S102, and records it in memory 120 as recorded data 160. Figure 22 shows an example of recorded data 160. "Distance from the anus" indicates the distance from the anus to the scope tip position, and may be the numerical value obtained in the processing of S103 in Figure 21, or it may be a numerical value converted to a unit such as centimeters. "Digestive tract diameter" indicates the diameter of the digestive tract observed at each distance, and may be the numerical value obtained in the processing of S102 in Figure 21, or it may be a numerical value converted to a unit such as centimeters. In the example in Figure 22, the distance "100" corresponds to the distance from the anus to the ileocecal region.

[0088] Figure 23 shows the withdrawal flow in the fourth embodiment. This flow is performed when the operator examines the large intestine while withdrawing the endoscope 10 from the ileocecal region to the anus. Step S122 may be omitted. In step S121, the processor 110 acquires frame image data of the moving image data captured in real time by the endoscope as image data of the endoscope image.

[0089] In step S123, the processor 110 infers the scope movement amount (scope movement distance) between the current frame and the previous frame from the endoscopic images of the current frame and the previous frame. For example, the processor 110 determines the optical flow as the scope movement amount.

[0090] In step S124, the processor 110 calculates the scope tip position by accumulating the inferred scope movement. For example, the processor 110 calculates the current scope tip position by cumulatively subtracting the scope movement from the distance from the anus to the ileocecal region recorded in the recording data 160 (100 in Figure 22). The processor 110 then obtains the diameter of the intestinal tract around the current scope tip position by referring to the recording data 160.

[0091] Alternatively, a further step S122 may be added. In step S122, the processor 110 infers the diameter of the intestinal tract from the endoscopic image of the current frame. For example, the same machine learning model as in step S102 can be used.

[0092] In step S124, the processor 110 accumulates the inferred scope movement to determine the scope tip position, and based on that scope tip position, the recorded data 160, and the diameter of the intestinal tract determined in step S122, it estimates the current scope tip position and the diameter of the intestinal tract around it.

[0093] For example, the processor 110 obtains the diameter of the intestinal tract corresponding to the scope tip position determined by the accumulation of scope movement from the recorded data 160, and verifies the validity of the scope tip position by comparing this diameter with the diameter of the intestinal tract determined in step S122. If the two diameters of the intestinal tract are approximately the same, the processor 110 sets the scope tip position determined by the accumulation of scope movement as the current scope tip position. If the two diameters of the intestinal tract are not the same, the processor 110 obtains a scope tip position from the recorded data 160 that matches the diameter of the intestinal tract determined in step S122, for example, and sets that as the current scope tip position. The processor 110 obtains the diameter of the intestinal tract around the current scope tip position from the recorded data 160.

[0094] In step S125, the processor 110 detects the risk of unintentional movement based on the diameter of the intestinal tract around the current position of the scope tip. The processor 110 determines that the risk is high when the tip of the endoscope is close to a point where the diameter of the intestinal tract changes abruptly, as described in Figures 2 and 5. The detected risk information may be binary information indicating the presence or absence of risk, or it may be continuous or stepwise information indicating the degree of risk.

[0095] In step S126, the processor 110 notifies the operator of the detected risk information. As described in the first embodiment, there may be various methods for presenting the risk information.

[0096] Figures 24 and 25 are explanatory diagrams of the fifth embodiment. Figure 24 shows the processing flow in the fifth embodiment. This flow is intended to be performed during an examination while the endoscope is being removed, but it may also be performed during insertion. In step S141, the processor 110 acquires frame image data of the moving image data captured in real time by the endoscope as image data of the endoscope image.

[0097] In step S142, the processor 110 estimates the location of the large intestine from the endoscopic image. For example, the processor 110 inputs the image data of the current frame into a machine learning model and causes the machine learning model to infer the location of the large intestine. The machine learning model is pre-trained to infer and output the location of the large intestine from the input endoscopic image.

[0098] In step S143, the processor 110 obtains the scope shape from the UPD. For example, the processor 110 obtains UPD data, which is three-dimensional coordinate data of the scope shape as described in Figure 13.

[0099] In step S144, the processor 110 detects the risk of unintentional movement based on the location of the colon and the scope shape. Figure 25 shows an example of the scope shape obtained by UPD. Since the insertion part of the endoscope follows the shape of the intestinal tract, the processor 110 can analyze from the scope shape obtained by UPD which part of the colon each position of the insertion part is located at. From the analysis results, the processor 110 determines that the risk of unintentional movement is high when the tip of the endoscope is close to a location where unintentional movement, as described in Figures 2 to 6, is likely to occur.

[0100] Furthermore, because the stiffness of the intestinal wall varies depending on the location, the processor 110 can analyze which part of the insertion section is receiving a strong reaction force from the intestinal wall. For example, as shown in Figure 15, if the scope shape includes a sharp bend, the processor 110 determines the area to which the sharp bend makes contact and thereby determines the stiffness of the intestinal wall in contact with that area. The processor 110 determines that the risk is high when the sharp bend of the endoscope is close to the boundary between the hard and soft parts of the intestinal wall, as explained in Figure 3. However, even without a sharp bend, the above risk determination is possible by knowing which area the insertion section near the tip of the endoscope makes contact with.

[0101] In step S145, the processor 110 notifies the operator of the detected risk information. As described in the first embodiment, there may be various methods for presenting the risk information.

[0102] The processor 110 may acquire the scope tip position instead of the scope shape in step S143. The method for acquiring the scope tip position is as described in S123, S124, etc., in Figure 23. Once the scope tip position is known, it is possible to estimate which part of the large intestine each insertion point is located in.

[0103] Figure 26 shows the processing flow of the sixth embodiment. In step S161, the processor 110 obtains the scope shape from the UPD. In step S162, the processor 110 obtains the subject's body position, i.e., the direction of gravity. The method for obtaining the body position is as described in the third embodiment. In step S163, the processor 110 obtains the stiffness of the scope. The stiffness of the scope indicates how much each position of the insertion part bends in response to external force, and is stored in advance in the memory 120, etc.

[0104] In step S164, a mechanical simulation is performed using the scope shape, the direction of gravity, and the stiffness of the scope to detect the risk of unintentional movement due to gravity, as explained in Figure 6.

[0105] In step S165, the processor 110 notifies the operator of the detected risk information. As described in the first embodiment, there may be various methods for presenting the risk information.

[0106] A seventh embodiment will now be described. The processor 110 detects the risk of unintentional movement due to tension release based on the scope shape and the diameter of the intestinal tract, or the scope shape and the location of the large intestine. The degree of curvature of the scope can be determined from the scope shape, and the tension on the scope can be determined from the degree of curvature. When the tension on the scope is released at a location where the diameter of the intestinal tract changes, or where the type of location changes, unintentional movement occurs. The processor 110 determines that the risk is high when the tip of the endoscope is close to such a location. The bent scope tries to return to its original shape due to its elasticity, but the force received from the intestinal wall maintains the bent state. This is the state under tension. When the diameter or location of the intestinal tract changes, the force received from the intestinal wall is released, the tension is released, and the scope tries to return from the bent state to the straightened state, which is unintentional movement. In the seventh embodiment, the risk of such unintentional movement can be detected.

[0107] The eighth embodiment will now be described. The flow during insertion is the same as in Figure 21. Figure 27 shows the flow during removal in the eighth embodiment. Image acquisition in S181, scope movement amount estimation in S183, and intestinal tract thickness estimation in S182 are the same as in S121 to S123 in Figure 23.

[0108] In S184, the processor 110 obtains the length of the active bending section of the endoscope. The length of the active bending section is a constant determined for each endoscope model and is stored in advance in memory 120 or the like.

[0109] In step S185, the processor 110 accumulates the scope movement amount inferred in step S183 to determine the scope tip position, and estimates the position of the boundary between the active and passive bending sections of the scope based on the scope tip position and the length of the active bending section obtained in step S184. The processor 110 then refers to the recorded data 160 to obtain the diameter of the intestinal tract near the boundary between the active and passive bending sections.

[0110] In step S186, the processor 110 detects the risk of unintentional movement based on the thickness of the intestinal tract near the boundary between the active and passive bending sections. The processor 110 determines that the risk of unintentional movement is high when the boundary between the active and passive bending sections is close to a location where unintentional movement caused by the active bending section, as described in Figures 4 and 5, occurs.

[0111] In step S187, the processor 110 notifies the operator of the detected risk information. As described in the first embodiment, there may be various methods for presenting the risk information.

[0112] Furthermore, embodiments 4 to 8 may be combined as appropriate. For example, multiple embodiments may be implemented in parallel, and the risk information with the highest risk among the risk information detected in each embodiment may be selected and presented to the operator based on that.

[0113] While embodiments and variations of the present disclosure have been described above, the present disclosure is not limited to each embodiment or its variations as they are. In the implementation stage, the components can be modified and made concrete without departing from the gist of the disclosure. Furthermore, various disclosures can be formed by appropriately combining multiple components disclosed in each of the above embodiments and variations. For example, some components may be deleted from all the components described in each embodiment or variation. Furthermore, components described in different embodiments and variations may be appropriately combined. In this way, various modifications and applications are possible without departing from the gist of the disclosure. In addition, any term that appears at least once in the specification or drawings together with a different term that is broader or synonymous may be replaced with that different term anywhere in the specification or drawings. [Explanation of Symbols]

[0114] 1...Colon, 2...Patient, 3...Examination table, 4...Operator, 6...Smartphone, 10...Endoscope, 11...Active bending section, 12...Passive bending section, 100...Endoscopy support system, 110...Processor, 120...Memory, 130...Model, 130a...Inference unit, 140...Program, 160...Recorded data, 190...Display, 250...Video processor, 290...Display, 300...Endoscopy system, 500...Learning system, 510...Processor, 520...Memory, 530...Learning model, 530a...Inference unit, 540...Program, 550...Training data, 570...Image data

Claims

1. Includes a processor that performs inference using a pre-trained model, The aforementioned trained model is trained to take multiple training endoscopic images taken in the gastrointestinal tract as input and output risk information for the occurrence of changes in the endoscope tip position that are independent of endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken. The endoscopic examination support system is characterized in that the processor inputs multiple endoscopic images captured in the gastrointestinal tract in real time during an endoscopic examination into the trained model, thereby causing the trained model to output the risk information.

2. In the endoscopic examination support system described in claim 1, The endoscopic examination support system is characterized in that the trained model is trained to output risk information indicating a higher degree of risk the closer the distance is between each position of the endoscope when each training endoscopic image is captured and the position where the change in the endoscope tip position, independent of the endoscope operation, occurs.

3. In the endoscopic examination support system described in claim 1, The endoscopic examination support system is characterized in that the trained model is trained using training data which includes a plurality of training endoscopic images, position information indicating the respective positions of the endoscope when each training endoscopic image was captured, and flag information indicating whether or not there is a change in the position of the endoscope tip at each position independent of the endoscope operation, with calculated risk information calculated based on the position information and the flag information used as the correct label.

4. In the endoscopic examination support system described in claim 3, The endoscopic examination support system is characterized in that the calculated risk information is calculated according to the distance between the respective positions of the endoscope when each learning endoscope image is captured and the position of the endoscope when there is a change in the endoscope tip position that is not dependent on the endoscope operation.

5. In the endoscopic examination support system described in claim 4, The endoscopic examination support system is characterized in that the calculated risk information indicates a higher risk level the closer the distance.

6. In the endoscopic examination support system described in claim 3, The aforementioned flag information is a label generated when a change is determined to have occurred when the amount of movement between predetermined frames in the plurality of learning endoscope images is greater than a predetermined amount. The endoscopic examination support system is characterized in that the trained model outputs risk information regarding the occurrence of a situation in which the amount of movement between predetermined frames becomes greater than a predetermined amount due to a change in the position of the endoscope tip that is not dependent on the endoscopic operation.

7. In the endoscopic examination support system described in claim 1, The endoscopic examination support system is characterized in that the risk information is information indicating the degree of risk that the change in the tip position of the endoscope will occur due to the interaction between the endoscope and the digestive tract wall, and is independent of the endoscopic operation.

8. In the endoscopic examination support system described in claim 7, The change in the tip position of the endoscope, which is not dependent on the aforementioned endoscope operation, When the tip of the endoscope passes through a location where the diameter of the digestive tract changes, When the tip of the endoscope passes the bend in the digestive tract, When the endoscope is supported by a reaction force from the digestive tract wall, and the part supported by the reaction force moves from a hard area to a soft area of ​​the digestive tract wall, or When the endoscope, which is not supported by the digestive tract wall, moves due to gravity, An endoscopic examination support system characterized by occurring in [location].

9. In claim 2, The trained model is further trained to output risk information indicating a higher degree of risk the closer the distance is, and the closer the shape of the endoscope at the time each training endoscope image was captured is to the shape of the endoscope at the time each training endoscope image was captured. The endoscopic examination support system is characterized in that the processor further inputs the shape information into the trained model, thereby causing the trained model to output the risk information.

10. In claim 3, The training data further includes shape information indicating the shape of the endoscope when each of the learning endoscope images was captured, The trained model is further trained to take the shape information as input and output the risk information. The endoscopic examination support system is characterized in that the processor further inputs the shape information into the trained model, thereby causing the trained model to output the risk information.

11. In claim 2, The trained model is further trained to output risk information indicating a higher degree of risk when the distance is close and the position of the subject when each training endoscopic image was taken is the same as the position of the subject when the endoscope tip position changes independently of the endoscope operation. The endoscopic examination support system is characterized in that the processor further inputs the positional information to the trained model, causing the trained model to output the risk information.

12. In claim 3, The aforementioned training data further includes positional information indicating the body position of the endoscopic patient when each of the training endoscopic images was captured. The aforementioned trained model is further trained to take the body position information as input and output the risk information. The endoscopic examination support system is characterized in that the processor further inputs the positional information to the trained model, causing the trained model to output the risk information.

13. The endoscopic examination support system acquires multiple endoscopic images taken in the gastrointestinal tract in real time during an endoscopic examination, The endoscopic examination support system takes multiple training endoscopic images taken in the digestive tract as input to a trained model that has been trained to output risk information regarding the occurrence of changes in the position of the endoscope tip that are independent of endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken. By inputting these multiple endoscopic images into the trained model, the system causes the trained model to output the risk information. A method for operating an endoscopic examination support system, characterized by including the following:

14. In endoscopy, it is possible to acquire multiple endoscopic images taken in the gastrointestinal tract in real time, By inputting multiple training endoscopic images taken within the digestive tract into a trained model that has been trained to output risk information regarding the occurrence of changes in the endoscope tip position independent of endoscopic manipulation at each position of the endoscope when each training endoscopic image was taken, the trained model is made to output the risk information. A program that causes a computer to execute something.

Images