Endoscope operation digitization system, endoscope operation data collection system, endoscope operation guide system, data processing device, endoscope data collection system, data processing method, insertion guide method, and insertion guide program

Abstract

An endoscope operation digitization system disclosed herein comprises: a first time-series image analysis unit for determining the change over time of an in-image feature part of time-series images obtained by an imaging unit by an operation of an endoscope operator performing an operation of inserting an endoscope having the imaging unit at a distal end into the body of a subject or extracting the endoscope therefrom; a second time-series image analysis unit for determining the change over time of an in-image feature part of time-series images obtained by an external camera for imaging the endoscope operator; a synchronization unit for synchronizing the time-series images obtained by the imaging unit and the time-series images obtained by the external camera; and a time-series operation information generation unit for generating operation data by digitizing the characteristics of the operation of the endoscope operator by dividing the type of the operations of the endoscope operator into the items of endoscope insertion and extraction, torsion of an endoscope insertion unit, and the angle of the endoscope tip according to changes in each of the synchronized images over time, and analyzing the presence or absence of the operation and the change in the amount of the operation for each of the items.

Description

Endoscope operation quantification system, endoscope operation data acquisition system, endoscope operation guide system, data processing device, endoscope data acquisition system, data processing method, insertion guide method, and insertion guide program

[0001] The present invention relates to an endoscopic operation quantification system for assisting insertion into a lumen, an endoscopic operation data acquisition system, an endoscopic operation guide system, a data processing device, an endoscopic data acquisition system, a data processing method, an insertion guide method, and an insertion guide program.

[0002] An endoscope is a device that allows observation of areas that are difficult to see from outside the insertion site by inserting an imaging unit, which consists of an imaging device, into the body and illuminating it with light from an attached illumination unit to obtain imaging results (endoscopic images) from the imaging unit. An endoscope has an insertion section that is inserted into the body, and an imaging device is provided, for example, at the tip of the insertion section. In an examination using an endoscope, the physician sequentially displays the images acquired by the imaging device at the tip of the insertion section inserted into the body, and adjusts the position of the tip of the insertion section while checking the displayed images to diagnose the health condition or disease state inside the body. In addition, the image information consisting of a time series of image frames acquired during the process from the start of insertion of the endoscope into the body to the removal of the endoscope from the body may be digitized and recorded.

[0003] However, inserting an endoscope into the body is a complex procedure involving various manipulations, and successful insertion into the target site is difficult without skilled personnel. When inexperienced individuals perform the insertion, appropriate insertion assistance is desirable to facilitate the procedure. For example, it is conceivable to collect information on the procedures performed by skilled individuals and then implement automated insertion assistance based on that information.

[0004] As a technology related to such information collection, in U.S. Patent No. 7,894,648 (hereinafter referred to as Patent Document 1), a technology for evaluating the quality of an endoscopic procedure performed by inserting and removing an endoscopic camera into and out of a lumen is disclosed. In the proposal of Patent Document 1, digital video data including images of an endoscopic examination procedure is acquired, the digital video data is analyzed to obtain information regarding the movement of the camera passing through the lumen, and quality measurement criteria regarding the endoscopic examination procedure are calculated from the information on the movement of the camera. By using the proposal of Patent Document 1, it is conceivable to collect information necessary for assisting endoscopic insertion and use it for insertion assistance.

[0005] U.S. Patent No. 7,894,648

[0006] However, even if the proposal of Patent Document 1 is used, there is a problem that sufficient information regarding the endoscopic insertion operation, which is necessary for appropriate insertion assistance, cannot be collected. An object of the present invention is to provide an endoscopic operation quantification system, an endoscopic operation data collection system, an endoscopic operation guidance system, a data processing device, an endoscopic data collection system, a data processing method, an insertion guidance method, and an insertion guidance program capable of performing information collection for enabling creation of guidance information for effectively assisting insertion of an insertion portion by using not only endoscopic images obtained in a situation where insertion of the insertion portion is difficult but also images of an external camera.

[0007] An endoscopic operation quantification system according to one aspect of the present invention comprises: a first time-series image analysis unit for determining the time change of feature portions within time-series images obtained by the imaging unit in response to the operation of an endoscope operator inserting or withdrawing an endoscope having an imaging unit at its tip into the body of a subject; a second time-series image analysis unit for determining the time change of feature portions within time-series images obtained by an external camera that photographs the endoscope operator; a synchronization unit for synchronizing the time-series images obtained by the imaging unit and the time-series images obtained by the external camera; and a time-series operation information generation unit that, according to the time-series changes of each of the synchronized images, analyzes the presence or absence of operation and the change in the amount of operation for each item, dividing the type of operation of the endoscope operator into endoscope insertion / withdrawal, twisting of the endoscope insertion unit, and angle of the endoscope tip, and generates operation data that quantifies the characteristics of the operation of the endoscope operator.

[0008] An endoscope operation data acquisition system according to one aspect of the present invention comprises an operation data recording unit that records operation data obtained by an endoscope operation quantification system, which divides the operation type of the endoscope operator into items such as endoscope insertion / removal, twisting of the endoscope insertion section, and angle of the endoscope tip, and records information resulting from the time-series determination of the presence or absence of operation and the change in the amount of operation for each item, along with images obtained by the endoscope imaging unit, in accordance with each timing in the time series, and provides this operation data to an operation guide generation unit.

[0009] An endoscope operation guide system according to one aspect of the present invention comprises: an acquisition unit that synchronizes and acquires a first time-series image from an imaging device provided in the endoscope insertion section and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion section; a time-series operation information generation unit that generates operation data relating to the operator's operation based on the first and second time-series images; and an endoscope operation determination unit that uses operation data previously generated by the time-series operation information generation unit as model operation data and operation data currently being generated by the time-series operation information generation unit as current operation data, and generates guide information for guiding the insertion operation based on a comparison between the current operation data and the model operation data.

[0010] A data processing device according to one aspect of the present invention comprises: an acquisition unit that synchronizes and acquires a first time-series image from an imaging device provided in the endoscope insertion section and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion section; a first time-series image analysis unit that acquires first operation information corresponding to the operator's operation based on the image changes of the first time-series image by analyzing the first time-series image; a second time-series image analysis unit that acquires second operation information corresponding to the operator's operation based on the image changes of the second time-series image by analyzing the second time-series image; and a time-series operation information generation unit that generates and records operation data related to the operator's operation based on the first and second information.

[0011] An endoscopic data acquisition system according to one aspect of the present invention comprises: an endoscope with an imaging device provided in the insertion section; an external camera for imaging an operator operating the endoscope; an acquisition unit for synchronously acquiring a first time-series image from the imaging device and a second time-series image from the external camera; a first time-series image analysis unit for acquiring first operation information corresponding to the operator's operations based on image changes in the first time-series image by analyzing the first time-series image; a second time-series image analysis unit for acquiring second operation information corresponding to the operator's operations based on image changes in the second time-series image by analyzing the second time-series image; a time-series operation information generation unit for generating operation data related to the operator's operations based on the first and second information; and a recording unit for recording the operation data.

[0012] A data processing method according to one aspect of the present invention is a data processing method for a data processing device having an acquisition unit, first and second time-series image processing units, and a time-series operation information generation unit, wherein the acquisition unit acquires a first time-series image from an imaging device provided in the endoscope insertion unit and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion unit in synchronization; the first time-series image analysis unit acquires first operation information corresponding to the operator's operation based on the image changes of the first time-series image by analyzing the first time-series image; the second time-series image analysis unit acquires second operation information corresponding to the operator's operation based on the image changes of the second time-series image by analyzing the second time-series image; and the time-series operation information generation unit generates and records operation data related to the operator's operation based on the first and second information.

[0013] An insertion guiding method according to one aspect of the present invention is an insertion guiding method for an endoscope operation guiding system comprising an acquisition unit, a time-series operation information generation unit, and an endoscope operation determination unit, wherein the acquisition unit acquires a first time-series image from an imaging device provided in the endoscope insertion unit and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion unit in synchronization, the time-series operation information generation unit generates operation data relating to the operator's operation based on the first and second time-series images, and the endoscope operation determination unit uses the operation data previously generated by the time-series operation information generation unit as model operation data and the operation data currently being generated by the time-series operation information generation unit as current operation data, and generates guide information for guiding the insertion operation based on a comparison between the current operation data and the model operation data.

[0014] An insertion guide program according to one aspect of the present invention causes a computer to synchronize and acquire a first time-series image from an imaging device provided in the endoscope insertion section and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion section. Based on the first and second time-series images, the computer generates operation data related to the operator's operation. The computer uses previously generated operation data as model operation data and currently generated operation data as current operation data. Based on a comparison between the current operation data and the model operation data, the computer executes a procedure to generate guide information for guiding the insertion operation.

[0015] According to the present invention, by utilizing not only endoscopic images obtained in situations where insertion of the insertion part is difficult, but also images from an external camera, it is possible to collect information that enables the creation of guide information that effectively supports the insertion of the insertion part.

[0016] This is a block diagram of an endoscope data acquisition system including a data processing device according to the first embodiment of the present invention. This is a diagram showing the information acquired by the information acquisition unit 33 and the information obtained by the time-series operation information generation unit 34. This is an explanatory diagram for explaining the state of the operating room when acquiring the information in Figure 2. This is an explanatory diagram for explaining the operation determination for pushing and pulling the insertion unit 12. This is an explanatory diagram for explaining the operation determination for pushing and pulling the insertion unit 12. This is an explanatory diagram for explaining the operation determination for pushing and pulling the insertion unit 12. This is a flowchart of the inference model creation flow when push and pull determination is performed by inference. This is an explanatory diagram for explaining the operation determination for rotation (twisting) operation of the insertion unit 12. This is a flowchart for explaining the operation determination in Figure 8. This is an explanatory diagram for explaining the bending amount determination for bending operation of the insertion unit 12. This is a flowchart for explaining the bending amount determination in Figure 10. This is a block diagram of the second embodiment. This is an explanatory diagram showing guide information displayed on the display screen 69a of the display unit 69. This is for explaining guide information. This is a flowchart for explaining the operation of the second embodiment. This is an explanatory diagram for explaining the operation of the second embodiment. This is an explanatory diagram for explaining the operation of the second embodiment. This graph shows the change in insertion amount, with the vertical axis representing the insertion amount and the horizontal axis representing the elapsed time.

[0017] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0018] (First Embodiment) Figure 1 is a block diagram showing an endoscopic data acquisition system including a data processing device according to the first embodiment of the present invention. This embodiment acquires information related to endoscopic operation by synchronizing the capture of endoscopic images and images from an external camera and analyzing each image.

[0019] The lumens inside the body that are examined with an endoscope may be narrowed by sphincter muscles, bend three-dimensionally, be clogged with metabolites, or have raised lumen walls due to lesions. For this reason, it is not always possible to clearly visualize the lumen in the length direction (direction of hole connection) ahead of insertion from the endoscopic image acquired by the imaging unit, which is composed of an imaging device installed at the tip of the endoscope's insertion tube. There may be such difficult-to-insert sections with poor visibility in the middle of the lumen, or when entering a lumen from outside the body, entering another lumen from there, or when branching, as described above.

[0020] Therefore, the endoscope operator may need to proceed with insertion while searching for a path for the endoscope insertion site, even when the cavity ahead in the insertion direction is not necessarily visible (a dead end that appears to be a difficult insertion area). Moreover, the condition of the lumen varies from person to person, and even in the same person, the condition of the lumen differs depending on changes in the patient's position at the examination site, changes over time, and the circumstances at the time, making insertion of the endoscope insertion site not always easy.

[0021] Considering differences in examination equipment and usage methods, in most cases, sufficient experience is required from medical professionals to smoothly insert the endoscope into the lumen. Therefore, when a novice performs the examination, support from an experienced physician was necessary. However, with recent policies such as "training physicians with comprehensive clinical abilities," physicians are increasingly performing endoscopic examinations in addition to various other medical duties. Consequently, even trained physicians cannot always dedicate themselves to endoscopic operation while maintaining their operational skills.

[0022] To address these problems, one could consider using AI (artificial intelligence) to assist with the insertion of the insertion device, as a way to more efficiently transfer technical knowledge and provide an inspection method that even beginners can easily perform. However, collecting useful training data is difficult, and building inference models for such AI is not easy.

[0023] Therefore, in order to reliably obtain information on the characteristics of endoscopic operation performed by skilled physicians, this embodiment collects information indicating the operator's movements, such as the operator's posture, how they hold the endoscope in both hands, and hand movements, not only from endoscopic images but also from images taken with an external camera.

[0024] The endoscopic data acquisition system shown in Figure 1 includes an endoscope system 1, an external camera 20, a data processing device 30, an output unit, and a recording unit 40. Although Figure 1 shows one external camera 20, multiple external cameras 20 may be used. The endoscope system 1 has a control unit 11. The control unit 11 may be composed of a processor using a CPU (Central Processing Unit) or FPGA (Field Programmable Gate Array), and may operate according to a program stored in a memory (not shown) to control each part, or it may implement some or all of its functions with hardware electronic circuits. The control unit 11 comprehensively controls each part of the endoscope system 1.

[0025] The endoscope system 1 includes an endoscope 10. The endoscope 10 has a long, slender insertion section 12 that is inserted into the body. An imaging section 12a is provided at the tip of the insertion section 12. The imaging section 12a, as an imaging device, includes an image sensor such as a CCD or CMOS sensor, and obtains an imaging signal by photoelectric conversion of an optical image from the subject. A curved section 12b is also provided at the tip of the insertion section 12.

[0026] An operating section 13 is provided at the base end of the insertion section 12. The operating section 13 is equipped with various operating devices for driving the bending section 12b to bend, such as a dial 13a, and the bending section 12b can be bent by operating the dial 13a. The operating section 13 is also equipped with a switch 13b for operating various medical devices (not shown), such as an air supply / water supply device.

[0027] The imaging unit 12a receives the optical image of the subject and obtains an imaging signal from within the body through photoelectric conversion. The imaging signal obtained by the imaging unit 12a is supplied to the internal video information generation unit 16. The internal video information generation unit 16 generates an endoscopic image by performing predetermined signal processing on the imaging signal. The internal video information generation unit 16 outputs the generated endoscopic image (hereinafter also referred to as internal image, internal video, or first time-series image) as internal video information to the data processing device 30.

[0028] The external camera 20 is provided with a control unit 21. The control unit 21 may be composed of a processor using a CPU or FPGA, or it may operate according to a program stored in a memory (not shown) to control each part, or it may implement some or all of its functions with hardware electronic circuits. The control unit 21 comprehensively controls each part of the external camera 20.

[0029] The external camera 20 has an imaging unit 22. The imaging unit 22 includes, for example, an image sensor such as a CCD or CMOS sensor, and obtains an imaging signal by photoelectric conversion of an optical image from a subject. The imaging signal obtained by the imaging unit 22 is supplied to the external video information generation unit 25. The external video information generation unit 25 generates an image by performing predetermined signal processing on the imaging signal. The external video information generation unit 25 outputs the generated image (hereinafter also referred to as the external image, external video, or second time-series image) as external video information to the data processing device 30.

[0030] The endoscope system 1 has an input / output unit 15, and the external camera 20 has an input / output unit 24. The input / output unit 15 and the input / output unit 24 can exchange data with each other. The endoscope system 1 also has a synchronization unit 14, and the external camera 20 has a synchronization unit 23. The synchronization unit 14 and the synchronization unit 23 work in coordination with each other to synchronize imaging by the imaging unit 12a and imaging by the imaging unit 22.

[0031] The synchronization unit 14 and the synchronization unit 23 can synchronize the imaging of the imaging unit 12a with the imaging unit 22 by employing various synchronization methods. For example, one of the synchronization units 14 and 23 may act as a master and generate a trigger, and the other receives the trigger via the input / output units 15 and 24, and controls the imaging units 12a and 22 to start imaging according to the received trigger, thereby synchronizing the internal video and the external video. Alternatively, the synchronization units 14 and 23 may assign timestamps to the internal video and the external video during imaging and synchronize the internal video and the external video by referring to the timestamps. Furthermore, synchronization may be performed manually using a clapperboard or the like. The term "synchronization" here refers to "matching and synchronizing the timing of operations," assuming a situation where the output image frames of the two imaging units are obtained and aligned almost simultaneously. However, it is important that the relationship between the acquisition timing of the output images of the two imaging units is understood, as it is sufficient to be able to analyze from the image of the imaging unit 22 what operations are causing the image to change moment by moment as the image captured and displayed in each frame of the imaging unit 12a changes. Furthermore, the frame rate of the image frames of the imaging unit 22 may be high, or even if the frame rate is low, it is acceptable as long as analysis is possible through frame interpolation, etc.

[0032] The synchronization information generation unit 17 generates the synchronization information necessary for synchronizing the internal video and the external video and outputs it to the data processing device 30. Although Figure 1 shows an example where synchronization information is generated separately from the internal video and the external video, the synchronization information may also be added to the internal video and the external video, respectively, to obtain the internal image information and the external image information.

[0033] The data processing device 30 includes a first time-series image analysis unit 31, a second time-series image analysis unit 32, an information acquisition unit 33, and a time-series operation information generation unit 34. Each part of the data processing device 30 may be composed of a processor using a CPU, GPU (Graphics Processing Unit), FPGA, NPU (Neural Processing Unit), etc. The data processing device 30 may operate according to a program stored in a memory (not shown) to control each part, or some or all of its functions may be realized by hardware electronic circuits. The first time-series image analysis unit 31 and the second time-series image analysis unit 32 function as acquisition units that acquire the first time-series image and the second time-series image in a synchronized manner.

[0034] The first time-series image analysis unit 31 is given internal image information and analyzes the internal image as a first time-series image. The first time-series image analysis unit 31 converts, for example, image changes in the internal image into information corresponding to the operator's operation (first operation information), such as angle information. In this way, the first time-series image analysis unit 31 obtains first operation information corresponding to the operator's operation based on the image changes in the first time-series image through the analysis process of the first time-series image.

[0035] Furthermore, the first time-series image analysis unit 31 also performs observation and extraction site determination to determine which part of the image is being captured by the imaging unit 12a. For example, the first time-series image analysis unit 31 can determine the observation site by referring to a database (not shown) that contains characteristic colors, shapes, and patterns of blood vessels visible on the surface of the part. Alternatively, the first time-series image analysis unit 31 may use a known AI (artificial intelligence) for site determination to determine the site.

[0036] The second time-series image analysis unit 32 is given external image information and analyzes the external image as a second time-series image. The second time-series image analysis unit 32 converts, for example, image changes in the external image into information corresponding to the operator's operations (second operation information), such as distance and angle. In this way, the second time-series image analysis unit 32 obtains second operation information corresponding to the operator's operations based on the image changes in the second time-series image through the analysis process of the second time-series image.

[0037] The information acquisition unit 33 selects information from the first and second time-series images necessary for collecting information related to operations on the endoscope. For example, the analysis results of the second time-series image may be necessary for specific operations within the endoscope. In this case, the information acquisition unit 33 selects the analysis results of the second time-series image to obtain information related to the operation of that operation. Also, if the insertion unit 12 does not move in the appropriate direction, or if the operator is located outside the shooting range of the external camera 20, there may be no image portion in the internal or external image necessary for collecting information related to endoscope operations. In this case, the information acquisition unit 33 selects and acquires an image frame that contains the image portion necessary for collecting information related to endoscope operations. The time-series operation information generation unit 34 generates and records time-series operation information, which is operation data indicating the content of the operator's operations, based on the analysis results of the information acquired by the information acquisition unit 33.

[0038] Figure 2 is a diagram showing the information acquired by the information acquisition unit 33 and the information obtained by the time-series manipulation information generation unit 34. Figure 3 is an explanatory diagram illustrating the state of the operating room when acquiring the information shown in Figure 2.

[0039] As shown in Figure 3, a patient 51 is lying on a bed 50 in the operating room. Figure 3 shows the surgeon 53 standing beside the bed 50 and attempting to insert the insertion device 12 into the patient's 51 anus with his hand 54. Three cameras 57a to 57c (hereinafter referred to as camera 57 when there is no need to distinguish between them) corresponding to the external camera 20 in Figure 1 are positioned at three locations near the bed 50. Each camera 57 can capture images in sync with each other by the synchronization unit 23 in Figure 1. Each camera 57 captures images of the patient 51 and surgeon 53 from different positions, allowing for images of the surgeon's 53 overall posture, arms, right wrist and fingers, right elbow, left fingers, and the patient's 51 posture from various angles. Although Figure 3 shows an example using three cameras 57 as the external camera 20, the number of cameras is not limited to this; it may be two or fewer, or four or more.

[0040] Furthermore, display devices 55 and 56 are positioned so that the patient 51 can see them. Although Figure 2 shows an example where two display devices 55 and 56 are used, only one of the display devices may be used. Also, the layout of the operating room is just one example, and the positions and number of cameras 57 and display devices 55 and 56 can be set as appropriate.

[0041] As described above, the endoscope 10 has an elongated insertion section 12 that is inserted into a body cavity, and an operating section 13 provided at the rear end of the insertion section 12. The insertion section 12 also has a flexible section 12b provided adjacent to the tip of the insertion section 12, and the flexible section 12b can be bent by a dial 13a provided on the operating section 13. The subject is imaged by the imaging section 12a provided at the tip of the control unit 21, and the imaged image (internal video) is obtained by the internal video information generation unit 16. This internal video is supplied to the display devices 55 and 56 in Figure 3 and displayed on the display screen.

[0042] The operator, such as a physician, operates the endoscope 10 and inserts the insertion section 12 of the endoscope 10 into the patient 51. While viewing the endoscopic image (internal image) on the display devices 55 and 56, the operator operates the dial 13a etc. provided on the control unit 13 to bend the curved section 12b or move the insertion section 12 forward and backward to insert the tip of the insertion section 12 to the area to be observed.

[0043] Specifically, during the process of inserting the insertion device 12 into the body, the internal conduits and other structures within the body into which the insertion device 12 is inserted are imaged, and internal images are obtained. In addition, during the process of inserting the insertion device 12 into the body, various parts of the surgeon's body are imaged while the insertion device 12 is being inserted, and external images are obtained.

[0044] The surgeon's insertion operations are generally performed on the insertion section 12 itself, that is, the "tube," and the dial 13a provided on the operating section 13. As shown in Figure 2, operations on the tube, which is the object of manipulation, include pushing and pulling the tube into the body and twisting the tube. Possible means of applying such pushing, pulling, or twisting to the tube include insertion using the surgeon's entire body, that is, through the surgeon's posture, or through the movement of the surgeon's hand, for example, the right hand.

[0045] Regarding operations such as pushing, pulling, and twisting on such a tube, it can be determined to a certain extent by the endoscopic image (internal image), and can also be determined by the external image captured by the camera 57. In FIG. 2, the triangle mark indicates that the operation can be determined to a certain extent, and the round mark indicates that the operation can be determined more reliably. For example, by the external image captured by the camera 57, distance detection and determination of fingers and elbows are possible. Also, determination of fingers and elbows is possible by the internal image captured by the imaging unit 12a.

[0046] Also, as shown in FIG. 2, operations on the dial 13a which is the operation target include operations in the left - right direction and up - down direction of the curved portion 12b using the dial 13a. Operations on the dial 13a can be considered to be caused by the movement of the operator's hand, for example, the movement of the left fingers. Regarding such operations on the dial 13a, it can be determined to a certain extent by either the endoscopic image (internal image) or the external image captured by the camera 57. Also, regarding the up - down operation of the dial 13a, determination using a gravity sensor is also possible.

[0047] In the following description, an example of determining the operation of the dial 13a, that is, the operation of the curved portion 12b, from the internal image is shown. However, when it is possible to electrically determine the operation of the dial 13a in the control unit 11, the control unit 11 can determine the operation amount for the curved portion 12b.

[0048] Also, it is possible to determine the standing position and posture of the operator who is the operation target by the external image captured by the external camera 20. Also, by imaging the expression and posture of the patient with the external camera 20, it is possible to determine the expression and posture of the patient as well.

[0049] Thus, for the same operation mode, it may be possible to determine using either the internal image or the external image. In the following description, an example of determining the operation using only one of the internal image or the external image will be described, but it is also possible to perform operation determination using both the external image and the internal image.

[0050] (Push - pull operation determination) FIGS. 4 to 6 are explanatory diagrams for explaining the operation determination regarding the push - pull of the insertion portion 12.

[0051] Figure 4 shows images (external images) PE1 and PE2 captured by camera 57a, which filmed the operator 53 inserting the insertion part 12 into the patient's body through the patient's anus. In the example in Figure 4, the operator 53 grasps the insertion part 12 with his right hand 54R and holds the operating part 13 with his left hand 54L. External images PE1 and PE2 are images taken one after the other in time, and the change from external image PE1 to external image PE2 allows for the determination of the operator's pushing and pulling operations.

[0052] The dashed circles in Figure 4 indicate the elbow and fingers of the operator 53. Image changes in the elbow and fingers of the external images PE1 and PE2 show that the operator 53 is moving his elbow from the outside to the inside of his body, bringing his right hand 54R, which is grasping the insertion part 12, closer to the patient's anus, thereby attempting to insert the insertion part 12 into the operator's body through the anus. Due to this insertion operation, the distance from the edge of the screen to the tip of the operator 53's elbow is greater in external image PE2 than in external image PE1 (distance D1). In other words, distance D indicates information related to the insertion operation. The second time-series image analysis unit 32 can determine this distance D from the image changes.

[0053] The operator's movements 53 can also be observed by a camera 57b that takes images from the head side of the patient 51. Figure 5 shows external images PE3 and PE4, which were acquired in sync with the external images PE1 and PE2 of Figure 4 at the same timing. Therefore, the operator's pushing and pulling operations can also be determined by the change from external image PE3 to external image PE4.

[0054] The dashed circles in Figure 5 indicate the elbow and fingers of the surgeon 53. A comparison of the external images PE3 and PE4 in Figure 5 shows that the surgeon 53 is moving his elbow from the outside to the inside of his body, bringing his right hand 54R, which is grasping the insertion part 12, closer to the patient's anus, thereby attempting to insert the insertion part 12 into the surgeon's body through his anus.

[0055] By determining the amount of change in the position of the operator's 53 elbow and hand, for example, distance D in Figure 4, through a comparison of external images PE1 and PE2, or external images PE3 and PE4, it is possible to determine the amount of pushing and pulling of the tube in Figure 2. For example, by having the information acquisition unit 33 select an external image and using the analysis results of the second time-series image analysis unit 32, the time-series operation information generation unit 34 can determine the pushing and pulling operation on the insertion unit 12 and the amount of that operation (pushing and pulling amount). That is, in the example of Figure 4, the second time-series image analysis unit 32 uses the image portion of the elbow in the external image as an in-image feature portion and determines the change in the in-screen position of the in-image feature portion in order to quantify the operation of the operator 53.

[0056] Figure 6 is used to explain the push-pull operation and push-pull amount obtained by the time-series operation information generation unit 34. In Figure 6, the horizontal axis is time and the vertical axis is the distance D from the left edge of the screen to the elbow. The upper part of the graph shows the movement of the operator's arm (particularly the elbow area image in this case) for analyzing the push-pull operation of the insertion unit 12 by the operator. The lower part shows a series of external images PE5 to PE7 acquired in time series by the camera 57. In each of the external images PE5 to PE7, the three vertical lines indicate the horizontal positions H1, H2, and H3 on the screen. External image PE5 was obtained by imaging near timing t1, for example, and external images PE6 and PE7 were obtained by imaging at timings t2 and t3, respectively.

[0057] The graph used to analyze the pushing and pulling operations of the insertion tube 12 by the operator shows a wave-like result, as the insertion tube is tubular, and when the operator holds (pinches) it with their fingertips and inserts it into the subject's body from the tip, their fingers bump into parts of the subject's body. Therefore, the operator needs to move their grip from the tip of the tube towards the base (proximal end) of the tube, resulting in the operator inserting the tube and then returning their arm to its original position. Typically, the patient is lying down so that the examination is performed in a relaxed state, and the direction of tube insertion is generally horizontal on a screen where gravity is positioned vertically. By analyzing the movement of the operator's arm and elbow in the horizontal direction on the screen, it is possible to determine whether insertion or removal is occurring.

[0058] Furthermore, the insertion section of the endoscope can be as long as 2000 mm, which can lead to significant bending and flexing within the body and lumen. During the pushing and pulling operation of the endoscope (insertion and withdrawal), for example, inserting 100 mm at the hand does not necessarily mean that the tip will be inserted 100 mm further inside. Moving the tip without hand manipulation is difficult, making hand manipulation crucial for the physician. Even if the movement of the imaging unit at the tip of the endoscope insertion section is detected by image, it is not easy to know the amount of insertion by the physician. Therefore, checking the movement of the hand using images from an external camera is often more helpful for insertion and withdrawal than checking changes in the endoscopic image.

[0059] External image PE5 shows an image taken at a time near when the operator 53 began insertion. As the operator 53 extends his elbow forward, at timing t2, for example, the elbow moves away from the left edge of the screen to a distance D. In this case, external image PE6 is obtained, in which the frames change chronologically as time progresses. As described above, by comparing external image PE5 and external image PE6, the distance D, that is, the amount the operator 53 pushed the insertion part 12 in, can be determined.

[0060] For measuring the amount of pushing and pulling, various arbitrary and appropriate positions within the screen can be selected, such as the four corners, four sides, or the center. However, since it is sufficient to detect elbow movement, stationary objects within the screen can also be used as a reference. Whether the elbow movement is toward the subject can be determined by whether the image of the grasped endoscope insertion part is to the left or right of the elbow, or by specifying the subject's position or determining the image of the area where the subject is located. Since the forward direction is the insertion direction, the operator's face position and orientation can also be used, allowing for detection in various ways. Furthermore, when setting up an external camera, a camera position is selected that captures the operator's arm movement and the subject within the field of view. At that time, the left or right side of the screen horizontally is detected, and it is determined whether the elbow is moving from the left or right periphery toward the center of the screen, and that direction can be used as the insertion direction.

[0061] In other words, an external camera captures video, the position of the operator's elbow within the resulting external image is determined, the repeated positional changes over time are monitored, and the change in the direction of the repeated elbow movement (left to right or right to left) is used to quantify the insertion distance. The numerical value used can be either a calculated value representing how far the endoscope actually entered the body, or simply a numerical value based on the change in the image. When using such numerical information to assist with endoscopic operations, the insertion speed may be classified and instructions such as "insert slowly" or "insert carefully." In addition, instructions may be given to stop insertion or to withdraw the endoscope, in which case it is sufficient to obtain information such as whether insertion is being attempted or whether the endoscope has been withdrawn.

[0062] For example, using the length of a doctor's arm as a reference, if a doctor with an upper arm length of 40 cm moves their elbow about 1 / 10 of the length of their upper arm on the screen, it becomes possible to quantify the amount of insertion and withdrawal. In this case, the quantified information can be used to give even more precise instructions, such as inserting 2 cm or inserting at a speed of 3 cm per second.

[0063] Furthermore, when graphing as shown in Figure 1, the insertion amount is plotted on the positive axis and the withdrawal amount on the negative axis. However, these are not cumulative amounts; instead, they are displayed as plus or minus zero when the movement is stopped or the direction is changed, thus representing the insertion amount at each specific moment. This method of representing numerical values ​​is thought to make it easier to focus on the immediate operation at that particular moment. When an operator encounters difficulties during insertion, they are aware of what angle or twist to use and how far to insert, or whether to move in the withdrawal direction, and we believe that instructions indicating the operation at that moment are effective.

[0064] Furthermore, to display the data in this way, the horizontal axis represents the operation time, and when the operator's upper arm does not move for a predetermined time (e.g., 3 seconds), it is considered that there is neither pushing nor pulling, so the vertical axis returns to the origin. When pushing in the insertion direction, the change over time is displayed in the "pushing" direction. When pulling in the withdrawal direction, the change over time is displayed in the "pulling" direction. In other words, the second time-series image analysis unit 32 sets the horizontal position of the image portion in which the operator's arm features are detected to zero when no change is detected in the horizontal direction for a predetermined time in relation to the elapsed operation time, and outputs a numerical value corresponding to the amount of change in each direction when there is a change in the insertion or withdrawal direction. Of course, a graph of accumulated amounts that sequentially accumulates these insertion and withdrawal amounts may also be used.

[0065] The operator 53, upon reaching distance D, releases his grip on the insertion section 12 with his right hand to prevent it from being pulled out before it hits the subject's body, and slides his elbow back to its original position while sliding it toward himself. External image PE7 shows the external image in this case. This operation allows the operator 53 to push the insertion section 12 in again. Normally, the speed of the elbow when returning it to its original position is faster than the speed of the elbow when pushing the insertion section 12 in. When advancing the insertion section during insertion, care is taken as the endoscopic insertion section is moved inside the body (while observing and confirming its movement) (the wave period, here illustrated as the rise becoming slower). However, when changing the hand position to the base of the insertion section (when changing grips), speed is required while ensuring that the insertion section does not come out of the body and does not move too much inside the body, so the wave period becomes shorter. Figure 6 illustrates that the speed of the decline from the peak at distance D becomes faster. This change in speed also allows for the determination of whether the insertion part 12 is being pushed in or pulled out.

[0066] Conversely, during withdrawal, the insertion part is moved cautiously while observing the inside of the body (the wave period becomes longer). However, if this continues, the operator's arm will be fully extended, so it is necessary to change the position of the hand from the base of the insertion part to the tip. In this case, speed is required to prevent the tip of the endoscope from suddenly slipping out of the body during the change of grip, and to avoid moving the tip of the endoscope too much inside the body, so the wave period becomes shorter. As shown in Figure 6, the wave decreases slowly and rises quickly, so the graph shows characteristics opposite to those during insertion.

[0067] In other words, the second time-series image analysis unit 32 analyzes the time changes of the image feature portion in the time-series image obtained by an external camera that photographs the endoscope operator. When the endoscope operator inserts the endoscope into the subject's body, the system analyzes the shape of the wave based on the numerical increase and decrease of the wave waveform obtained by representing the horizontal change in the position of the image portion (image feature portion) in which the operator's arm features are detected on the screen as a graph with the horizontal axis being the time axis, and determines whether an insertion operation, an extraction operation, or neither is being performed. Such a determination can also be made by analyzing the change in numerical values ​​indicating, for example, the position of the elbow per unit time, even without graphing. The determination can be made whether the movement of the elbow is detected or the movement of the arm including the elbow is detected. In other words, the system determines whether an insertion operation, an extraction operation, or neither is being performed by analyzing the speed of the repeated horizontal changes in the position of the image portion in which the operator's arm features are detected on the screen.

[0068] The period from timing t1 to timing t4 in Figure 6 shows the insertion operation performed by the operator 53. From timing t4 onward, it shows the movement of the operator's elbow during the removal operation. The elbow's speed is relatively slow when removing the insertion part 12, and relatively fast when returning the elbow to its position before removal, resulting in a unique wave-like waveform with the characteristics described above.

[0069] Thus, the time-series operation information generation unit 34 can obtain information on the method of insertion operation, including the amount of insertion operation per unit time and its changes, and the method of withdrawal operation, including the amount of withdrawal operation per unit time and its changes, from the elbow movements of the operator 53 during insertion and withdrawal of the insertion unit 12. In other words, this invention includes a technique for determining whether an insertion operation, a withdrawal operation, or not is being performed by analyzing the shape of the wave (which arises from the time difference between rising and falling) based on the numerical increase and decrease of the wave waveform change shown in the graph, for example, when the endoscope operator inserts the endoscope into the subject's body, by determining the time change of the image feature part in the time-series image of the endoscope operator, detecting the horizontal change in the position of the image part showing the operator's arm (arm feature detected) in the screen, and then analyzing the shape of the wave (which arises from the time difference between rising and falling) based on the numerical increase and decrease of the wave waveform change shown in the graph.

[0070] In Figure 6, a simplified example is shown in which insertion and removal operations are determined solely by the movement of the operator's 53 elbow. However, the system may also utilize other movements of the operator's 53 obtained by the camera 57 to determine insertion and removal operations. For example, the insertion part 12 can be inserted by shifting the operator's 53's center of gravity, or by moving the operator's 53's fingers. As shown in Figure 2, the information acquisition unit 33 and the time-series operation information generation unit 34 may determine push-pull operations by obtaining the operator's 53's posture, wrist and finger movements, etc., from external images obtained by the camera 57. Furthermore, the information acquisition unit 33 and the time-series operation information generation unit 34 may also use the analysis results of the internal images by the first time-series image analysis unit 31 as a supplement to determine insertion and removal operations.

[0071] (Recording) The time-series operation information generation unit 34 outputs the generated push-pull operation information to the output unit and the recording unit 40. The output unit and the recording unit 40 have an operation data recording unit 41 configured with a predetermined recording medium. The operation data recording unit 41 records information regarding the push-pull operation of the insertion unit 12 (tube). The information regarding the push-pull operation is collected over time whether the operator is pushing or pulling the insertion unit 12 and the amount of push-pull. The time-series operation information generation unit 34 may graph the push-pull operation information and then record it in the operation data recording unit 41. Since synchronization between the internal image and the external image is established, the time axis of the push-pull operation information corresponds to the time axis of the image frames of the internal image. Therefore, at the timing when an image of a specific part is captured as an internal image, information regarding the push-pull operation when inserting into that specific part is obtained.

[0072] Furthermore, in order to establish a correspondence between each image frame of the endoscopic image and various operations such as pushing and pulling, the operation data recording unit 41 is also supplied with and records the determination results of the observation area and internal image information from the first time-series image analysis unit 31. By recording the determination results of the observation area and internal image information in the operation data recording unit 41, it is possible to determine the content of operations at a specific area based on the information recorded in the operation data recording unit 41 even after the examination.

[0073] (Inference of push-pull operation) Figure 7 is a flowchart showing the flow of creating an inference model when push-pull determination is performed by inference. Note that the flow in Figure 7 may be executed by the data processing device 30, or it may be executed by other computer systems not shown.

[0074] For example, the data processing device 30 includes a neural network (not shown). Deep learning is a multi-layered structure of the machine learning process using neural networks. A typical example is the "feedback neural network," which sends information from front to back to make a judgment. In its simplest form, this requires only three layers: an input layer consisting of m1 neurons, a hidden layer consisting of m2 neurons given by parameters, and an output layer consisting of m3 neurons corresponding to the number of classes to be classified. The neurons in the input layer and the hidden layer, and the neurons in the hidden layer and the output layer are connected by connection weights, and a bias value is added between the hidden layer and the output layer, making it easy to form logic gates. Three layers are sufficient for simple classification, but by using many hidden layers, it becomes possible to learn how to combine multiple features during the machine learning process. In recent years, networks with 9 to 152 layers have become practical due to their relationship with learning time, judgment accuracy, and energy consumption.

[0075] Various publicly known networks can be used as the network for machine learning. For example, R-CNN (Regions with CNN features), FCN (Fully Convolutional Networks), and 3DCNN (3D Convolutional Neural Network) using CNN (Convolutional Neural Network) can be used. These involve a process called "convolution" to compress image features, operate with minimal processing, and are strong in pattern recognition. Furthermore, to handle more complex information and to support information analysis where the meaning changes depending on the order and sequence, a "recurrent neural network" (fully connected recurrent neural network) where information flows bidirectionally can be used.

[0076] To realize these technologies, conventional general-purpose computing circuits such as CPUs and FPGAs can be used, but since much of the processing in neural networks involves matrix multiplication, specialized GPUs and Tensor Processing Units (TPUs) are sometimes used. In recent years, these artificial intelligence (AI) dedicated hardware, called "Neural Network Processing Units (NPUs)," have been designed to be integrated and embedded together with CPUs and other circuits, and are sometimes part of the processing circuit.

[0077] Furthermore, inference models may be obtained using various publicly known machine learning techniques, not limited to deep learning. For example, there are methods such as support vector machines and support vector regression. The learning here involves calculating the classifier weights, filter coefficients, and offsets, but there are also methods that utilize logistic regression. When a machine is to make a judgment, a human needs to teach the machine how to make the judgment. In this example, a method was adopted to derive image judgment using machine learning, but other rule-based methods that apply rules acquired by humans through experience and heuristics may also be used to make specific judgments.

[0078] The data processing device 30 generates training data for training a neural network. Specifically, in S1, the data processing device 30 acquires external images (external videos) for multiple cases using the external camera 20. The data processing device 30 receives the acquired external videos and displays the external images on display devices 55, 56, etc. (S2). In S3, the data processing device 30 determines whether or not to include internal images (internal videos) acquired by the imaging unit 12a in the construction of the inference model. If it does, in S4, the data processing device 30 compares the internal images and external images and synchronizes the origin of time and frames while considering the frame rate.

[0079] In S5, the data processing device 30 determines whether it is an insertion or removal time. If it is an insertion or removal time, in S6, the data processing device 30 adds an annotation to the corresponding frame indicating that it is an insertion or removal time.

[0080] Next, in S7, the data processing device 30 determines whether or not there are any comments from the operator. If there are comments, in S8, the data processing device 30 annotates the comments on the corresponding frames. For example, if comments from an expert are annotated, it is conceivable that effective support can be provided using these comments.

[0081] Next, in S9, the data processing device 30 determines whether the hand and elbow holding the tip of the endoscope are within the frame. If they are within the frame, in S10, the data processing device 30 annotates the corresponding frame to indicate the position of the hand and elbow. This eliminates the need to determine the position of the elbow and hand through image analysis. The data processing device 30 also annotates the amount and direction of elbow movement per unit frame. This makes it possible to infer the amount and direction of elbow movement, i.e., the pushing and pulling direction and amount of pushing and pulling of the insertion section 12, from the input image.

[0082] Next, in S11, the data processing device 30 determines whether a sufficient number of data points for training data have been collected. If not, in S12, the data processing device 30 selects a different video and returns to processing in S2. If sufficient data points have been collected, the data processing device 30 creates an inference model (S13).

[0083] Next, in S14, the data processing device 30 receives a test image and determines whether a good inference result has been obtained. If not, in S15, the data processing device 30 modifies the training data by selecting or discarding training data, and returns to processing in S13. If a good inference result is obtained, the process is terminated.

[0084] In this way, by inputting an external image or both an external and an internal image to the inference model constructed in the data processing device 30, information such as the push / pull direction and amount, as well as comments, is presented as an inference result corresponding to each frame of the input image.

[0085] (Determination of rotation amount) Figure 8 is an explanatory diagram for explaining the determination of the rotation (twisting) operation of the insertion part 12, and Figure 9 is a flowchart for explaining the operation determination in Figure 8.

[0086] Figure 8 shows an internal image of a specific location. In Figure 8, the shaded areas indicate relatively dark areas, and the roughly circular shaded areas indicate the deeper parts of the lumen. The dashed lines on the internal image in Figure 8 represent vertical and horizontal lines (hereinafter, this horizontal line is simply referred to as the horizontal line) passing through the center of the image.

[0087] In step S21 of Figure 9, the first time-series image analysis unit 31 performs image analysis on the internal image and detects feature points of the internal image PI1, which is an image of a predetermined specific area. For example, the first time-series image analysis unit 31 may detect the pointed shape of the internal image PI1 enclosed by the dashed circle as a feature point. Note that internal images PI2 to PI4 in Figure 8 were obtained by imaging the same specific area as internal image PI1.

[0088] The first time-series image analysis unit 31 may, as shown in S22, determine a feature point located in the center of the first quadrant of a predetermined frame X when detecting a feature point as a feature part within an image. The feature point of internal image PI1 is located at the edge of the first quadrant. In this case, depending on the movement of the insertion unit 12 (imaging unit 12a), the feature point may move out of the field of view. For this reason, the first time-series image analysis unit 31 selects internal image PI2 as frame X and detects a feature point located in the center of the first quadrant. At a predetermined timing, the angle between the horizontal line on internal image PI2 and the feature point is θ0 degrees. The first time-series image analysis unit 31 determines the angle θ0 between the horizontal line and the feature point (S23).

[0089] In S24, the first time-series image analysis unit 31 determines whether or not there is a frame that is temporally adjacent to and similar to the internal image PI2. If no such frame exists (NO in S24), the first time-series image analysis unit 31 proceeds to S28 to determine whether or not the analysis is complete. If the analysis is complete, the process ends; otherwise, the process returns to S22 to determine feature points for another frame X.

[0090] If the first time-series image analysis unit 31 determines in S24 that there are temporally adjacent and similar frames, in S25 it determines a feature point that is identical to the feature point of frame X and is located in the center of the Nth quadrant of frame X+m. Then, in S26 the first time-series image analysis unit 31 determines the angle θ1 between the horizontal line and the feature point for the internal image PI3, which is frame X+m.

[0091] Figure 8 shows that, relative to the internal image PI2, the imaging unit 12a rotated in a plane parallel to the imaging surface, i.e., the insertion unit 12 was twisted, resulting in a change in the angle between the horizontal line and the feature point in image PI3 to θ1 degree, and in image PI4 to a change in the angle between the horizontal line and the feature point to θ2 degrees. In other words, the direction and magnitude of the change in the angle between the horizontal line and the feature point correspond to the twisting direction and amount of twist of the insertion unit 12. The first time-series image analysis unit 31 obtains the twisting direction and amount of twist as analysis results based on the changes in the internal image. In other words, the time-series operation information generation unit 34 obtains the time difference of m frames and the information of θ1-θ2 as information on the twisting operation (S27). For example, the first time-series image analysis unit 31 may determine that a clockwise rotation of the feature point is a leftward twist, and a counterclockwise rotation of the feature point is a rightward twist.

[0092] Next, the first time-series image analysis unit 31 returns to processing S24 and repeats steps S24 to S27. In this way, information regarding the twisting operation is obtained for every m frames. The amount of twisting operation is a maximum of 180 degrees in both the left and right directions.

[0093] The information acquisition unit 33 and the time-series operation information generation unit 34 generate information related to twisting operations using the analysis results of the internal image by the first time-series image analysis unit 31 and output it to the output unit and the recording unit 40. The operation data recording unit 41 of the output unit and the recording unit 40 records information related to the twisting operation of the insertion unit 12 (tube). The information related to twisting operations is collected at time intervals corresponding to the time of the internal image, indicating in which direction the operator is twisting the insertion unit 12 and the amount of twisting. The time-series operation information generation unit 34 may graph the information on twisting operations before recording it in the operation data recording unit 41. In this way, information related to the twisting operation when inserting into a specific area is obtained at the timing when an image of that specific area is captured as an internal image.

[0094] (Bending Determination) Figure 10 is an explanatory diagram for explaining the determination of the amount of bending during the bending operation of the insertion part 12, and Figure 11 is a flowchart for explaining the bending amount determination in Figure 10.

[0095] Figure 10 shows an internal image of a predetermined specific region using the same notation as in Figure 8. In S31 of Figure 11, the first time-series image analysis unit 31 performs image analysis of the internal image and detects feature points of the internal image PI11, which is an image of a predetermined specific region (S32). For example, the first time-series image analysis unit 31 may detect the portion enclosed by the dashed circle in the internal image PI11 as feature point lm0. Note that internal images PI12 and PI13 in Figure 10 were obtained by imaging the same specific region as internal image PI11. When detecting feature points, the first time-series image analysis unit 31 may determine feature points that are located in the central part of the internal image PI11, which is an image of a predetermined frame X. In the example of Figure 10, feature point lm0 is an image portion that shows approximately the center in the deep direction of the lumen.

[0096] In S34, the first time-series image analysis unit 31 determines whether or not there is a frame that is temporally adjacent to and similar to the internal image PI12. If no such frame exists (NO in S34), the first time-series image analysis unit 31 proceeds to S37 to determine whether or not the analysis is complete. If the analysis is complete, the process ends; otherwise, the process returns to S32 to determine feature points for another frame X.

[0097] If the first time-series image analysis unit 31 determines in S34 that there are temporally adjacent and similar frames, in S35 it determines the position of the feature point lm1 of frame X + m, which is the same feature point as the feature point of frame X. Then, in S36 the first time-series image analysis unit 31 determines the time difference, direction of position change, and amount of change for the m frames of the internal image PI12 which is frame X + m.

[0098] In Figure 10, by changing the orientation of the optical axis of the imaging unit 12a up, down, left, and right, that is, by operating the dial 13a, the curved portion 12b of the insertion unit 12 is curved left, right, up, and down. As a result, with respect to the internal image PI 11, image PI 12 shows that the position of the feature points in the image has changed horizontally, and image PI 13 shows that the position of the feature points in the image has changed vertically. Specifically, the direction and magnitude of the change in the horizontal position of the feature points correspond to the direction and magnitude of the left-right orientation of the curved portion 12b, i.e., the operation of the left-right dial 13a, and the direction and magnitude of the change in the vertical position of the feature points correspond to the direction and magnitude of the up-down orientation of the curved portion 12b, i.e., the operation of the up-down dial 13a.

[0099] The first time-series image analysis unit 31 obtains the operating direction and amount of the dial 13a as analysis results based on the changes in the internal image. The time-series operation information generation unit 34 obtains the time difference of m frames and the operating direction and amount of the dial 13a as dial operation information. In the example in Figure 10, internal image PI12 is an image obtained by operating the dial to the right from the state of internal image PI11, and internal image PI13 is an image obtained by operating the dial downward from the state of internal image PI11.

[0100] Next, the first time-series image analysis unit 31 returns to processing S34 and repeats steps S34 to S36. In this way, information regarding the dial operation is obtained for every m frames. The upper limits of the dial operation amount may be, for example, 210 degrees up, 90 degrees down, and 100 degrees left and right.

[0101] The information acquisition unit 33 and the time-series operation information generation unit 34 generate information related to dial operation using the analysis results of the internal image by the first time-series image analysis unit 31 and output it to the output unit and the recording unit 40. The operation data recording unit 41 of the output unit and the recording unit 40 records information related to the dial operation of the dial 13a. The information related to dial operation is collected for each time period corresponding to the time of the internal image, indicating in which direction the operator rotates the dial 13a and the amount of rotation. The time-series operation information generation unit 34 may also graph the dial operation information and then record it in the operation data recording unit 41. In this way, at the timing when an image of a specific part is captured as an internal image, information related to the dial operation when inserting into that specific part is obtained.

[0102] Thus, the present invention enables the construction of an endoscope operation quantification system that quantifies the characteristics of the operations performed by an endoscope operator when inserting or withdrawing an endoscope having an imaging unit at its tip into a subject's body. Specifically, a first time-series image analysis unit for determining the time changes of image features in time-series images obtained by the endoscope imaging unit and a second time-series image analysis unit 32 for determining the time changes of image features in time-series images obtained by an external camera that photographs the endoscope operator divide the types of operations performed by the endoscope operator into items such as endoscope insertion / withdrawal, twisting of the endoscope insertion unit, and angle of the endoscope tip, and a time-series operation information generation unit 34 analyzes and generates changes in the amount of operation for each item. If these amounts of operation are represented on the same time axis, a graph like the one in Figure 1 can be created, but it is preferable to provide a synchronization unit that synchronizes the time-series images obtained by the endoscope imaging unit and the time-series images obtained by the external camera. Based on the time-series changes of each synchronized image, the types of operations performed by the endoscope operator can be categorized into endoscope insertion / removal, twisting of the endoscope insertion section, and angle of the endoscope tip. The changes in the amount of operation for each item can be analyzed, and by combining this with the endoscopic images (internal images) at each timing, it is possible to determine under what circumstances (situations in which endoscopic images were obtained) and by what operations insertion into the lumen can be performed, thereby obtaining guidance information. Although there are differences in the structure of the lumen, just as there are differences in body shape among subjects, similar images can often be obtained inside the lumen. In such cases, it may be difficult to determine a difficult insertion situation with just one image or frame. Therefore, it may be possible to determine a difficult situation for endoscope insertion using multiple images obtained in a time series.

[0103] Furthermore, the time-series operation information generation unit 34 also records information regarding the operation of the switch 13b provided on the endoscope operation unit in the operation data recording unit 41. In the example in Figure 1, the switch 13b is an on / off switch, and the time-series operation information generation unit 34 indicates that an on or off operation was performed based on the pulse shape. In addition, by applying annotation using the scale provided on the insertion unit 12, the amount of insertion of the insertion unit 12 into the body can also be inferred.

[0104] Switch 13b represents a group of multiple switches, and the operation of each switch is to be independently distinguishable. These switches correspond to functions such as water supply, air supply, suction, light source switching, special light observation, and image processing. With these operations, the endoscope may be inserted, observed, or diagnosed. Recording what kind of endoscopic operations a skilled model physician performs in what situations (such as the relationship between endoscopic images and switch operations) can be provided as a reference for inexperienced physicians. For example, when anterior obstruction occurs during endoscope insertion, air may be injected into the lumen to inflate it. By recording when this switch operation was performed, and in conjunction with the change in the endoscopic image from a contracted lumen to an expanded state, it is possible to determine what techniques the skilled model physician used.

[0105] Thus, the operation data recording unit 41 records various operation data related to at least one endoscopic operation, such as pushing and pulling, twisting, dial operation, and switch operation of the tube (insertion section 12), which are the first and second operation information. In addition, this operation data is recorded in graph form.

[0106] Alternatively, as shown in Figure 18, the insertion starting point, such as the anus, may be used as the origin to show how far the endoscope has entered the body. The vertical axis represents how far the endoscope has entered the body from the insertion starting point, and the horizontal axis represents the elapsed time during the insertion operation. This can be calculated by accumulating the values ​​from the insertion / removal graph in Figure 1 or the insertion / removal information detected in Figure 6. Alternatively, an indicator or code indicating the insertion amount may be attached to the endoscope insertion section, captured by an external camera, and the code corresponding to the inserted amount read from the insertion section, analyzed, and converted into an insertion amount. A database linking codes and insertion amounts may be prepared and referenced, or numerical values ​​may be written to the codes and read. The second time-series image analysis unit 32 determines the characteristics of the operator's arm from the images captured by the external camera in relation to the elapsed operation time. As an application, the shooting position of the external camera may be adjusted to capture the insertion situation, and numbers, symbols, or codes written on the tube of the endoscope insertion section may be read. In this case, the system monitors the time series by determining which part was inserted at what point in each image. In other words, the second time-series image analysis unit 32 converts the insertion length, which is the change in the insertion length of the endoscope insertion unit in the insertion or withdrawal direction, into a numerical value corresponding to the endoscope insertion and withdrawal operations performed by the endoscope operator, in relation to the passage of operating time. This corresponds to the cumulative value of the change in the vertical axis of the graph in Figure 1, so it can also be converted from the graph in Figure 1. This graph provides information on how far the endoscope insertion unit has entered the body from its tip, so it is possible to determine which anatomical position (location) in the lumen the endoscope insertion unit has reached when it has entered the body by a specific length from its tip (e.g., reaching 600 mm indicates passing through the sigmoid colon, and reaching the left colic flexure (splenic flexure) after passing through the sigmoid colon when it has reached 400 mm), and the pushing and pulling operations leading up to that point can be determined by the waveform of the fine changes in the insertion amount (for example, if the insertion amount increases, it indicates that it is being inserted, and if the insertion amount decreases, it indicates that the endoscope insertion unit is being moved in the withdrawal direction). This information may be used as a guide.

[0107] In this embodiment, images from the endoscope and an external camera are captured in sync, and information regarding the endoscopic operation is obtained by analyzing each image. This makes it possible to collect information indicating the operator's posture, how they hold the endoscope in both hands, and their hand movements, thereby reliably obtaining information on the characteristics of an experienced physician's endoscopic operation.

[0108] (Second Embodiment) Figure 12 is a block diagram showing the second embodiment. Figure 12 shows an endoscope operation guide system using the endoscope data acquisition system of Figure 1. In Figure 12, the same reference numerals are used for components that are the same as those in Figure 1, and their descriptions are omitted.

[0109] The endoscope operation guide system 60 includes an endoscope data acquisition system 61, an endoscope operation guide generation unit 63, and a model operation information generation unit 81. Each of the endoscope data acquisition system 61, the endoscope operation guide generation unit 63, and the model operation information generation unit 81 may be composed of a processor using a CPU, FPGA, NPU, etc., and may operate according to a program stored in a memory (not shown) to control each part, or may some or all of the functions be realized by hardware electronic circuits.

[0110] The endoscope data acquisition system 61 has the same configuration as shown in Figure 1. The operation data recording unit 41 outputs the recorded various operation data and internal image information to the endoscope operation guide generation unit 63. The endoscope operation guide generation unit 63 compares the operation data currently acquired by the endoscope data acquisition system 61 (hereinafter also referred to as current operation data) with past operation data obtained from the model operation information generation unit 81 (hereinafter also referred to as model operation data), and generates guide information for endoscope insertion support based on the comparison results.

[0111] In this case, considering differences in the type of endoscope used and the surgical procedure, it may not be possible to simply compare the two. For example, comparing operational data from upper gastrointestinal endoscopy with operational data from lower gastrointestinal endoscopy may not be meaningful. Therefore, the target classification unit 70 generates target classification data based on user operations, indicating what type of target the current operational data output from the endoscopy data acquisition system 61 pertains to, and provides this data to the endoscopy operation guide generation unit 63. Medical records, CT scans, MRI images, etc., can be used as target classification data.

[0112] The model operation information generation unit 81 includes an operation data recording unit 41 similar to the operation data recording unit 41 provided in the endoscope data acquisition system 61. The operation data recording unit 41 in the model operation information generation unit 81 records model operation data that was previously collected by the endoscope data acquisition system 61, for example, by the endoscope operation of a model physician skilled in endoscope operation. The endoscope operation data extraction unit 82 extracts model operation data of the endoscope operation from the operation data recording unit 41 and outputs it to the endoscope operation guide generation unit 63. The target unit classification unit 83 generates target unit classification data indicating what kind of target the model operation data output from the endoscope operation data extraction unit 82 pertains to, and outputs it to the endoscope operation guide generation unit 63. The model operation information generation unit 81 also outputs internal image information from the operation data recording unit 41 to the endoscope operation guide generation unit 63.

[0113] The model operation data recorded in the operation data recording unit 41 of the model operation information generation unit 81 includes, for example, information on insertion procedures performed by skilled physicians in areas where insertion is difficult, i.e., textbook-like information on how to perform procedures in areas where insertion is difficult. For inexperienced endoscopic insertion operators, it is conceivable that they may need not only information on insertion procedures performed by skilled physicians, but also information on standard insertion procedures. The standard operation database 84 stores standard operation data (hereinafter, standard operation data may also be referred to as model operation data), which is information on such standard procedures. Standard operation data from the standard operation database 84 is also supplied to the endoscopic operation guide generation unit 63.

[0114] The endoscope operation guide generation unit 63 includes an endoscope operation determination unit 64 and a display unit 69. The endoscope operation determination unit 64 generates guide information for guiding insertion operations based on a comparison between model operation data previously generated by the time-series operation information generation unit 34 and current operation data currently being generated by the time-series operation information generation unit 34.

[0115] The model operation data recorded in the operation data recording unit 41 of the model operation information generation unit 81 can be thought of as, for example, the quantification of the characteristics of an operation performed by a skilled physician when inserting, examining, or withdrawing an endoscope into the body of a specific subject, using the endoscopic operation quantification system, which is a feature of this invention. In other words, when the output of the first time-series image analysis unit for determining the time change of the image feature portion in the time-series image obtained by the endoscope imaging unit and the output of the second time-series image analysis unit 32 for determining the time change of the image feature portion in the time-series image obtained by the external camera that photographs the endoscope operator are synchronized with the time-series image obtained by the endoscope imaging unit by the synchronization unit, the time-series operation information generation unit analyzes and generates the presence or absence of an operation and the change in the amount of operation for each item of the operation, divided into items of endoscope insertion / withdrawal, twisting of the endoscope insertion part, and angle of the endoscope tip, for each endoscope image during the endoscope operation, and this analysis result becomes the standard model operation data. The current operation data generated during endoscopic operation by an unskilled operator is obtained by classifying the type of operation into categories such as endoscope insertion / removal, twisting of the endoscope insertion section, and angle of the endoscope tip, and determining the presence or absence of operation and the time-series change in the amount of operation for each category. By comparing the model operation data with the current operation data at each timing in the above time series and combining it with the images obtained by the endoscopic imaging unit, guide information can be obtained. The endoscopic operation quantification system can provide useful information to such a guide generation unit.

[0116] The endoscope operation determination unit 64 includes a site determination unit 65, an insertion difficulty determination unit 66, a reference determination unit 67, and a guide generation unit 68. The site determination unit 65 determines which part of the endoscope the internal image from the endoscope data acquisition system 61 belongs to. Note that if the current operation data from the operation data recording unit 41 includes information on the determination result of the observation site, the site determination unit 65 can be omitted. The site determination unit 65 obtains model operation data corresponding to the determined site from the endoscope operation data extraction unit 82 or the reference operation database 84.

[0117] As mentioned above, the insertion state of the endoscope does not always change or progress as intended by the surgeon. For example, even if the surgeon attempts to insert (advance) the endoscope, it does not necessarily result in a smooth advance. For instance, due to flexion of the endoscope or friction with the intestinal wall, it may only advance slightly, or if a sticking phenomenon occurs in the splenic bay, the tip may even move away from the direction of advancement. Continuing to advance the endoscope in such difficult insertion situations can further worsen the insertion situation or cause pain to the patient.

[0118] The insertion difficulty determination unit 66 determines the insertion difficulty situation, while the standard determination unit 67 determines the standard operation situation in which a standard operation should be performed. The insertion difficulty situation occurs when the insertion of the insertion part becomes relatively difficult. The insertion difficulty determination unit 66 determines whether or not the insertion is difficult, i.e., whether or not the insertion is difficult, for example by image analysis of the internal image. Alternatively, for example, the insertion difficulty determination unit 66 may acquire first operation information corresponding to the operator's operation based on image changes in the internal image, acquire second operation information corresponding to the operator's operation based on image changes in the external image, and determine the insertion difficulty situation based on a comparison of the first operation information and the second operation information. For example, if the second operation information indicates that the insertion part 12 is being inserted, but the first operation information indicates that insertion has not been performed, in such a case, the operation difficulty determination unit 66 determines that the insertion is difficult. Alternatively, for example, the insertion difficulty determination unit 66 may determine that the insertion is difficult if the internal image is an image of a blocked lumen. Furthermore, for example, the insertion difficulty determination unit 66 may determine that insertion is difficult if the internal image does not include any hollow image portions in the insertion direction of the insertion unit.

[0119] For example, when the insertion tube is advancing through the lumen, the image portion at the back of the lumen (the end in the direction of the lumen length) where illumination light from the tip of the endoscope does not reach will have a low-brightness lumen cross-sectional shape (often roughly circular). As the insertion tube advances through the lumen, this image portion will be located approximately in the center of the endoscopic image, and a continuous image will be obtained at the periphery of the image in which the pattern of the lumen wall moves toward the periphery of the image. The insertion difficulty determination unit 66 can determine that the insertion tube is advancing through the lumen in this manner by analyzing the internal image, and conversely, it can determine that insertion is difficult and the insertion tube is not advancing through the lumen, such as when there is no cavity in front of the insertion tube. The insertion difficulty determination unit 66 determines that insertion is difficult when an internal image that does not include such a cavity is input.

[0120] The standard determination unit 67 determines a standard operation situation in which a standard operation should be performed, other than a situation where insertion is difficult. For example, it determines a situation in which a standard operation should be performed, such as when an internal image obtained when the aforementioned insertion part is advancing inside the lumen is input.

[0121] The guide generation unit 68 compares the current operation data from the endoscope data acquisition system 61 acquired by the site determination unit 65 with the model operation data from the model operation information generation unit 81 or the reference operation database 84, and generates guide information based on the comparison result. In this case, the guide generation unit 68 may generate guide information by comparing the current operation data with the model operation data in the case of insertion difficulty. Alternatively, in the case of basic operation conditions, the guide generation unit 68 may omit the comparison between the current operation data and the model operation data and generate guide information by comparing the current operation data with the reference operation data. The guide generation unit 68 provides the generated guide information to the display unit 69 for display. The display unit 69 is composed of a display device such as an LCD (liquid crystal display panel), and is capable of displaying internal images, external images, and guide information.

[0122] For example, the guide generation unit 68 may be configured to display a warning on the display unit 69 when insertion is difficult. The guide generation unit 68 may also be configured to display a warning on the display unit 69 when the difference between the current operation data and the model operation data exceeds a predetermined threshold. For example, in a situation where insertion is difficult, if at least one of the push / pull amount, twist amount, or dial operation amount of the insertion part differs by a predetermined value or more between the current operation data and the model operation data, the guide generation unit 68 may display a warning on the display unit 69 indicating that insertion is difficult for the operator.

[0123] Furthermore, the guide generation unit 68 may generate and display guide information based on the standard operation data corresponding to the standard operation status when a standard operation status is reached.

[0124] Figures 13 and 14 are for illustrating the guide information. Figure 13 is an explanatory diagram showing the guide information displayed on the display screen 69a of the display unit 69. Figure 14 is a flowchart showing the insertion guide generated by the endoscope operation guide generation unit 63.

[0125] Figure 13 shows an example of guide displays G1 to G6 displayed on the display screen 69a. The endoscope operation guide generation unit 63 displays internal images (hereinafter referred to as endoscope image displays) IE supplied from the endoscope data acquisition system 61 on the left side of the display screen 69a. The endoscope operation guide generation unit 63 also displays insertion guides on or next to the endoscope image displays IE. In the example of guide displays G1 and G2, the endoscope image displays IE show images of a predetermined specific area, the shaded areas indicate areas with relatively low brightness, and the roughly circular shaded areas indicate the direction towards the back of the lumen. In Figure 8, the dashed straight line on the endoscope image displays IE represents a vertical and horizontal line passing through the center of the endoscope image displays IE, and their intersection indicates the position on the optical axis of the imaging unit 12a. The dashed circle in the center of the endoscope image displays IE indicates the position towards the back of the lumen.

[0126] In S40 of Figure 14, the endoscope operation guide generation unit 63 determines whether the insertion direction of the insertion unit 12 is correct (OK). If the insertion direction is OK, the endoscope operation guide generation unit 63 displays a circular frame in green around the lumen image on the endoscope image display IE, for example (S41). An example of this display in the guide display G1 of Figure 13 is shown.

[0127] In the example of guide display G1, the endoscopic image display IE indicates that the optical axis of the imaging unit 12a, located at the tip of the insertion unit 12, is located at the back of the lumen, and that the imaging unit 12 is imaging in the direction in which the insertion unit 12 should advance. In this case, the endoscopic operation guide generation unit 63 displays a guide display G1a of a predetermined color or pattern within a circular frame, indicating that it is OK to insert the insertion unit 12. For example, a color such as green may be used for guide display G1a to encourage insertion. The endoscopic operation guide generation unit 63 also displays the message "Please insert" as guide display G1b in guide display G1.

[0128] In this example, the endoscope operation guide generation unit 63 creates the guide display G1 based on the reference operation data corresponding to the reference operation status, without comparing the current operation data with the reference operation data.

[0129] Even if the insertion direction is determined to be OK in S40, it may be necessary to fine-tune the orientation. In S42, the endoscope operation guide generation unit 63 determines whether or not it is necessary to bend the curved portion 12b of the insertion portion 12. If the endoscope operation guide generation unit 63 determines that it is necessary to bend the curved portion 12b, it flashes, for example, the arc-shaped portion of the circular frame corresponding to the bending direction in green (S48). Guide display G2 shows an example of the display in this case.

[0130] In the example of guide display G2, the endoscopic image display IE indicates that the optical axis of the imaging unit 12a, located at the tip of the insertion section 12, is slightly shifted to the left from the back of the lumen, and that the imaging unit 12 is imaging an object slightly shifted to the left from the direction in which the insertion section 12 should advance. In this case, the guide generation unit 68 compares the current operation data with the reference operation data to determine that it is better to bend the insertion section 12 to the right, generates guide information indicating this determination result, and provides it to the display unit 69. For example, as in the example of guide display G2, the endoscopic operation guide generation unit 63 displays arc-shaped guide displays G2a1 and G2a2 of a predetermined color or pattern to indicate that it is better to bend the insertion section 12 to the right. In guide display G2, a circular frame is formed by the arc-shaped guide displays G2a1 and G2a2, and the position of the guide display G2a2 on the circumference indicates the direction in which the insertion section 12 should be curved. For example, the guide indicator G2a2 may use a flashing color such as green to indicate the direction of curvature, and the guide indicator G2a1 may use a color such as blue.

[0131] Furthermore, the endoscope operation guide generation unit 63 displays a guide display G2a3 using an image of an arrow indicating the direction of bending. In addition, the endoscope operation guide generation unit 63 displays the message "right bend" as guide display G2b in guide display G2. Furthermore, the endoscope operation guide generation unit 63 may also display the operation method using guide display G2c, which uses images of a dial, a finger (filled in), and an arrow for bending. Guide display G2c indicates which of the two dials for up / down and left / right should be operated by color coding (diagonal lines and solid color in Figure 13). The display of guide display G2 allows even physicians who are inexperienced in insertion procedures to learn the operation method based on model operation data, enabling them to perform insertion relatively easily and smoothly.

[0132] If the endoscope operation guide generation unit 63 determines in S40 that the insertion direction is not OK, it determines in S45 whether the insertion direction is incorrect (NG). If the insertion direction is not NG, the endoscope operation guide generation unit 63 proceeds to S42; if the insertion direction is NG, it proceeds to S46. In S46, the endoscope operation guide generation unit 63 determines whether the device is stuck. If a stuck device is expected, the endoscope operation guide generation unit 63 displays a circular frame around the lumen image, for example, in red. If the insertion direction is NG, the endoscope operation guide generation unit 63 may also display an arc-shaped portion corresponding to the NG direction in red.

[0133] Furthermore, if the endoscope operation guide generation unit 63 determines that the image is stuck, in S47 it will cause a circular frame around the lumen image to flash red, for example, in order to recover from the stuck state. Guide displays G3 to G6 show examples of displays in this case.

[0134] In the examples of guide displays G3 to G6, the endoscopic image display IE shows that the image of the back of the lumen is not displayed, and an oval-shaped bubble is displayed, indicating that the lumen to be inserted has been lost. In the example of guide display G3, the endoscopic operation guide generation unit 63 displays guide display G3a, which flashes in a predetermined color or pattern within a circular frame, indicating that it is better to withdraw the insertion part 12 without inserting it. For example, guide display G3a may use a flashing color such as red to indicate that insertion should not be performed. The endoscopic operation guide generation unit 63 also displays the message "withdrawal direction" as guide display G3b in guide display G3. Note that in the case of a predicted stack, the red circular frame does not flash.

[0135] In the example of guide display G4, the endoscope operation guide generation unit 63 displays a guide display G4a of a predetermined color or pattern in a circular frame indicating that it is better to withdraw the insertion part 12 without inserting it. For example, as guide display G4a, a color such as red may be flashed to indicate that insertion should not be performed. In the example of guide display G4, the endoscope operation guide generation unit 63 also displays an image showing the withdrawal procedure from the physician's perspective as guide display G4b.

[0136] In the example of guide display G5, the endoscope operation guide generation unit 63 displays a guide display G5a in a predetermined color or pattern within a circular frame indicating that it is not advisable to insert the insertion section 12. For example, the guide display G5a may be a flashing color such as red to indicate that insertion should not be performed. In the example of guide display G5, the endoscope operation guide generation unit 63 also uses the output of an insertion shape detection device (not shown) to display an image showing the insertion shape of the insertion section as guide display G5b (S43 in Figure 14). Guide display G5b allows the user to understand which part of the endoscope is difficult to insert.

[0137] In the example of guide display G6, the endoscope operation guide generation unit 63 displays a guide display G6a in a predetermined color or pattern within a circular frame indicating that it is better not to insert the insertion section 12. For example, the guide display G6a may be a flashing color such as red to indicate that insertion should not be performed. In the example of guide display G6, the endoscope operation guide generation unit 63 also displays an image as guide display G6b that shows the relationship between time and the insertion amount of the insertion section 12 (S43 in Figure 14). Note that curve C1 in guide display G6b is obtained from current operation data, and curve C2 is obtained from model operation data.

[0138] In S44, the endoscope operation guide generation unit 63 determines whether or not to terminate the display of guide information. If it does not terminate, the endoscope operation guide generation unit 63 returns to S40; if it does terminate, it terminates the guide display.

[0139] (Operation) Next, the operation of the embodiment configured as described above will be explained with reference to Figures 15 to 17. Figure 15 is a flowchart for explaining the operation of the second embodiment, and Figures 16 and 17 are explanatory diagrams for explaining the operation of the second embodiment.

[0140] In step S51 of Figure 15, the data processing device 30 of the endoscopic data acquisition system 61 acquires a first time-series image from the imaging unit 12a. The data processing device 30 also acquires information regarding the operation of various switches on the endoscope. In step S52, the data processing device 30 acquires a second time-series image from the external camera 20.

[0141] In S53, the data processing device 30 determines whether the first time-series image and the second time-series image are synchronized (OK). If they are not synchronized, the data processing device 30 performs synchronization countermeasures (S54). For example, it is possible to establish synchronization based on the changes in the first time-series image and the changes in the second time-series image. Once synchronization is established, the data processing device 30 adds synchronization information to the first and second time-series images.

[0142] The data processing device 30 determines, based on the external image, whether or not the position of the operator, such as a doctor, and the movement of their arm can be confirmed (S56). If it is not possible to determine, the data processing device 30 displays a warning on the display unit 69 for adjusting the position of the camera or the operator (S57). If it is possible to determine in S56, the data processing device 30 determines, based on the internal image, whether or not the image shows the insertion direction, and determines any changes in the internal image (S58).

[0143] In S59, the data processing device 30 determines, for example, the insertion direction of the insertion section 12, the twist, and the changes in the up, down, left, and right directions of the bending section 12b based on the second time-series image (external image). In addition, in S60 to S62, the data processing device 30 determines the image of the external image or its changes, determines the operator's posture and changes in posture based on the external image, and determines the insertion direction and insertion speed of the insertion section 12 based on the external image.

[0144] In this way, the data processing device 30 acquires operation data, which is information about the operator's actions when operating the endoscope 10, in the endoscope data acquisition system 61, and records it in the operation data recording unit 41. The operation data recording unit 41 outputs the operation data and the first time-series image to the endoscope operation guide system 60.

[0145] In S63, the endoscope operation determination unit 64 of the endoscope operation guide system 60 determines whether or not the insertable pattern position can be confirmed from the internal image frame. The insertable pattern position is the position at the back of the lumen, and if the image frame of the internal image includes the image portion at the back of the lumen, it is determined that the insertable pattern position can be confirmed.

[0146] Figure 16 shows internal images PI21-PI26 and external images PE21-PE26 obtained by the endoscopic data acquisition system 61 during the endoscope insertion process. Figure 17 also shows internal images PI31-PI36 and external images PE31-PE36 obtained by the endoscopic data acquisition system 61 during the endoscope insertion process. Note that external images PE21-PE26 and external images PE31-PE36 were obtained by imaging with different external cameras. In Figures 16 and 17, the shaded areas in internal images PI21-PI26 and PI31-PI36 represent the image portion deep within the lumen.

[0147] Internal images PI21, PI22, PI26, PI31, PI32, and PI36 include the image portion at the back of the lumen. Therefore, at the timing of these image frames, the endoscope operation determination unit 64 moves the processing from S63 to the next step S66. Note that internal images PI23 to PI25 and internal images PI33 to PI35 do not include the image portion at the back of the lumen, and an elliptical bubble is displayed, indicating that the lumen to be inserted has been lost.

[0148] In S66, the endoscope operation determination unit 64 determines whether the endoscope is advancing to the insertable pattern position based on the change in the internal image frame. That is, S66 determines whether the insertion unit 12 is advancing toward the back of the lumen by the insertion operation of the insertion unit 12 by the operator. If the insertion unit 12 is advancing toward the back of the lumen, the endoscope operation determination unit 64 creates guide information indicating that insertion is OK and displays it on the display unit 69.

[0149] For example, if there is a change from internal image PI21 to internal image PI22 or from internal image PI31 to internal image PI32, it means that the endoscope has been inserted and the deeper part of the lumen has not been missed. In this case, the endoscope operation determination unit 64 displays an insertion OK guide on the display unit 69 in S66, and then determines in S70 whether the examination is complete or not. If the examination is not complete, the process returns to S51.

[0150] On the other hand, if the endoscope operation determination unit 64 determines in S63 that an insertable pattern position cannot be confirmed, it executes S64 and S65 and then returns to processing S63.

[0151] In the endoscopic data acquisition system 61, the operator of the endoscope 10 may be a skilled physician or an inexperienced physician. Therefore, the endoscope operation determination unit 64 compares the current operation data with the model operation data and, if it determines that the current operation is similar to the operation shown in the model operation data (for example, the operation of a skilled physician), it provides the current operation data to the operation data recording unit 41 of the model operation information generation unit 81 and records it as model operation data. That is, in S64, the endoscope operation determination unit 64 provides the model operation information generation unit 81 with the first and second time-series images from the disappearance of the insertable pattern position to the rediscovery of the insertable pattern position and records them.

[0152] The changes from internal image PI23 to internal image PI26 and from internal image PI33 to internal image PI36 represent the image changes from the disappearance to the rediscovery of such insertable pattern positions. The endoscope operation determination unit 64 provides internal images PI23 to PI26 and external images PE23 to PE26, and internal images PI33 to PI36 and external images PE33 to PE36 to the operation data recording unit 41 of the model operation information generation unit 81 and records them as model operation data (recovery guide information).

[0153] Furthermore, if the endoscope operation determination unit 64 determines that the current operation is not similar to the operation indicated by the model operation data, such as in the case of operation by an inexperienced physician, and if model operation data (recovery guide information) corresponding to the image change exists, it generates guide information based on the model operation data and displays it on the display unit 69 (S65). That is, as shown from external image PE25 to external image PE26 and from external image PE35 to external image PE36, the endoscope operation determination unit 64 provides and displays recovery guide information on the display unit 69 for calmly withdrawing the insertion part 12 and reinserting it. By referring to the guide display in S65, an inexperienced physician can relatively easily rediscover the insertable pattern position.

[0154] If the endoscope operation determination unit 64 determines in S66 that the endoscope has not progressed to the insertable pattern position based on changes in the internal image frames, it proceeds to processing S67 and S68. In S67, if the endoscope operation determination unit 64 determines, by comparing the current operation data with the model operation data, that the current operation is similar to the operation indicated by the model operation data (for example, the operation of a skilled physician), it provides the current operation data to the operation data recording unit 41 of the model operation information generation unit 81 and records it as model operation data. That is, the endoscope operation determination unit 64 provides the first and second time-series images of the insertion unit 12 moving towards the insertable pattern position to the model operation information generation unit 81 and records them.

[0155] Furthermore, if the endoscope operation determination unit 64 determines that the current operation is not similar to the operation indicated by the model operation data, such as when the operation is performed by an inexperienced physician, and if model operation data corresponding to the image change exists, it generates guide information based on the model operation data and displays it on the display unit 69 (S68). By referring to the guide display in S68, the inexperienced physician can relatively easily advance the insertion unit 12 toward the insertable pattern position.

[0156] If the endoscope operation determination unit 64 determines in S70 that the examination is complete, in S71 it synchronizes the timing and graphs the data recorded in the operation data recording unit 41 of the model operation information generation unit 81, and then terminates the process.

[0157] In this embodiment, guide information is generated by comparing previously acquired model operation data with current operation data, enabling even inexperienced physicians to perform smooth insertion operations based on the guide information.

[0158] The present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, some components of all the components shown in the embodiments may be deleted. Moreover, components from different embodiments may be appropriately combined.

[0159] For example, using the data processing device shown in Figure 1, data may be acquired on how far the endoscope has entered the body, with the insertion starting point, such as the anus, as the origin.

[0160] Figure 18 is a graph showing the change in insertion distance, with the vertical axis representing the insertion distance and the horizontal axis representing the elapsed time. In other words, Figure 18 shows how far the endoscope has entered the body from the starting point of insertion, at each point in time during the procedure.

[0161] The insertion amount can be calculated from the values ​​in the insertion / extraction graph in Figure 1 or from the insertion / extraction information detected in Figure 6. Alternatively, an indicator or code that indicates the insertion amount can be attached to the endoscope insertion section, and this can be imaged with an external camera. The code corresponding to the inserted amount can be read at the insertion section, analyzed, and converted into an insertion amount to obtain the insertion amount. A database linking codes and insertion amounts can be prepared and referenced, or numerical values ​​can be written to the codes and read. The second time-series image analysis unit 32 determines the characteristics of the operator's arm from the images taken by the external camera in relation to the passage of operation time. As an application, the shooting position of the external camera can be adjusted to capture the insertion situation, and numbers, symbols, or codes written on the tube of the endoscope insertion section can be read. In this case, the unit monitors each image to determine at what point in time the endoscope was inserted and to what extent. In other words, the second time-series image analysis unit 32 sets the insertion length amount, which is the change in the insertion length of the endoscope insertion section in the insertion direction or withdrawal direction in relation to the passage of operation time, as a numerical value corresponding to the endoscope insertion and withdrawal operations performed by the endoscope operator. This value corresponds to the cumulative change in the vertical axis value of the graph in Figure 1, so it can also be converted from the graph in Figure 1.

[0162] Furthermore, the graph in Figure 18 provides information on how far the endoscope has been inserted from the tip, allowing us to determine which anatomical location (region) within the lumen the endoscope has reached when it has been inserted a specific length into the body. For example, reaching 600 mm indicates that the endoscope is passing through the sigmoid colon, and reaching 400 mm indicates that it has moved from the sigmoid colon to the left colic flexure (splenic flexure). In addition, by examining the waveform of the fine-tuned insertion volume, we can see, for example, that an increase in insertion volume indicates that the endoscope is being inserted, and a decrease in insertion volume indicates that the endoscope is being moved in the withdrawal direction, allowing us to understand the pushing and pulling operations leading up to the currently imaged area. This kind of information can be used as a guide.

[0163] Furthermore, many of the controls and functions described here, primarily those explained in flowcharts, can be configured by programs, and these controls and functions can be realized by a computer reading and executing these programs. These programs can be recorded or stored, in whole or in part, as computer program products on portable media such as flexible disks, CD-ROMs, and non-volatile memory, or on storage media such as hard disks and volatile memory, and can be distributed or provided at the time of product shipment or via portable media or communication lines. Users can easily implement the endoscopic operation digitization system, endoscopic operation data acquisition system, endoscopic operation guide system, data processing device, endoscopic data acquisition system, data processing method, insertion guide method, and insertion guide program of this embodiment by downloading and installing the program on a computer via a communication network, or by installing it on a computer from a recording medium.

Claims

1. An endoscope operation quantification system comprising: a first time-series image analysis unit for determining the time changes of feature parts in time-series images obtained by the imaging unit as a result of an endoscope operator inserting or withdrawing an endoscope having an imaging unit at its tip into the body of a subject; a second time-series image analysis unit for determining the time changes of feature parts in time-series images obtained by an external camera that photographs the endoscope operator; a synchronization unit for synchronizing the time-series images obtained by the imaging unit and the time-series images obtained by the external camera; and a time-series operation information generation unit that generates operation data that quantifies the characteristics of the endoscope operator's operation by analyzing the presence or absence and changes in the amount of operation for each item, divided into the categories of endoscope insertion / withdrawal, twisting of the endoscope insertion unit, and angle of the endoscope tip, according to the time-series changes of each synchronized image.

2. The endoscope operation quantification system according to claim 1, characterized in that the time-series operation information generation unit analyzes, based on the determination of the first time-series image analysis unit, whether or not there is twisting of the endoscope insertion section by the endoscope operator and the amount of change in the angle of the endoscope tip, and based on the determination of the second time-series image analysis unit, whether or not there is an endoscope insertion and withdrawal operation by the endoscope operator and the amount of such operation, and generates the operation data.

3. The endoscopic operation quantification system according to claim 1, further comprising: a recording unit capable of recording the results of determining the type of operation performed by the endoscope operator in accordance with the time-series changes in the insertion amount of the endoscope, or the torsion and angle of the endoscope tip, based on information on the insertion amount of the endoscope, or information on the torsion and angle of the endoscope tip.

4. The endoscopic operation quantification system according to claim 1, characterized in that the second time-series image analysis unit analyzes time-series images obtained by an external camera when the endoscope operator inserts the endoscope into the subject's body, and determines whether or not the endoscope was inserted or withdrawn and the amount of operation based on the horizontal change information of the position of the image portion in the screen in which the arm features of the endoscope operator are detected.

5. The endoscopic operation quantification system according to claim 2, characterized in that the second time-series image analysis unit determines whether an insertion operation is being performed, an extraction operation is being performed, or no operation is being performed, by analyzing the speed of repeated horizontal changes in the position of the image portion in which the arm features of the endoscope operator are detected.

6. The endoscopic operation quantification system according to claim 2, wherein the second time-series image analysis unit sets the value to zero when no horizontal positional change is detected for a predetermined time in relation to the elapsed operation time of the image portion in which the operator's arm features are detected, and when there is a positional change in the insertion direction or withdrawal direction, a numerical value corresponding to the amount of change in each direction is used as the operation data.

7. The endoscopic operation quantification system according to claim 2, wherein the second time-series image analysis unit determines the insertion length amount, which is the change in the insertion length of the endoscope insertion unit in the insertion direction or withdrawal direction with respect to the passage of operation time, as a numerical value corresponding to the endoscope insertion and withdrawal operations performed by the endoscope operator.

8. An endoscope operation data collection system comprising an operation data recording unit that records operation data obtained by the endoscope operation quantification system described in claim 1, wherein the operation data is divided into the categories of endoscope insertion / removal, twisting of the endoscope insertion section, and angle of the endoscope tip, and information is obtained as a result of determining the presence or absence of operation and the change in the amount of operation for each category in a time series, and images obtained by the endoscope imaging unit, in accordance with each timing in the time series, and can be provided to the operation guide generation unit.

9. An endoscope operation guide system comprising: an acquisition unit that synchronizes and acquires a first time-series image from an imaging device provided in the endoscope insertion section and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion section; a time-series operation information generation unit that generates operation data related to the operator's operation based on the first and second time-series images; and an endoscope operation determination unit that uses operation data previously generated by the time-series operation information generation unit as model operation data and operation data currently being generated by the time-series operation information generation unit as current operation data, and generates guide information for guiding the insertion operation based on a comparison between the current operation data and the model operation data.

10. The endoscope operation guide system according to claim 9, further comprising a database for recording standard operation information relating to standard insertion operations of the endoscope insertion section, wherein the endoscope operation determination unit omits the comparison of the current operation data with the model operation data and generates the guide information based on a comparison of the standard operation information with the current operation data.

11. The endoscope operation determination unit comprises an insertion difficulty determination unit that determines the difficulty of insertion of the endoscope insertion part based on the first time-series image, and generates guide information based on a comparison of the current operation data and the model operation data in the insertion difficulty situation, according to claim 9.

12. The endoscope operation guide system according to claim 11, wherein the insertion difficulty determination unit acquires first operation information corresponding to the operator's operation based on the image changes of the first time series image, acquires second operation information corresponding to the operator's operation based on the image changes of the second time series image, and determines the insertion difficulty based on a comparison of the first operation information and the second operation information.

13. The endoscopic operation guide system according to claim 9, wherein the first time-series image is an image of the lumen inside the body, the second time-series image is an image obtained by capturing the movement of the operator's arm, and the time-series operation information generation unit quantifies the changes in the image of the lumen and the changes in the image showing the movement of the operator's arm to obtain the operation data.

14. A data processing device comprising: an acquisition unit that synchronizes and acquires a first time-series image from an imaging device provided in the endoscope insertion section and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion section; a first time-series image analysis unit that acquires first operation information corresponding to the operator's operation based on the image changes of the first time-series image by analyzing the first time-series image; a second time-series image analysis unit that acquires second operation information corresponding to the operator's operation based on the image changes of the second time-series image by analyzing the second time-series image; and a time-series operation information generation unit that generates and records operation data related to the operator's operation based on the first and second information.

15. The data processing device according to claim 14, wherein the first and second operation information includes information relating to at least one operation of the push-pull operation, twisting operation, dial operation and switch operation of the endoscope insertion section.

16. The data processing apparatus according to claim 14, wherein the time-series operation information generation unit records the operation data as a graph.

17. An endoscope data acquisition system comprising: an endoscope equipped with an imaging device in the insertion section; an external camera for imaging the operator operating the endoscope; an acquisition unit for synchronously acquiring a first time-series image from the imaging device and a second time-series image from the external camera; a first time-series image analysis unit for acquiring first operation information corresponding to the operator's operations based on image changes in the first time-series image by analyzing the first time-series image; a second time-series image analysis unit for acquiring second operation information corresponding to the operator's operations based on image changes in the second time-series image by analyzing the second time-series image; a time-series operation information generation unit for generating operation data related to the operator's operations based on the first and second information; and a recording unit for recording the operation data.

18. A data processing method for a data processing device having an acquisition unit, first and second time-series image processing units, and a time-series operation information generation unit, wherein the acquisition unit acquires a first time-series image from an imaging device provided in the endoscope insertion unit and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion unit in a synchronized manner; the first time-series image analysis unit acquires first operation information corresponding to the operator's operation based on the image changes of the first time-series image by analyzing the first time-series image; the second time-series image analysis unit acquires second operation information corresponding to the operator's operation based on the image changes of the second time-series image by analyzing the second time-series image; and the time-series operation information generation unit generates and records operation data relating to the operator's operation based on the first and second information.

19. An insertion guiding method for an endoscope operation guide system comprising an acquisition unit, a time-series operation information generation unit, and an endoscope operation determination unit, wherein the acquisition unit acquires a first time-series image from an imaging device provided in the endoscope insertion unit and a second time-series image from an external camera that images the operator performing the insertion operation of the endoscope insertion unit in synchronization; the time-series operation information generation unit generates operation data relating to the operator's operation based on the first and second time-series images; and the endoscope operation determination unit generates guide information for guiding the insertion operation based on a comparison between the current operation data and the model operation data, using the operation data previously generated by the time-series operation information generation unit as model operation data and the operation data currently being generated by the time-series operation information generation unit as current operation data.

20. An insertion guide program that causes a computer to execute a procedure for synchronizing and acquiring a first time-series image from an imaging device installed in the endoscope insertion section and a second time-series image from an external camera that images the operator performing the insertion operation in the endoscope insertion section; generating operation data related to the operator's operation based on the first and second time-series images; using previously generated operation data as model operation data and currently being generated operation data as current operation data; and generating guide information for guiding the insertion operation based on a comparison between the current operation data and the model operation data.

Images