Monitoring system and evaluation method for endoscope insertion operation

Abstract

The monitoring system comprises: an endoscope shape detection device (3) for detecting the shape of an insertion portion (6a) of an endoscope (6); and a motion detection device for detecting the motion of a hand operating the endoscope (6).

Description

Technical Field

[0001] The present invention relates to a monitoring system and an evaluation method for an insertion operation of an endoscope. Background Art

[0002] Endoscopes are widely used in the medical field, but it is not easy to insert the insertion portion of the endoscope into the subject. For example, the large intestine has a complex shape (travel path). In order to insert the insertion portion of the endoscope into the large intestine without causing pain to the patient, the doctor needs to have high skills in operating the endoscope.

[0003] In addition, by using an endoscope shape acquisition device that displays the insertion shape of the insertion portion of the endoscope in real time, the doctor can confirm the shape of the insertion portion while inserting the insertion portion into the subject. However, the quality of the doctor's insertion operation cannot be judged by the insertion shape alone.

[0004] International Publication No. WO2015 / 070866 proposes an endoscope inspection apparatus that monitors the quality of endoscope inspection by presenting a predetermined score compared with an experienced person for the travel speed, travel direction, and immobility time at a selected point in an insertion portion.

[0005] However, even if the score is displayed in the device according to the above proposal, it is not possible to indicate how to specifically move the hand holding the insertion portion to operate the insertion portion so that the endoscope can be efficiently inserted without applying unreasonable force to the insertion object.

[0006] Therefore, an object of the present invention is to provide a monitoring system and an evaluation method of an endoscope insertion operation that can present an endoscope insertion operation method that only requires a hand holding an insertion portion to move. Summary of the invention

[0007] Means for solving problems

[0008] One aspect of the present invention is a monitoring system including: a shape detection device that detects the shape of an insertion portion of an endoscope; and a motion detection device that detects the motion of a hand that operates the endoscope.

[0009] One embodiment of the present invention is a method for evaluating the insertion operation of an endoscope, wherein a shape of an insertion portion of an endoscope inserted into a subject is detected by a shape detection device, and at least one of the position and posture of a hand of an operator operating the endoscope holding the insertion portion in a three-dimensional space is detected by a motion detection device. A processor calculates at least one parameter of the hand's movement amount, movement speed, rotation angle, and rotation speed based on information related to at least one of the position and posture of the hand in the three-dimensional space output from the motion detection device, and outputs the at least one parameter in temporal association with information related to the insertion shape of the insertion portion from the shape detection device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 1 is a block diagram showing the structure of an endoscope insertion training system according to an embodiment of the present invention.

[0011] Figure 2 It is a diagram showing an example of an insertion shape image displayed based on the insertion shape information from the shape calculation device according to the embodiment of the present invention.

[0012] Figure 3 This is a stereoscopic view of a marking device worn on the wrist of the user's right hand according to an embodiment of the present invention.

[0013] Figure 4 This is a diagram for illustrating the configuration of four markers when the marker device is worn on the wrist according to an embodiment of the present invention.

[0014] Figure 5 This is a diagram for explaining an embodiment of the present invention, in which a user stands on the anus side of a model and inserts an insertion portion into the model.

[0015] Figure 6 This is a diagram of an insertion portion captured by a second imaging device for explaining the embodiment of the present invention.

[0016] Figure 7 It is a diagram for explaining a plurality of marks provided on the outer peripheral surface of the insertion portion according to the embodiment of the present invention.

[0017] Figure 8 This is a block diagram showing an endoscope insertion operation evaluation device and a plurality of imaging devices according to an embodiment of the present invention.

[0018] Fig. 9 This is a block diagram showing the configuration of an endoscope insertion operation evaluation device according to an embodiment of the present invention.

[0019] Fig.10This is a flowchart showing an example of the processing flow of the insertion method recording program of the endoscope insertion operation evaluation device according to the embodiment of the present invention.

[0020] Fig.11 It is a diagram showing an example of the data structure of evaluation data according to the embodiment of the present invention.

[0021] Fig.12 This is a flowchart showing an example of the processing flow of the insertion method display program of the endoscope insertion operation evaluation device according to the embodiment of the present invention.

[0022] Fig.13 FIG. 1 is a diagram showing an example of an image showing the motion ranges of the right hands of two designated doctors according to the embodiment of the present invention.

[0023] Fig.14 This is a diagram showing another example of an image showing the motion ranges of the right hands of two designated doctors according to the embodiment of the present invention.

[0024] Fig.15 This is a diagram showing an example of an image showing a graph showing changes in the amount of advancement and retreat and the amount of twisting of the insertion portion according to the embodiment of the present invention.

[0025] Fig.16 It is a diagram showing an example of an image showing a hand movement and a change in an insertion shape according to an embodiment of the present invention.

[0026] Fig.17 It is a diagram for explaining the change of the insertion shape and the change of the animated hand displayed in two windows according to the embodiment of the present invention.

[0027] Fig.18 This is a diagram showing an example of an image in which an insertion shape is superimposed on an image of a model and displayed in a window according to the embodiment of the present invention.

[0028] Fig.19 It is a diagram for explaining the arrangement of a force gauge as a force sensor according to an embodiment of the present invention.

[0029] Fig. 20 This is a diagram showing an example of an image showing a graph showing changes in the force applied to a model according to the embodiment of the present invention. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0031] Figure 1: is a structural diagram showing the structure of the monitoring system of the present embodiment. The monitoring system is mainly composed of an endoscope insertion training system (hereinafter, referred to as the training system) 1, and the training system 1 includes an endoscope device 2, an endoscope shape acquisition device 3, and an endoscope insertion operation evaluation device 4. The endoscope device 2, the endoscope shape acquisition device 3, and the endoscope insertion operation evaluation device 4 are arranged near a model of a part of the human body (hereinafter referred to as the model) 102 fixed on a workbench 101. Here, the model 102 is a colon model. The model 102 has a main body 102a that imitates the shape of the human body, an intestinal tube part 102b that imitates the large intestine in the main body 102a, and an anus part 102c that imitates the anus. The interior of the intestinal tube part 102b is connected to the anus part 102c.

[0032] The endoscope device 2 includes an endoscope 6, a light source device 7, an image processing device 8, and a display device 9. The light source device 7, the image processing device 8, and the display device 9 are mounted on the cart 101a or fixed to the cart 101a.

[0033] The endoscope 6 includes an insertion portion 6a, an operating portion 6b, and a universal cable 6c. The insertion portion 6a has a front end portion 11, a bendable bending portion 12, and a flexible tube portion 13 in sequence from the front end toward the base end. The front end portion 11 is the front end portion of the insertion portion 6a, that is, the front end portion of the endoscope 6, and is a front end hard portion (front end structural portion). An objective lens optical system and an imaging element are built into the front end portion 11. Therefore, the endoscope 6 has an insertion portion 6a inserted into a subject (here, the model 102).

[0034] The bending portion 12 includes a plurality of bending pieces and can be bent in the up, down, left, and right directions according to the operation of the two bending operation knobs 14 of the operation portion 6b. The flexible tube portion 13 is bent according to the external force. The two bending operation knobs 14 are inserted through the insertion portion 6a to pull and relax the operation wire to bend the bending portion 12. The flexible tube portion 13 is a tubular member that is flexible and bends according to the external force.

[0035] In the insertion portion 6a, a plurality of magnetic sensors 6d for the endoscope shape acquisition device 3 are arranged at predetermined intervals. In addition, in the present embodiment, the plurality of magnetic sensors are arranged at equal intervals along the central axis of the insertion portion 6a, but they may be arranged at different intervals depending on the part where the insertion method is to be evaluated. For example, if the part to be evaluated is the sigmoid colon of the large intestine, the number of the plurality of magnetic sensors may be greater at the front end side portion of the insertion portion 6a than at the base end side portion, and the intervals between adjacent magnetic sensors may be narrowed.

[0036] A connector is provided at the front end of the universal cable 6c, and the endoscope 6 is connected to the light source device 7 and the image processing device 8 in a detachable manner through the connector. Here, the endoscope 6 is an endoscope that can be inserted into the large intestine. In addition, although not shown in the figure, a light guide is inserted through the universal cable 6c, and the endoscope 6 is configured so that the illumination light from the light source device 7 is emitted from the front end of the insertion portion 6a through the light guide. In addition, a plurality of signal lines for a plurality of magnetic sensors 6d are also inserted through the universal cable 6c.

[0037] A doctor (hereinafter referred to as a user) who wants to evaluate the insertion operation of an endoscope uses the training system. The user moves the right hand RH holding the insertion portion 6a in various directions. While holding the insertion portion 6a with the right hand RH, the user presses the insertion portion 6a into the large intestine of the model 102 along the axial direction of the insertion portion 6a, or withdraws it from the large intestine. Furthermore, while holding the insertion portion 6a with the right hand RH, the user rotates the insertion portion 6a around the axis of the insertion portion 6a.

[0038] The image processing device 8 receives an imaging signal from an imaging element disposed at the distal end portion 11 of the insertion portion 6a, performs predetermined image processing on the imaging signal, and generates an endoscopic image. The image processing device 8 is connected to the display device 9 via a signal line (not shown). The image signal of the endoscopic image is output to the display device 9, and the endoscopic image is displayed on the display screen of the display device 9.

[0039] The endoscope shape acquisition device 3 includes a shape calculation device 21 and a magnetic field generating device 22. The shape calculation device 21 is placed on the cart 101a, and the magnetic field generating device 22 is placed on the workbench 101. The shape calculation device 21 is connected to the image processing device 8 via a signal line not shown. The shape calculation device 21 receives signals from a plurality of magnetic sensors 6d built into the endoscope 6 via the image processing device 8. The shape calculation device 21 is connected to the magnetic field generating device 22 via a signal line 23.

[0040] The magnetic field generating device 22 generates a predetermined magnetic field according to the driving signal from the shape computing device 21, and each magnetic sensor 6d detects the magnetic field generated by the magnetic field generating device 22. The detection signals of the magnetic field from the plurality of magnetic sensors 6d are provided to the shape computing device 21. In addition, a plurality of magnetic field generating elements may be provided in the insertion portion 6a instead of the plurality of magnetic sensors 6d, and a magnetic sensor may be provided outside the model 102 instead of the magnetic field generating device 22 to detect the position and posture of each magnetic field generating element.

[0041] The shape calculation device 21 calculates the shape of the insertion portion 6a in real time based on the signals from each magnetic sensor 6d. The shape calculation device 21 is connected to the endoscope insertion operation evaluation processor 33 described later via a signal line 21a. The information on the insertion shape of the insertion portion 6a calculated by the shape calculation device 21 is output to the endoscope insertion operation evaluation processor 33. The endoscope insertion operation evaluation processor 33 can generate an insertion shape image observed from a designated viewpoint direction based on the received information on the insertion shape, and display it on the display device 34. Therefore, the shape calculation device 21 constitutes a shape detection device that detects the insertion shape of the insertion portion 6a.

[0042] Figure 2 1 is a diagram showing an example of an insertion shape image displayed based on the insertion shape information from the shape calculation device 21 . Figure 2 The insertion shape IPF of the insertion portion 6a is shown displayed in one window W on the display screen 34a of the display device 34, but the insertion shape IPF can also be displayed superimposed on other images by a software program in the endoscope insertion operation evaluation processor 33.

[0043] Return to Figure 1 The endoscope insertion operation evaluation device 4 includes a first imaging device 31, a second imaging device 32, an endoscope insertion operation evaluation processor 33, a display device 34, an input device 35, and a bird's-eye view camera 36. The input device 35 includes a keyboard 35a and a mouse 35b.

[0044] The first imaging device 31 is fixed to a frame 101c fixed to the table 101b. The second imaging device 32 is fixed to the table 101. The endoscope insertion operation evaluation processor 33 is placed on the table 101b.

[0045] The first imaging device 31 includes a plurality of cameras 31a. The plurality of cameras 31a captures the user's operation of operating the insertion portion 6a of the endoscope 6 to insert it into the model 102 from the upper right of the user. Therefore, the plurality of cameras 31a can capture the user's right hand RH.

[0046] In this embodiment, the movement of the user's right hand RH is detected by infrared motion capture. Specifically, the marker device 51 for motion capture is worn on the wrist of the user's right hand, and each camera 31a as an infrared camera shoots the marker device 51. The movement of the user's right hand is detected based on the movement of each marker 54 in the image captured by each camera 31a. Here, the position and posture of the right hand are detected. That is, the three-dimensional position and posture of the marker device 51 are acquired over time to detect the behavior of the user's right hand RH.

[0047] Figure 3It is a stereoscopic view of a marking device 51 worn on the wrist of the right hand RH of the user. The marking device 51 has a fixing belt 52, a main body 53, a plurality of spherical markers 54, and a plurality of shaft components 55. The fixing belt 52 has elasticity for being worn and fixed on the wrist. The fixing belt 52 is in the shape of a belt, and has a fixing unit not shown, such as a surface fastener, and is worn on the wrist in a manner that can be loaded and unloaded. The main body 53 is fixed to the surface of the fixing belt 52 by an adhesive, a screw, or the like. A plurality of (here, 4) shaft components 55 are fixed to the main body 53 in a manner protruding from the main body 53. That is, the marking device 51 has a plurality of markers 54, and is configured to be mounted on the wrist of the hand of the user who is the operator of the operating insertion portion 6a.

[0048] Figure 4 This is a diagram for explaining the configuration of the four markers 54 when the marker device 51 is worn on the wrist. Figure 5 1 is a diagram for explaining a situation in which the user DU stands on the anus 102c side of the model 102 and inserts the insertion unit 6a into the model 102. Figure 5 As shown, the user DU wears the marker device 51 on the wrist of the right hand RH, and holds the insertion portion 6a with the right hand RH to push, pull, and twist the insertion portion 6a.

[0049] When the user holds the insertion portion 6a with the right hand RH, the right hand RH rotates around the longitudinal axis CO of the right arm RA within a range of approximately +90 degrees to approximately -90 degrees. The first camera 31 captures the user's right hand RH from approximately above. Therefore, the marking device 51 is worn on the wrist of the right hand RH in such a manner that the main body 53 is located at a position extending from the thumb of the right hand RH through the first metacarpal bone MB toward the radius RB when viewed from the camera direction of the first camera 31.

[0050] Each marker 54 is attached to the front end of the shaft member 55 extending from the main body 53. When the marker device 51 is attached to the wrist of the right hand RH, the four markers 54 are located in a direction perpendicular to the plane PL formed by the imaging direction of the first imaging device 31 and the length axis CO of the right arm RA of the right hand RH. When the imaging device 31 captures the wrist of the right hand RH without rotating it, the four markers 54 are arranged so that a pair of markers 54 are located on both sides of the length axis CO. Here, when the wrist is captured from the thumb side, the four markers 54 are arranged so that two of the four markers 54 are located on one side with respect to the length axis CO of the right arm RA and the other two markers 54 are located on the other side with respect to the length axis CO of the right arm RA.

[0051] Furthermore, when the right hand RH rotates around the longitudinal axis CO of the right arm RA, if the four marks 54 immediately become the shadow of the right arm RA, the marks 54 cannot be photographed by the camera device 31. Figure 4As shown, the distance Lm between the center points of the two marks 54 arranged across the length axis CO is greater than the maximum outer diameter Wm of the right arm RA ( Figure 3 ) is large. The four marks 54 are arranged in such a manner that at least three of the four marks 54 can be photographed by the camera device 31 regardless of how the arm rotates. Here, at least two of the plurality of marks 54 are arranged at a distance of 80 mm or more, so that the state of the plurality of marks 54 protruding from the wrist can be detected by any of the plurality of cameras 31a. As an example, at least two of the combinations of two marks 54 are arranged at a distance of 80 mm or more and 100 mm or less.

[0052] The first camera device 31 includes a plurality of cameras 31a for photographing a plurality of markers 54 from a plurality of directions in a three-dimensional space. Here, the first camera device 31 is located above the position of the user's right hand RH and to the right or in front of the user. The six cameras 31a of the camera device 31 photograph the user's right hand RH from different directions. The marker device 51 moves according to the movement of the right hand RH, but the camera device 31 is set to photograph the range of movement during the normal insertion operation.

[0053] As described later, by such arrangement of the four marks 54 , the user's insertion operation of the insertion portion 6 a , specifically, various operations including the twisting operation of the right hand RH, are detected by being photographed by the imaging device 31 .

[0054] The second camera 32 is a camera that takes a picture of the vicinity of the anus 102c from the side of the model 102. Characters and graphics for detecting the insertion amount are provided on the surface of the insertion portion 6a inserted into the anus by printing or the like. The insertion portion 6a is operated not only by the right hand RH but also by the left hand by rotating the operation portion 6b. The second camera 32 is provided to detect the movement of the insertion portion 6a in consideration of the situation that the insertion portion 6a is operated not only by the right hand RH but also by the left hand. Therefore, the second camera 32 takes a picture of the insertion portion 6a inserted into the model 102 as the subject.

[0055] Figure 6 This is a diagram for explaining the insertion portion 6 a captured by the second imaging device 32 . Figure 7 It is a diagram for explaining a plurality of marks provided on the outer peripheral surface of the insertion portion 6a.

[0056] like Figure 6 As shown, the second imaging device 32 is fixed to the workbench 101 in such a manner that it can capture the insertion portion 6a inserted into the anus 102c of the model 102. Figure 6As indicated by the one-dot chain line, the range captured by the imaging device 32 includes the insertion portion 6a inserted into the anus 102c.

[0057] like Figure 6 and Figure 7 As shown, a plurality of marks are provided on the outer surface of the insertion portion 6a by printing or the like. A plurality of dotted lines 61 are provided on the outer surface of the insertion portion 6a at predetermined intervals and are provided in a ring shape along the circumference of the insertion portion 6a. In addition, as one mark, a first character string 62 is provided in a ring shape along the circumference of the insertion portion 6a. As another mark, a second character string 63 is also provided in a ring shape along the circumference of the insertion portion 6a.

[0058] The characters of the first character string 62 are different from the characters of the second character string 63. Here, the characters of the first character string 62 are numbers or lowercase letters, and the characters of the second character string 63 are uppercase letters. Here, the characters of the first character string 62 and the second character string 63 are printed in a manner that can be correctly read when the insertion portion 6a is viewed with the distal end direction DD of the insertion portion 6a as the top and the proximal end direction PD as the bottom. The first character string 62 and the second character string 63 are provided between two dotted lines arranged in the circumferential direction of the insertion portion 6a.

[0059] Between the first character string 62 and the second character string 63, there are also provided a ring-shaped line 64 drawn in the circumferential direction of the insertion portion 6a and a spiral line 65 provided on the outer peripheral surface of the insertion portion 6a as other markings. When the insertion portion 6a is observed from a direction perpendicular to the longitudinal axis, the spiral line 65 is inclined at a predetermined angle relative to the longitudinal axis of the insertion portion 6a and is drawn on the outer peripheral surface of the insertion portion 6a at 360 degrees around the longitudinal axis. That is, the predetermined marking 66 provided on the outer surface of the insertion portion 6a includes: the line 64 as the first band, which surrounds the outer periphery of the insertion portion 6a in a ring shape in a direction perpendicular to the longitudinal axis of the insertion portion 6a; and the line 65 as the second band, which is provided at an angle relative to the line 64 and surrounds the outer periphery of the insertion portion 6a in a spiral shape.

[0060] Regarding the line 64 captured by the imaging device 32, even if the insertion portion 6a rotates around the longitudinal axis, the position of the line 64 does not change in the image obtained by the imaging device 32. On the other hand, regarding the line 65 captured by the imaging device 32, when the insertion portion 6a rotates around the longitudinal axis, the position of the line 65 changes in the image obtained by the imaging device 32.

[0061] And, if Figure 7 As shown, the plurality of spiral lines 65 are drawn in the same phase. In other words, the positions of the starting points and the ending points of the spiral lines 65 in the plurality of predetermined marks 66 are respectively located on an imaginary line of the outer peripheral surface of the insertion portion 6a parallel to the longitudinal axis.

[0062] The first and second character strings 62, 63 and lines 64, 65 are included in one specified mark 66 and are arranged at a specified position between two dotted lines 61. A plurality of specified marks 66 are depicted on the insertion portion 6a at a specified distance from each other. The specified distance is, for example, 10 cm. Therefore, the distance between two adjacent lines 64 is also the same as the specified distance. In the range captured by the camera device 32, the camera device 32 must be set in a manner that includes one specified mark 66.

[0063] The first and second character strings 62 and 63 are recognized by character recognition processing in the image obtained by the camera 32. The insertion amount of the insertion portion 6a into the large intestine is calculated based on the recognized characters and the positions of the characters. In addition, here, the two character strings 62 and 63 are used so that even if the characters of one are blocked by the hand, the characters of the other can be read.

[0064] As described later, in the image obtained by the camera device 32, the twist amount, that is, the twist angle of the insertion portion 6a is calculated based on the distance between the two lines 64 and 65 in each predetermined mark 66. Specifically, the twist amount is calculated based on the ratio of the length L1 of the line 64 in the image obtained by the camera device 32 and the length L2 between the line 64 and the line 65. Figure 6 In the example, the length L2 is the distance between the end of the wire 64 of the insertion portion 6a and the end of the wire 65 (the longer distance in this example). Figure 6 In the example, the distance between the end of the line 64 and the end of the line 65 is two, and the longer distance L2 is used here, but the shorter distance L3 may also be used. The reason for calculating the twist amount based on the ratio of the length L1 of the line 64 and the length L2 between the line 64 and the line 65 is that the twist amount can be accurately calculated even if the distance between the insertion portion 6a and the imaging device 32 changes.

[0065] The bird's-eye view camera 36 is disposed above the model 102 and is a third imaging device that captures an image of the intestinal tract 102b of the model 102. An imaging signal of the bird's-eye view camera 36 is output to the endoscope insertion operation evaluation processor 33.

[0066] Figure 8 is a block diagram showing the endoscope insertion operation evaluation processor 33 and a plurality of imaging devices. Fig. 9 2 is a block diagram showing the structure of the endoscope insertion operation evaluation processor 33 .

[0067] The endoscope insertion operation evaluation processor 33 is connected to the shape calculation device 21 of the first imaging device 31, the second imaging device 32, and the bird's-eye view camera 36, ​​which is a third imaging device for photographing the model 102. The endoscope insertion operation evaluation processor 33 performs calculations described later and generates display data on the display device 34.

[0068] like Fig. 9 As shown, the endoscope insertion operation evaluation processor 33 is a processor having a central processing unit (hereinafter referred to as CPU) 71, ROM 72, RAM 73, hard disk device (hereinafter referred to as HDD) 74, and various interface circuits (hereinafter referred to as I / F) 75, 76, 77, 78, 79, 80. The CPU 71, ROM 72, RAM 73, HDD 74, and various I / Fs 75, 76, 77, 78, 79, 80 are connected to each other via a bus 81.

[0069] The I / F 75 is an interface between each camera 31a of the first imaging device 31 and the bus 81. The I / F 76 is an interface between the second imaging device 32 and the bus 81. The I / F 77 is an interface between the bird's-eye view camera 36 and the bus 81. The I / F 78 is an interface between the shape calculation device 21 and the bus 81. The I / F 79 is an interface between the display device 34 and the bus 81. The I / F 80 is an interface between the input device 35 and the bus 81.

[0070] The HDD 74 has a storage area 74a for storing an insertion operation evaluation program EP, a storage area 74b for storing evaluation data ED, a storage area 74c for storing a motion capture program MC, and a storage area 74d for storing an animation generation program AP. The insertion operation evaluation program EP includes an insertion operation recording program EP1 and an insertion operation display program EP2.

[0071] The motion capture program MC is software that detects the positions of the plurality of markers 54 in the plurality of images obtained by the plurality of cameras 31a, and calculates the position and posture in the three-dimensional space of the right hand RH to which the marker device 51 is attached. Therefore, the CPU 71 that executes the motion capture program MC constitutes a motion detection device that detects the motion of the hand of the user who operates the endoscope 6, which holds the insertion portion 6a, including the position and posture in the three-dimensional space. In particular, the motion detection device is a motion capture device that has a first camera 31 that captures the marker device 51 attached to the user's hand, and detects the motion of the hand based on the motion of the marker device 51 captured by the first camera 31.

[0072] The animation generation program AP is software that generates an animation image of the movement of the right hand RH based on the information on the position and posture of the right hand RH calculated by the motion capture program MC. The processing of the insertion operation evaluation program EP and the data structure of the evaluation data ED will be described later.

[0073] The CPU 71 can acquire imaging signals from each camera 31a via the I / F 75. Similarly, the CPU 71 can acquire imaging signals from the second imaging device 32 and the bird's-eye view camera 36 via the I / Fs 76 and 77. The CPU 71 can also output generated display data to the display device 34 via the I / F 79.

[0074] Then, the CPU 71 generates data of the user's hand movement and the like and records evaluation data ED by reading and executing the insert operation evaluation program EP from the HDD 74. The CPU 71 can generate display data for evaluation using the evaluation data ED.

[0075] (effect)

[0076] Next, the operation of the endoscope insertion operation evaluation processor 33 will be described.

[0077] The calculation, recording and output of the motion information of the right hand RH, the shape information of the insertion shape IPF, the motion information of the insertion portion 6a and the image information in the endoscope insertion operation evaluation processor 33 are performed by the insertion operation evaluation program EP, the motion capture program MC and the animation generation program AP. As described above, these programs are recorded in the ROM 72 or the HDD 74, read out by the CPU 71 and expanded and executed in the RAM 73 to realize the detection and other functions.

[0078] (1) Records

[0079] Fig.10 2 is a flowchart showing an example of the flow of processing of the insertion operation recording program EP1 of the endoscope insertion operation evaluation processor 33. The CPU 71 reads the insertion operation recording program EP1 from the HDD 74, expands it in the RAM 73, and executes it.

[0080] The CPU 71 obtains the imaging signal from each camera 31a of the first imaging device 31, and calculates the position, posture and twisting amount of the right hand RH in the three-dimensional space (step (hereinafter referred to as S) 1). Specifically, the CPU 71 calculates the position (x, y, z) and posture (vx, vy, vz) of the right hand RH as a rigid body in the three-dimensional space based on the position of each marker 54 of the marking device 51 in the image through the motion capture program MC. vx, vy, vz respectively represent the directions of the reference vectors of the x, y, z axes of the right hand RH as a rigid body relative to the x, y, z axes of the three-dimensional space. The CPU 71 also calculates the twisting amount (r) of the right hand RH based on the calculated position and posture. The twisting amount (r) is information indicating the rotation angle of the right arm RA around the length axis CO. In this way, the CPU 71 generates information on the position, posture and twisting amount of the right hand RH in the three-dimensional space.

[0081] The CPU 71 calculates the movement amount (AM), movement speed (SM), rotation angle (AR) and rotation speed (SR) of the right hand RH based on the information of the position, posture and twisting amount of the right hand RH calculated in S1 (S2). Specifically, the CPU 71 calculates the movement amount (AM) of the right hand RH in the insertion direction of the insertion part 6a based on the information of the position and posture of the right hand RH calculated in S1. The movement amount (AM) is the amount of push and pull of the right hand RH in the insertion direction of the insertion part 6a. The CPU 71 calculates the movement speed (SM) of the right hand RH in the insertion direction of the insertion part 6a based on the information of the position and posture of the right hand RH calculated in S1. The movement speed (SM) is the speed of push and pull of the insertion part 6a of the right hand RH. The CPU 71 calculates the rotation angle (AR) of the right hand RH based on the information of the twisting amount (r) of the right hand RH calculated in S1. The rotation angle (AR) is the angle of the right hand RH as a rigid body around the length axis of the insertion part 6a. The CPU 71 calculates the rotation speed (SR) of the right hand RH based on the information of the twisting amount (r) of the right hand RH calculated in S1. The rotation speed (SR) is the angular speed at which the right hand RH, which is a rigid body, rotates around the longitudinal axis of the insertion portion 6a.

[0082] The CPU 71 obtains the imaging signal from the second imaging device 32, and calculates the information of the insertion amount (L) and the rotation amount (R) of the insertion portion 6a (S3). Specifically, the CPU 71 generates an image based on the imaging signal of the imaging device 32. Then, the CPU 71 calculates the insertion amount (L) and the rotation amount (R) of the insertion portion 6a based on the position and movement of the character or line based on the generated imaging image.

[0083] Therefore, as described later, the CPU 71 calculates the specified parameters based on the specified markings set on the outer surface of the insertion portion 6a captured by the second camera device 32 based on the camera image of the second camera device 32, and outputs them in temporal association with information about the insertion shape of the insertion portion 6a.

[0084] The CPU 71 acquires the insertion shape information (SH) from the endoscope shape acquisition device 3 (S4). Furthermore, the CPU 71 acquires the imaging signal from the overhead camera 36 and generates an image (S5). The CPU 71 associates the information acquired from S1 to S5 with the time information at the same time and outputs it as evaluation data ED of the HDD 74 (S6). The evaluation data ED is recorded in the HDD 74.

[0085] The evaluation data is output by repeating the processing of S1 to S6 in a predetermined cycle. Each information obtained in S1 to S5 is associated with the time information. That is, the endoscope insertion operation evaluation processor 33 is configured to calculate the parameters of the hand movement amount, movement speed, rotation angle, and rotation speed based on the information related to the position and posture of the hand in the three-dimensional space, and output the parameters in a time-related manner with the information on the insertion shape.

[0086] Fig.11 : is a diagram showing an example of the data structure of the evaluation data ED. The evaluation data ED is table data TBL including information such as position, posture, and twist amount as information related to the movement of the right hand RH. As described above, the information of the position (x, y, z), posture (vx, vy, vz), and twist amount (r) of the right hand RH is calculated by the motion capture program MC.

[0087] Furthermore, the evaluation data ED includes information on the insertion amount (L) and the rotation amount (R) as information related to the movement of the insertion portion 6a. The insertion amount (L) is information calculated by character recognition of the first character string 62 and the second character string 63 on the outer surface of the insertion portion 6a in the image of the camera 32. The rotation amount (R) is information calculated based on the distance between the two lines 64 and 65 on the outer surface of the insertion portion 6a in the image of the camera 32.

[0088] The evaluation data ED includes shape information (SH) of the insertion portion 6 a . The shape information (SH) is information on the insertion shape from the shape calculation device 21 .

[0089] Furthermore, the evaluation data ED includes image information (I) of the bird's-eye view camera 36 . The image information (I) is information of an image generated based on an imaging signal from the bird's-eye view camera 36 .

[0090] In addition, here, the evaluation data ED is Fig.11 Although it is one table data TBL as shown, it may be composed of a plurality of table data.

[0091] In addition, the evaluation data ED may not include Fig.11 All the information shown only needs to include at least one piece of information related to the movement of the hand. (2) Display

[0092] Fig.121 is a flowchart showing an example of the process flow of the insertion operation display program EP2 of the endoscope insertion operation evaluation processor 33. The CPU 71 reads the insertion operation display program EP2 from the HDD 74, expands it in the RAM 73, and executes it. The endoscope insertion operation evaluation processor 33 can generate a plurality of display screens for evaluating the insertion operation. The user can operate the input device 35 to provide a desired display command to the CPU 71. The CPU 71 performs a process corresponding to the received display command.

[0093] The CPU 71 determines whether a display command is issued by the user from the input device 35 (S11). When a display command is not issued (S11: No), no processing is performed. When a display command is issued (S11: Yes), the CPU 71 generates display data corresponding to the display command (S12). Then, the CPU 71 outputs the generated display data to the display device 34 (S13).

[0094] Next, an example of the generated display data is shown.

[0095] 1) Range of hand movements

[0096] For example, the user may wish to compare the range of motion of the right hand when inserting the insertion portion 6a into the large intestine with other users. In this case, when the user inputs a predetermined command, Fig.13 Such an image is displayed on the display device 34 .

[0097] Fig.13 The figure shows an example of an image showing the movement range of the right hand RH of the user and the designated other user. When the user wants to view a graph showing the change in the position of the right hand RH of the two users (doctors) to be compared, the evaluation data ED of the two doctors is specified, and when a specified command is input, the image is displayed. Fig.13 Window DP1 in the display screen 34a as shown. Fig.13 This is a display example when a user who is a doctor in training (resident doctor) and an expert are selected as two doctors.

[0098] The window DP1 displays a graph showing the change in the position of the right hand RH of the user when the right hand RH of the user is viewed from above. The solid line represents the trajectory of the right hand RH of the user who is a training doctor, and the dotted line represents the trajectory of the right hand RH of the expert. Therefore, the trajectory of the position of the right hand RH from the start of insertion to the withdrawal in the insertion operation of the two doctors is represented by the solid line and the dotted line. It can be seen that the movement of the right hand RH of the training doctor has more useless movements than the movement of the right hand RH of the expert.

[0099] Fig.14 is shown with Fig.13Similarly, another example of the image showing the motion range of the right hand RH of two designated doctors is shown. Fig.14 The screen is based on Fig.13 Different commands to display.

[0100] Fig.14 The window DP2 displays a graph showing the change in the position of the right hand RH when the model 102 is viewed from the anus 102c side. Fig.14 As can be seen from FIG. 1 , there are more useless movements in the movement of the right hand RH of the trainee indicated by the solid line than in the movement of the right hand RH of the expert indicated by the dotted line.

[0101] Fig.13 and Fig.14 The graphs are based on Fig.11 The table data TBL is drawn using information on the position (x, y, z) of the hand movement.

[0102] in addition, Fig.13 and Fig.14 The graph may be displayed as it is gradually drawn over time. For example, when the graph is drawn starting from time t0 and the line of the graph is drawn longer over time, the movement of the user's hand over time is easy to understand. That is, the endoscope insertion operation evaluation processor 33 may output display data that draws the position of the hand in a graph over time based on the information on the position of the hand.

[0103] 2) The amount of advance and retreat and twist of the insertion part

[0104] For example, the user may want to observe the change in the amount of advance and retreat (i.e., push and pull) of the insertion portion 6a and the change in the amount of twisting of the insertion portion 6a when the insertion portion 6a is inserted into the large intestine. Fig.15 Windows DP3 and DP4 are displayed on the display screen 34a as shown. Fig.15 This is a diagram showing an example of an image showing a graph showing changes in the amount of advancement and retreat and the amount of twisting of the insertion portion.

[0105] The window DP3 displays a graph showing the change in the amount of advance and retreat (i.e., push and pull) of the insertion portion 6a. The window DP4 displays a graph showing an example of an image showing the change in the amount of twisting of the insertion portion 6a. When the user inputs a specified command, the display Fig.15 That is, the endoscope insertion operation evaluation processor 33 outputs first display data and second display data, wherein the first display data displays information related to a first parameter among the hand movement amount, movement speed, rotation angle, and rotation speed within a first time, and the second display data displays information related to a second parameter among the hand movement amount, movement speed, rotation angle, and rotation speed within a second time.

[0106] exist Fig.15 , the time axis of the horizontal axis of the curve graph in the window DP3 and the curve graph in the window DP4 is consistent, that is, synchronized. The time taken by the front end portion 11 of the insertion portion 6a to pass through the sigmoid colon is approximately 20 seconds to 70 seconds. The time taken by the front end portion 11 of the insertion portion 6a to pass through the splenic flexure is approximately 70 seconds to 90 seconds. The time taken by the front end portion 11 of the insertion portion 6a to pass through the transverse colon is approximately 90 seconds to 100 seconds. Therefore, the user can observe these curve graphs to know the push-pull amount of the insertion portion 6a and the twist amount of the insertion portion 6a.

[0107] For example, in Fig.15 In the case of , it is shown that the insertion portion 6a is pulled and twisted between 60 seconds and 80 seconds. Therefore, when the user is a training doctor (resident), as long as the pushing and pulling amount of the insertion portion 6a and the twisting amount of the insertion portion 6a of the expert are displayed, they can compare with their own and improve their own insertion operation.

[0108] Fig.15 The graphs are based on Fig.11 The information of the insertion amount (L) and the rotation amount (R) of the insertion portion 6a in the table data TBL is drawn.

[0109] 3) Hand movement and insertion shape

[0110] For example, the user may want to observe how the insertion shape SH of the insertion portion 6a changes according to the movement of the right hand RH. In this case, when the user inputs a prescribed command, the display Fig.16 The windows DP5 and DP6 in the display screen 34a are shown. Fig.16 This is a diagram showing an example of an image showing the movement of a hand and the change in the insertion shape.

[0111] The window DP5 displays the insertion shape IPF of the insertion portion 6a. The window DP6 displays the movement of the right hand RH in an animation. When the user inputs a specified command, the display Fig.16 of the image.

[0112] The insertion shape of the insertion portion 6a is displayed in the window DP5, and the animation image of the right hand is displayed in the window DP6. The animation generation program AP generates an animation image of the movement of the right hand RH based on the information of the position (x, y, z) and posture (vx, vy, vz) of the right hand RH. That is, the endoscope insertion operation evaluation processor 33 outputs display data that synchronizes the insertion shape with the animation image of the hand generated based on the information of the position and posture of the hand.

[0113] Fig.17 It is a figure used for demonstrating the change of the insertion shape displayed in the window DP5 and the change of the animated hand displayed in the window DP6. Fig.17 The changes of the images of the two windows DP5 and DP6 with the passage of time are shown. The windows DP5t and DP6t show the images at the same time t, the windows DP5(t+m) and DP6(t+m) show the images at the same time (t+m) after the time t, and the windows DP5(t+n) and DP6(t+n) show the images at the same time (t+n) after the time (t+m).

[0114] like Fig.17 As shown, the insertion shape IPF of the insertion portion 6a is displayed synchronously with the right hand RHa of the animation. Fig.16 Therefore, the user can know how the insertion shape IPF of the insertion portion 6a changes according to the movement of the right hand RH.

[0115] Unlike rigid instruments, flexible endoscopes such as colon endoscopes do not move one-to-one with the doctor's operation in that the insertion portion 6a of the endoscope 6 does not move in a one-to-one manner with respect to the doctor's operation. For example, when the doctor pushes the endoscope 6 into the subject's body, the endoscope 6 sometimes bends in the intestinal tract, and the front end portion 11 of the endoscope 6 does not move inward relative to the intestinal tract. In this case, generally speaking, most doctors in training (residents) cannot recognize how the insertion portion 6a inserted into the body moves in correspondence with their own endoscope operation. Therefore, it is very important for doctors in training (residents) to know how the insertion shape IPF of the insertion portion 6a changes in correspondence with the movement of the right hand RH in order to master the insertion operation.

[0116] Fig.16 The Windows DP5 image is based on Fig.11 The image of window DP6 is drawn based on the shape information in the table data TBL. Fig.11 The information of the position, posture and twist amount of the hand in the table data TBL is generated and drawn by the animation generation program AP. In particular, since the hand movement is displayed in animation and the insertion shape of the insertion part is displayed synchronously, it is easy to understand.

[0117] In addition, Fig.16 In the example shown in FIG. 5 , only the insertion shape IPF is displayed in the window DP5 . However, an image on which the insertion shape IPF is superimposed may be displayed on the image of the model 102 captured by the overhead camera 36 .

[0118] Fig.18 1 is a diagram showing an example of an image displayed in the window DP5, in which the insertion shape IPF is superimposed on the image of the model 102. Fig.18 With such an image, the user can easily understand the state of the insertion portion 6a in the intestinal tube portion 102b.

[0119] Furthermore, by using the above-described table data TBL, various displays other than the above-described display example can be performed.

[0120] In addition, the endoscope insertion operation evaluation processor 33 may acquire various information from the endoscope device 2, generate and output display data including the user's operation information on the endoscope device 2. For example, when a hardness variable operation is performed on the insertion portion 6a, if display data including information indicating that the hardness variable operation has been performed is generated, the user can evaluate the insertion operation in association with the operation information on the endoscope device 2.

[0121] Furthermore, the position of the distal end portion 11 of the insertion portion 6a can be estimated based on the shape information of the endoscope shape acquisition device 3, and the estimated position information can also be included to generate and output display data. Based on the position information of the distal end portion 11, a message such as "passing through the sigmoid colon" can also be displayed on the graph displayed by the display device 34.

[0122] In addition, a force sensor capable of detecting force based on a user's operation may be provided on the model 102 . Fig.19 This is a diagram for explaining the arrangement of a force gauge as a force sensor. The force gauge 91 measures the force of compression or extension. The force gauge 91 is installed on the workbench 101 in such a way that a sensor portion 91a for force detection of the force gauge 91 contacts the surface of the model 102 on the opposite side of the anus 102c. A cable 92 for outputting a measurement signal extends from the force gauge 91.

[0123] The model 102 is fixed on two linear guides 93, and the two linear guides 93 are fixed on the workbench 101. That is, the model 102 as the subject is mounted on the two linear guides 93 as a stand, and the stand moves smoothly within a prescribed range in the length direction along the direction in which the insertion portion 6a is inserted. The model 102 is fixed on the linear guides 93 in such a manner that it moves within a prescribed range on the linear guides 93. The dynamometer 91 is fixed on the workbench 101 in such a manner that the sensor portion 91a of the dynamometer 91 abuts against the model 102. In addition, the linear guide 93 may be one, or the model 102 may be moved in the length direction by rolling the model 102 with rollers or by magnetic levitation.

[0124] like Fig.19 As shown, the output of the force gauge 91 is output to the endoscope insertion operation evaluation processor 33 through the cable 92. The endoscope insertion operation evaluation processor 33 receives the measurement signal of the force gauge 91. Therefore, the force gauge 91 constitutes a force detection unit that detects the force applied to the phantom 102 or the insertion portion 6a as the subject.

[0125] The force gauge 91 thus arranged can detect the force of the user's operation on the insertion portion 6a, that is, the force of pushing and pulling. That is, the force gauge 91 detects the force applied to the subject or the insertion portion 6a in the length direction along the direction of inserting the insertion portion 6a and in a direction different from the length direction (here, the opposite direction). In addition, the force gauge 91 may detect only one of the forces in the length direction along the direction of inserting the insertion portion 6a and in a direction different from the length direction (here, the opposite direction).

[0126] Fig. 20 1 is a diagram showing an example of an image showing a graph indicating changes in the force applied to the model. The endoscope insertion operation evaluation processor 33 outputs information related to the force in association with information about the insertion shape IPF in time. As a result, a graph indicating changes in the insertion force corresponding to the passage of time is displayed in the window DP7. If the image Fig.15 Arrange 2 like this Fig. 20 The user can easily know the difference in insertion force between the two doctors by looking at the curve graph.

[0127] As described above, according to the above embodiment, it is possible to provide a monitoring system and an evaluation method of an endoscope insertion operation that can provide a prompt for how to move a hand holding an insertion portion.

[0128] In addition, as a component that can detect the force of inserting the insertion part 6a into the model 102 through the user's operation, in addition to the dynamometer 91, for example, a torque sensor (not shown) that measures the user's near-front rotational force (the degree of torque with which the user twists the insertion part 6a) can be added, and the information from the torque sensor can be added to the detection result.

[0129] The user can use the evaluation data to display desired display data, thereby being able to evaluate the insertion operation. In particular, since the hand movement information and the insertion shape information of the endoscope shape acquisition device 3 are recorded in time association, the hand movement and the insertion shape of the insertion portion 6a can be associated and confirmed.

[0130] Furthermore, the above-mentioned embodiment has been described as an endoscope insertion training system for training the insertion operation of an endoscope, but the above-mentioned embodiment can also be applied to a system for simply evaluating the insertion operation instead of for training.

[0131] Furthermore, the above-described embodiment is a mode in which the insertion operation of the endoscope into the model is taken as an example, but it can also be used as an evaluation device for the insertion operation into a patient or the like.

[0132] In addition, in the present embodiment, the recording function, display function, etc. of the endoscope insertion operation evaluation processor 33 are executed by software, but each function may be configured as an individual electronic circuit, or may be configured as a circuit block in an integrated circuit such as an FPGA (Field Programmable Gate Array). In addition, in the present embodiment, for example, the endoscope insertion operation evaluation processor 33 may be configured to have one or more CPUs.

[0133] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention.

[0134] This application is filed based on Japanese Patent Application No. 2019-101321 filed in Japan on May 30, 2019 as a priority claim, and the above-mentioned disclosed content is cited in the description and claims of this application.

Claims

1. A monitoring system comprising: a shape detection device that detects an insertion shape of an insertion portion of an endoscope inserted into a subject; a motion detection device that detects at least one of a position and a posture in a three-dimensional space of a hand of an operator who operates the endoscope and holds the insertion portion; and An endoscope operation evaluation device comprises a processor, wherein the processor is configured to calculate at least one parameter of the hand's movement amount, movement speed, rotation angle, and rotation speed based on information related to at least one of the position and posture of the hand in the three-dimensional space output from the motion detection device, and output the at least one parameter in temporal association with information related to the insertion shape of the insertion portion from the shape detection device.

2. The monitoring system according to claim 1, wherein: The motion detection device is a motion capture device as follows: it has a first camera device that captures a marking device attached to the hand of the operator, and detects the movement of the hand based on the movement of the marking device captured by the first camera device.

3. The monitoring system according to claim 2, wherein: The marking device is configured to have a plurality of markings and is mounted on the wrist of the operator's hand. The first imaging device includes a plurality of cameras for imaging the plurality of marks from a plurality of directions in the three-dimensional space.

4. The monitoring system according to claim 3, wherein: At least two of the plurality of marks are arranged at intervals of 80 mm or more so that a state in which the plurality of marks protrude from the wrist can be detected by any of the plurality of cameras.

5. The monitoring system according to claim 2, wherein: The monitoring system includes a second imaging device for capturing an image of the insertion portion inserted into the subject. The processor calculates a specified parameter based on a specified marker based on the camera image of the second camera device, and outputs the specified parameter in temporal association with information about the insertion shape of the insertion part. The specified marker is set on the outer surface of the insertion part photographed by the second camera device.

6. The monitoring system according to claim 5, wherein: The prescribed identification has: a first belt that surrounds the outer circumference of the insertion portion in an annular shape in a direction perpendicular to the longitudinal axis of the insertion portion; and The second belt is provided obliquely with respect to the first belt and spirally surrounds the outer circumference of the insertion portion.

7. The monitoring system according to claim 1, wherein: The monitoring system further includes a force detection unit for detecting a force applied to the subject or the insertion unit. The processor outputs the information about the force in temporal association with the information about the insertion shape.

8. The monitoring system according to claim 7, wherein: The subject is mounted on a gantry that moves within a predetermined range in a longitudinal direction along a direction in which the insertion portion is inserted. The force detection unit detects a force applied to the subject or the insertion portion in at least one of the longitudinal direction and a direction different from the longitudinal direction.

9. The monitoring system according to claim 1, wherein: The processor outputs the first display data and the second display data, The first display data displays information related to a first parameter among the movement amount, the movement speed, the rotation angle, and the rotation speed of the hand within a first time period. The second display data displays information related to a second parameter among the movement amount, the movement speed, the rotation angle, and the rotation speed of the hand within a second time.

10. The monitoring system according to claim 1, wherein: The processor outputs display data for displaying the insertion shape in synchronization with an animated image of the hand generated based on the information of the position and the posture.

11. The monitoring system according to claim 1, wherein: The processor outputs display data that plots the position of the hand in a graph over time based on the information about the position of the hand in the three-dimensional space.

Images