Endoscope system, method for controlling endoscope system, and control program

Abstract

The present invention facilitates the measurement of the size of an object in an endoscopic examination. An endoscope system (100) is provided with: an acquisition unit (101) that acquires an endoscopic image; a detection unit (102) that detects a first position in the endoscopic image corresponding to a position of a distal end portion of a treatment tool (41) protruding from a distal end of an insertion portion (42) of an endoscope device (4); an estimation unit (103) that estimates a second position in the endoscopic image corresponding to a position at which the distance from the distal end portion of the treatment tool (41) is a first distance; and a display control unit (105) that causes a display device (5) to display a composite image obtained by combining position information, which includes information indicating the first position and information indicating the second position, with the endoscopic image.

Description

Endoscope system, control method for endoscope system, and control program

[0001] The present disclosure relates to an endoscope system, a control method for an endoscope system, and the like.

[0002] Endoscopes are widely used in the medical and engineering fields to examine the position and size of objects in narrow spaces and to perform predetermined treatments on the objects. For example, in the medical field, endoscopes are used to examine and treat diseased tissues (objects) such as tumors and masses occurring in the trachea and digestive tract.

[0003] The scope of application of endoscopic therapy is determined in the guidelines based on the size of the diseased tissue, and doctors determine whether or not the diseased tissue needs to be resected and select the resection method depending on the size of the diseased tissue.

[0004] In endoscopic examinations, the size of an object is estimated based on the experience of a doctor or other professional who has reviewed an endoscopic image of the object. However, there is room for improvement in the accuracy and reproducibility of sizes estimated by visual inspection of the endoscopic image.

[0005] Various techniques are known for measuring the size of an object using an endoscope.

[0006] For example, Patent Document 1 discloses an object measuring device that measures the size of an object by using lines drawn at equal intervals on an endoscope distal end hood.

[0007] Patent Document 2 discloses an endoscopic length measuring tool having a light emitting means for illuminating a scale provided on a protruding portion that protrudes from the distal end of an endoscope through an endoscopic treatment tool insertion hole.

[0008] Patent document 3 discloses a method of endoscopic measurement that uses augmented reality techniques to identify features provided at the tip of an endoscopic instrument, determine the position and orientation of the instrument in an endoscopic image, and superimpose a virtual ruler on the tip of the instrument that is in contact with an object in the endoscopic image.

[0009] Patent Document 4 discloses an imaging device that irradiates an object with laser light, distorts the object due to distortion based on the light reflected from the object, and creates a captured image of the object. The device also combines a distorted scale, which is an indicator of the size of the object in the captured image and has distortion equivalent to that caused by distortion, with the captured image.

[0010] Patent Document 5 discloses an endoscope device that emits planar measurement auxiliary light into the field of view of the imaging optical system of the insertion section of the endoscope, and displays a scale on an intersection line formed at the point where the plane formed by the measurement auxiliary light intersects with the target object.

[0011] Furthermore, a technique is known in which the position of the tip of a treatment tool in an endoscopic image is detected by image analysis using artificial intelligence (for example, Patent Document 6).

[0012] Japanese Patent Publication No. 2013-248353 Japanese Publication No. 2-045030 Publication Japanese Special Publication No. 2020-516408 Publication Japanese Patent Publication No. 2011-069965 WO2019 / 017019 Japanese Patent Publication No. 2023-030681

[0013] The techniques disclosed in Patent Documents 1 to 5 all require special devices and equipment, and therefore have the problem that they cannot be applied to endoscopic examinations using general endoscopic examination equipment that is commercially available (i.e., does not have special equipment).

[0014] For example, to apply the techniques disclosed in Patent Documents 1 and 2, an endoscopic examination device equipped with an endoscope distal end hood having lines drawn at equal intervals and a protrusion provided with a scale is required. To apply the techniques disclosed in Patent Documents 4 and 5, an endoscopic examination device equipped with a device for irradiating laser light and a device for emitting planar auxiliary measurement light is required. To apply the technique disclosed in Patent Document 3, an identification feature must be provided at the tip of the treatment tool.

[0015] The present disclosure provides an endoscope system and the like that can measure the size of an object during endoscopic examination using any endoscopic examination device that does not require special equipment.

[0016] An endoscopic system according to one aspect of the present disclosure includes an acquisition unit that acquires an endoscopic image captured by an endoscopic device, a detection unit that detects a first position in the endoscopic image corresponding to the position of a tip of a treatment tool protruding from the tip of an insertion section of the endoscopic device, an estimation unit that estimates a second position in the endoscopic image corresponding to a position that is a first distance from the tip of the treatment tool using a trained estimation model, and a display control unit that displays on a display device a composite image obtained by combining the endoscopic image with positional information including information indicating the first position and information indicating the second position.

[0017] A control method for an endoscopic system according to one aspect of the present disclosure is a control method for an endoscopic system executed by one or more computers, and includes an acquisition step of acquiring an endoscopic image captured by an endoscopic device; a detection step of detecting a first position in the endoscopic image corresponding to the position of a tip of a treatment tool protruding from the tip of an insertion section of the endoscopic device; an estimation step of estimating a second position in the endoscopic image corresponding to a position that is a first distance from the tip of the treatment tool using a trained estimation model; and a display control step of displaying on a display device a composite image obtained by combining the endoscopic image with positional information including information indicating the first position and information indicating the second position.

[0018] The endoscopic system according to each aspect of the present disclosure may be realized by a computer. In this case, the control program for the endoscopic system that causes the computer to operate as each part (software element) of the endoscopic system and realizes the endoscopic system on the computer, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present disclosure.

[0019] According to one aspect of the present disclosure, the size of an object during endoscopic examination can be measured using any endoscopic examination equipment without special equipment.

[0020] 1 is a block diagram showing an example of a schematic configuration of an endoscopic system according to a first embodiment of the present disclosure. FIG. 2 is a functional block diagram showing an example of a configuration of an endoscopic system. FIG. 3 is a diagram explaining an example of a process for detecting the tip of a treatment tool. FIG. 4 is a diagram showing an example of a method for capturing a reference image used for learning an estimation model. FIG. 5 is a diagram showing an example of a reference image. FIG. 6 is a diagram showing another example of a method for capturing a reference image used for learning an estimation model. FIG. 7 is a diagram showing the relationship between the position of the tip of the treatment tool in an endoscopic image and positions at distances of 5 mm, 10 mm, 15 mm, and 20 mm from the tip of the treatment tool. FIG. 8 is a diagram showing an example of position information displayed on a composite image. FIG. 9 is a diagram showing an example of a composite image displayed on a display device. FIG. 10 is a flowchart showing an example of a processing flow performed by an endoscopic system. FIG. 11 is a block diagram showing an example of a configuration of an estimation device according to a second embodiment of the present disclosure. FIG. 12 is a diagram showing an example of a composite image displayed when a first operation is received. FIG. 13 is a block diagram showing an example of a schematic configuration of an endoscopic system according to a third embodiment of the present disclosure. FIG. 14 is a functional block diagram showing an example of a configuration of an endoscopic system. FIG. 15 is a block diagram showing an example of a schematic configuration of an endoscopic system according to a fourth embodiment of the present disclosure.

[0021] First Embodiment Hereinafter, one embodiment of the present disclosure will be described in detail with reference to FIGS.

[0022] (Overview of the endoscopic system 100) The endoscopic system 100 is a system capable of displaying position information 114 in an endoscopic image 111 captured by an endoscopic device 4, which facilitates measuring the size of an object appearing in the endoscopic image 111.

[0023] When the endoscopic system 100 according to the present disclosure is applied to the medical field, the endoscopic device 4 may be a device used in endoscopic treatment, and the target may be diseased tissue such as a polyp, an ulcer, or a tumor. When the endoscopic system 100 according to the present disclosure is applied to the engineering field, the endoscopic device 4 may be a device for inspecting the condition of an object based on images captured in a narrow space, and the target may be a component used in the object or an event occurring in the object. Here, the event occurring in the object may be, for example, a crack, breakage, corrosion, or the like. In this disclosure, an example in which the endoscopic system 100 is installed in a medical facility H will be described.

[0024] (Configuration of Endoscope System 100) First, the configuration of the endoscope system 100 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the schematic configuration of the endoscope system 100.

[0025] As shown in FIG. 1 , the endoscopic system 100 may include an endoscopic device 4, an image management device 6, an estimation device 1, and a display device 5. The endoscopic device 4, the image management device 6, the estimation device 1, and the display device 5 may be directly connected, or as shown in FIG. 1 , may be communicatively connected via a local area network (LAN). Alternatively, the endoscopic device 4, the image management device 6, the estimation device 1, and the display device 5 may be communicatively connected via the Internet. The number of each of the endoscopic device 4, the image management device 6, the estimation device 1, and the display device 5 included in the endoscopic system 100 may be one or more. When using endoscopic images 111 captured in advance (e.g., recorded), the endoscopic device 4 is not a required component of the endoscopic system 100.

[0026] [Endoscopic Device 4] The endoscopic device 4 is a known endoscopic device and includes an insertion section 42 that is inserted into the body of a subject from its distal end (the tip farthest from the main body of the endoscopic device 4), a control section (not shown) attached to a proximal end different from the distal end, a light that illuminates the inside of the body of the subject, and a camera that can capture images of the inside of the body of the subject. For example, by performing a predetermined operation on the control section, a doctor (user) can (1) bend the first end of the insertion section 42, (2) cause the treatment tool 41 to protrude from the distal end of the insertion section 42 to perform a predetermined treatment, and (3) capture an endoscopic image 111 that shows the inside of the body of the subject. Hereinafter, the distal end of the insertion section 42 will be referred to as the tip of the insertion section 42.

[0027] The endoscopic image 111 may be an image acquired while the endoscopic device 4 is being used, or may be configured to be instantly displayed on the display device 5 described below. Alternatively, the endoscopic image 111 may be recorded and stored in the image management device 6 or the storage unit 11 of the estimation device 1. When the treatment tool 41 is protruded from the tip of the insertion section 42, an endoscopic image 111 showing the tip ET of the treatment tool 41 can be acquired.

[0028] The insertion section 42 may be provided with a channel 43 (which may also be referred to as a forceps hole) through which the treatment tool 41 is inserted so as to be able to advance and retreat. By making the distal end portion ET of the treatment tool 41 protrude from the distal end opening of the channel 43, the treatment tool 41 can be made to protrude from the distal end of the insertion section 42.

[0029] [Image Management Device 6] The image management device 6 is a computer that stores and manages medical images. The image management device 6 may include a control unit and a memory unit. In one example, the control unit may be a CPU (Central Processing Unit). The memory unit may store endoscopic images 111 captured using an endoscopic device to show the inside of a subject's body, in association with the subject's identification information, the date and time of capture, and identification information indicating the doctor who captured the endoscopic images 111. The image management device 6 may read the endoscopic images 111 from the memory unit in response to an instruction from the estimation device 1 and output them to the estimation device 1.

[0030] [Estimation Device 1] The estimation device 1 is a computer used by a medical professional such as a doctor. The estimation device 1 analyzes an endoscopic image 111 captured inside the body of a subject as follows. The estimation device 1 generates a composite image 115 by combining position information 114 with the endoscopic image 111. The configuration of the estimation device 1 will be described later with a specific example.

[0031] The position information 114 includes (1) information indicating a first position T in the endoscopic image 111, which corresponds to the position of the tip ET of the treatment tool 41 shown in the endoscopic image 111, and (2) information indicating a second position in the endoscopic image 111, which corresponds to a position that is a first distance from the detected tip ET of the treatment tool 41. Then, the estimation device 1 displays the composite image 115 on the display device 5.

[0032] The position information 114 may include at least one piece of information indicating the second position. For example, a numerical value indicating the size of the lesioned tissue, as defined in the guidelines for endoscopic treatment, may be used as the first distance as a criterion for determining whether or not to resect the lesioned tissue and for selecting a resection method. A physician can easily compare the position information 114 shown in the composite image 115 generated from the endoscopic image 111 capturing the state in which the distal end ET of the treatment tool 41 is brought close to the edge of the lesioned tissue with the size of the lesioned tissue shown in the composite image 115 on the screen. In other words, the composite image 115 generated by the estimation device 1 can facilitate the physician performing the endoscopic treatment in determining whether or not to resect the lesioned area and selecting a resection method.

[0033] Alternatively, the position information 114 may include, in addition to information indicating the second position, information indicating positions in the endoscopic image 111 corresponding to each of a plurality of positions whose detected distances from the distal end ET of the treatment tool 41 are different from the first distance. For example, the position information 114 may include information indicating positions in the endoscopic image 111 corresponding to each of positions whose detected distances from the distal end ET of the treatment tool 41 are the first distance, twice the first distance, three times the first distance, .... In this case, the first distance may be set arbitrarily. For example, the first distance may be 2 mm or 5 mm. In one example where the first distance is 5 mm, the position information 114 may include information indicating positions in the endoscopic image 111 corresponding to each of positions whose detected distances from the distal end ET of the treatment tool 41 are 5 mm, 10 mm, 15 mm, and 20 mm. Such position information 114 may be used as a scale within the endoscopic image 111. The doctor can easily measure the size of the diseased area by comparing on the screen the position information 114 shown in the composite image 115 generated from the endoscopic image 111 capturing the state in which the tip ET of the treatment tool 41 is brought close to the edge of the diseased tissue with the size of the diseased tissue shown in the composite image 115.

[0034] [Display Device 5] The display device 5 is a display capable of displaying various types of information on a screen. The display device 5 may be a display included in the endoscope device 4. Alternatively, the display device 5 may be a display (display unit) included in the estimation device 1.

[0035] (Configuration of Estimation Apparatus 1) Next, the configuration of the estimation apparatus 1 will be described with reference to Fig. 2. Fig. 2 is a functional block diagram showing an example of the configuration of an endoscope system 100. The estimation apparatus 1 includes a control unit 10 and a storage unit 11. In one example, the control unit 10 may be a CPU (Central Processing Unit). The control unit 10 reads a control program, which is software stored in the storage unit 11, and expands it into a memory such as a RAM (Random Access Memory) to execute various functions.

[0036] The storage unit 11 may be configured with a non-volatile storage device such as a hard disk or flash memory. The storage unit 11 may store an endoscopic image 111 captured inside the body of a subject. The storage unit 11 may also store a detection model 112 and an estimation model 113, which will be described later. The storage unit 11 may also store position information 114 estimated from the endoscopic image 111 by the estimation device 1, and a composite image 115 generated by the estimation device 1.

[0037] The control unit 10 may include an acquisition unit 101 , a detection unit 102 , an estimation unit 103 , a composite image generation unit 104 , and a display control unit 105 .

[0038] The acquisition unit 101 acquires an endoscopic image 111 captured by the endoscopic device 4. The acquisition unit 101 may acquire the endoscopic image 111 from the endoscopic device 4 or from the image management device 6. If the endoscopic image 111 is stored in advance in the storage unit 11, the acquisition unit 101 may acquire the endoscopic image 111 by reading it out from the storage unit 11.

[0039] The detection unit 102 detects a first position T in the endoscopic image 111, which corresponds to the position of the distal end ET of the treatment tool 41 protruding from the distal end of the insertion section 42 of the endoscopic device 4. FIG. 3 is a diagram illustrating an example of a process for detecting the distal end ET of the treatment tool 41. For simplicity of explanation, FIG. 3 illustrates only the treatment tool 41 depicted in the endoscopic image 111. In one example, the detection unit 102 may generate a bounding box surrounding the treatment tool 41 in the endoscopic image 111 using the trained detection model 112 and detect the first position T based on the coordinates of the bounding box. In the example illustrated in FIG. 3 , the coordinates (X1, Y1) of the bounding box surrounding the treatment tool 41 in the endoscopic image 111 are detected as the coordinates of the first position T. The first position T (X1, Y1) is the coordinate of a corner of the bounding box surrounding the treatment tool 41 detected in the endoscopic image 111, which is located within the endoscopic image 111.

[0040] [Detection Model 112] Here, machine learning for creating the detection model 112 will be described. Training endoscopic images showing the treatment tool 41 protruding from the tip of the insertion section 42 are collected, and the training endoscopic images are classified and saved by type of treatment tool 41. Using the saved training endoscopic images, transfer learning is performed using YOLO, MobileNet, EfficientNet, VGG16, etc., to create a detection model 112 for each type of treatment tool 41. When an endoscopic image 111 is input into the detection model 112 created in this manner, the treatment tool 41 shown in the endoscopic image 111 is recognized, and the treatment tool 41 is surrounded by a bounding box. Because the bounding box contains information about the position of the detected treatment tool 41, it is possible to obtain coordinates corresponding to the position T of the tip ET of the treatment tool 41 in the endoscopic image 111 (X1 and Y1 in FIG. 3 ).

[0041] Note that the position at which the treatment tool 41 appears in the endoscopic image 111 may differ depending on the endoscope model and manufacturer of the endoscopic device 4. Therefore, learning endoscopic images may be collected for each manufacturer of the endoscopic device 4, and a detection model 112 may be created for each manufacturer.

[0042] The detection model 112 may be a model that recognizes the shape of the treatment tool 41 from the endoscopic image 111 and detects its tip portion ET. In this case, learning endoscopic images showing the treatment tool 41 protruding from the tip of the insertion section 42 are collected, and the learning endoscopic images are classified and saved according to the shape of the tip portion ET of the treatment tool 41. Using the saved learning endoscopic images, transfer learning is performed using YOLO, MobileNet, EfficientNet, VGG16, or the like, and a detection model 112 is created for each shape of the tip portion ET of the treatment tool 41.

[0043] 2 , the estimation unit 103 estimates a second position in the endoscopic image 111 corresponding to a position that is a first distance from the distal end ET of the treatment tool 41, using the trained estimation model 113. The estimation unit 103 may store position information 114 including information indicating the first position T detected by the detection unit 102 and information indicating the estimated second position in the storage unit 11. The second position may be, for example, a position in the endoscopic image 111 that corresponds to a position that is 10 mm away from the distal end ET of the treatment tool 41.

[0044] The estimation unit 103 may estimate, together with the second position, positions in the endoscopic image 111 corresponding to one or more positions whose distance from the distal end ET of the treatment tool 41 is different from the first distance, detected by the detection unit 102. For example, the estimation unit 103 may estimate a third position in the endoscopic image 111 corresponding to a position whose distance from the distal end ET of the treatment tool 41 is a second distance different from the first distance, using the trained estimation model 113. The following describes, as an example, a configuration in which the estimation unit 103 estimates positions in the endoscopic image 111 corresponding to positions whose distances from the distal end ET of the treatment tool 41 are 5, 10, 15, and 20 mm.

[0045] [Estimation Model 113] Next, a specific example will be described of a machine learning method for generating the estimation model 113. The machine learning of the estimation model 113 can use a plurality of reference images captured by the endoscope device 4 of the plate-like member R1 fixed to the distal end portion ET of the treatment tool 41 protruding from the distal end of the insertion section 42 of the endoscope device 4.

[0046] A position for fixing the distal end ET of the treatment tool 41 is set on the plate-shaped member R1, and a mark indicating a position that is a first distance from the predetermined position is displayed on the plate-shaped member R1. That is, when the distal end ET of the treatment tool 41 is fixed to a predetermined position on the plate-shaped member R1, the mark displayed on the plate-shaped member R1 becomes a mark indicating a position that is the first distance from the distal end ET of the treatment tool 41. Note that, when estimating positions in the endoscopic image 111 that correspond to one or more positions that are different from the first distance from the distal end ET of the treatment tool 41, marks corresponding to the positions to be estimated can be displayed on the plate-shaped member R1.

[0047] First, the treatment tool 41, with the plate-like member R1 fixed to the tip ET, is kept protruding a predetermined distance (e.g., 10 mm) from the tip of the insertion section 42 of the endoscopic device 4, and multiple reference images showing the treatment tool 41 and the plate-like member R1 are obtained using the endoscopic device 4.

[0048] FIG. 4 is a diagram showing an example of a method for capturing a reference image used to train the estimation model 113. FIG. 4 shows a state in which a plate-shaped member R1 is fixed to the distal end ET of the treatment tool 41, which is inserted through the channel 43, and protrudes a predetermined distance (e.g., 10 mm) from the distal end of the insertion section 42 of the endoscope device 4. The plate-shaped member R1 is fixed to the distal end ET of the treatment tool 41 substantially parallel to a plane perpendicular to the direction of protrusion of the treatment tool 41. Marks L5, L10, L15, and L20 are shown on the plate-shaped member R1. For example, the marks L5, L10, L15, and L20 may indicate positions 5 mm, 10 mm, 15 mm, and 20 mm from the distal end ET of the treatment tool 41, respectively. That is, the distances between the intersections of the axis D shown in Figure 4 with the marks L5, L10, L15, and L20 shown on the plate-shaped member R1 and the tip ET of the treatment tool 41 are 5 mm, 10 mm, 15 mm, and 20 mm in actual dimensions.

[0049] 5 is a diagram illustrating an example of a reference image. As shown in FIG. 5, the reference image shows a treatment tool 41 and a plate-like member R1. The estimation model 113 has machine-learned the correspondence between the position of the distal end ET of the treatment tool 41 in the reference image and the positions of the marks L5, L10, L15, and L20. This allows the estimation model 113 to accurately estimate, based on the position of the distal end ET of the treatment tool 41 detected in the endoscopic image 111, a position in the endoscopic image 111 that corresponds to a position that is a first distance from the position of the distal end ET of the treatment tool 41.

[0050] In the plate-shaped member R1, the mark indicates a position where the distance from the distal end ET of the treatment tool 41 is the first distance, but this is not limiting. For example, the mark may be indicated by a dot. Fig. 6 is a diagram showing another example of a method for capturing a reference image used for training the estimation model 113. Marks L5, L10, L15, and L20 shown on the plate-shaped member R2 shown in Fig. 6 indicate positions where the distance from the distal end ET of the treatment tool 41 is 5 mm, 10 mm, 15 mm, and 20 mm, respectively.

[0051] When acquiring a reference image, it is desirable to acquire a reference image each time the position of the distal end ET of the treatment tool 41 is changed in various ways. For example, consider two images A and B taken using the endoscope device 4 of two points that are the same distance apart. In image A, the two points are captured in the center of the screen, while in image B, the two points are captured at the edge of the screen. In this case, the distance between the two points in image A is greater than the distance between the two points in image B. This is because the endoscopic image 111 exhibits fisheye lens characteristics, and the edges of the image are compressed compared to the center of the image.

[0052] The estimation model 113 is created by performing machine learning using the position of the tip ET of the treatment tool 41 in the reference image as an explanatory variable and the position of the mark in the reference image as a target variable. A known machine learning library (e.g., Scikit-Learn) may be applied to create the estimation model 113. The estimation unit 103 may estimate information indicating the second position (e.g., coordinates) from the coordinates (X1, Y1) of the first position T detected in the endoscopic image 111 using the trained estimation model 113.

[0053] Fig. 7 is a diagram showing the relationship between the position of the distal end ET of the treatment tool 41 in the endoscopic image 111 and positions at distances of 5 mm, 10 mm, 15 mm, and 20 mm from the distal end ET of the treatment tool 41. The horizontal axis in Fig. 7 represents the X-coordinate value x (pixel) of the position of the distal end ET of the treatment tool 41 in the endoscopic image 111. The vertical axis in Fig. 7 represents the X-coordinate value x of the position in the endoscopic image 111 estimated as the position at distances of 5 mm, 10 mm, 15 mm, and 20 mm from the distal end ET of the treatment tool 41.Ｌ５ , x Ｌ１０ , x Ｌ１５ , and x Ｌ２０ From the plotted results shown in FIG. 7, for example, the following regression equation can be obtained: x Ｌ５ =0.9051x-136.06x Ｌ１０ =0.6802x-38.458x Ｌ１５ =0.5056x+82.897x Ｌ２０ = 0.351x + 228.32 The estimation unit 103 calculates the value x Ｌ５ , x Ｌ１０ , x Ｌ１５ , and x Ｌ２０ may be estimated (calculated) from the coordinates (X1, Y1) of the first position T detected in the endoscopic image 111 using the regression equation.

[0054] Although FIG. 7 illustrates a configuration in which the estimation unit 103 estimates the X coordinate of the second position from the X coordinate value of the first position T, the estimation unit 103 is not limited to this configuration. For example, the estimation unit 103 may be configured to estimate the Y coordinate of the second position from the Y coordinate value of the first position T. Alternatively, the estimation unit 103 may be configured to estimate a position corresponding to a curve corresponding to the second position from the coordinate value of the first position T. The curve estimated by the estimation unit 103 may be, for example, a curve such as the marks L5, L10, L15, and L20 shown in the endoscopic image 111 (see FIG. 5 ). In these cases, an estimation model 113 created by machine learning using the position of the distal end ET of the treatment tool 41 in the reference image as an explanatory variable and the position of the marks in the reference image as a target variable can be used.

[0055] 2 , the composite image generation unit 104 generates a composite image 115 by combining the endoscopic image 111 with position information 114 including information indicating the first position T and information indicating the second position. Here, in the composite image 115, the position information 114 may be displayed in the endoscopic image 111 as a scale bar having the first position T as a first end and indicating the second position on a straight line extending from the first end to the second end. The scale bar may have scales indicating positions corresponding to each of the multiple positions estimated by the estimation unit 103.

[0056] The display control unit 105 displays the composite image 115 on the display device 5. Note that the composite image generation unit 104 is not an essential component of the estimation device 1. If the estimation device 1 does not include the composite image generation unit 104, the estimation device 1 stores position information 114 in the storage unit 11, the position information 114 including information indicating the first position T detected by the detection unit 102 and information indicating the second position estimated by the estimation unit 103. The display control unit 105 may then read out the endoscopic image 111 and the position information 114 from the storage unit 11, and display on the display device 5 a composite image 115 in which the position information 114 (e.g., the scale bar described above) is combined (superimposed) on the endoscopic image 111.

[0057] When the estimation unit 103 estimates a position corresponding to a curve corresponding to the position of the second position from the coordinate values ​​of the first position T, the display control unit 105 may cause the display device 5 to display a composite image 115 including the position information 114 as a point indicating the first position T and a curve indicating the second position.

[0058] FIG. 8 is a diagram showing an example of position information 114 displayed on the composite image 115. FIG. 8 shows a scale bar W1 that is displayed when the coordinates of the first position T in the endoscopic image 111 are (X2, Y2), and a scale bar W2 that is displayed when the coordinates are (X3, Y3). FIG. 8 also shows a grid for representing the fisheye lens characteristics in the endoscopic image 111. The scale bars W1 and W2, which have different coordinates for the detected first position T, have different intervals between their scale marks. It can be seen that even with the same actual interval of 5 mm, the scale appears narrower as one approaches the end of the endoscopic image 111.

[0059] For example, when the estimation apparatus 1 instantly acquires an endoscopic image 111 captured by the endoscopic device 4, the display control unit 105 may cause the display device 5 to display a composite image 115 including a scale bar corresponding to the changed first position T each time the first position T of the distal end ET of the treatment tool 41 is changed in the endoscopic image 111. For example, if the first position T is (X2, Y2), the display control unit 105 causes the display device 5 to display a composite image 115 including a scale bar W1, and then, if the first position T is changed to (X3, Y3), the display control unit 105 causes the display device 5 to display a composite image 115 including a scale bar W2.

[0060] 9 is a diagram showing an example of a composite image 115 displayed on the display device 5. When the distal end ET of the treatment tool 41 ("E" in FIG. 9 and FIG. 12 described later) is positioned near the object P, the display control unit 105 displays a composite image 115 including a scale bar W3 that indicates a second position on a straight line extending from the first end to the second end, with the first position T as the first end. A doctor who checks the composite image 115 shown in FIG. 9 can measure the size of the object P (e.g., a polyp) to be "15 mm."

[0061] As described above, the endoscopic system 100 includes an acquisition unit 101, a detection unit 102, an estimation unit 103, and a display control unit 105. The acquisition unit 101 acquires an endoscopic image 111. The detection unit 102 detects a first position T in the endoscopic image 111, which corresponds to the position of the tip end ET of the treatment tool 41 protruding from the tip of the insertion unit 42 of the endoscopic device 4. The estimation unit 103 estimates a second position in the endoscopic image 111, which corresponds to a position that is a first distance from the tip end ET of the treatment tool 41. The display control unit 105 causes the display device 5 to display a composite image 115 obtained by combining the endoscopic image 111 with position information 114 including information indicating the first position T and information indicating the second position.

[0062] (Processing Performed by Endoscope System 100) Next, processing performed by the endoscope system 100 (i.e., the estimation device 1) will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of processing executed by the endoscope system 100.

[0063] First, the acquisition unit 101 acquires the endoscopic image 111 (step S1: acquisition step).

[0064] Next, the detection unit 102 detects a first position T in the endoscopic image 111 (step S2: detection step).

[0065] Next, the estimation unit 103 estimates a second position in the endoscopic image 111 that corresponds to a position that is the first distance from the distal end ET of the treatment tool 41 (step S3: estimation step).

[0066] Next, the display control unit 105 causes the display device 5 to display a composite image 115 that combines the endoscopic image 111 with position information 114 that includes information indicating the first position T and information indicating the second position (step S4: display control step).

[0067] As a result, the endoscopic system 100 can present the physician with a composite image 115 including position information 114 that facilitates measurement of the size of an object shown in the endoscopic image 111. For example, when the physician wishes to measure the size of an object shown in the endoscopic image 111, the physician simply moves the distal end ET of the treatment tool 41 to the vicinity of the object. The composite image 115 displayed on the display device 5 displays position information 114, such as a scale bar indicating the distance from the distal end ET of the treatment tool 41, allowing the physician to easily and accurately measure the size of the object.

[0068] [Embodiment 2] Another embodiment of the present disclosure will be described below with reference to Figures 11 and 12. For ease of explanation, members having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[0069] 11 is a block diagram showing an example of the configuration of an endoscopic system 100a according to Embodiment 2 of the present disclosure. The endoscopic system 100a differs from the endoscopic system 100 in that the endoscopic system 100a includes an estimation device 1a including a reception unit 12 that receives a first operation by a doctor (user).

[0070] The accepting unit 12 accepts a first operation for changing the aspect of the position information 114 in the composite image 115. The accepting unit 12 may be, for example, a foot pedal. In this case, the accepting unit 12 may be configured to accept one first operation each time the foot pedal is depressed. Alternatively, the accepting unit 12 may be configured to accept a first operation according to the length of time the foot pedal is continuously depressed.

[0071] The receiving unit 12 outputs information about the received first operation to, for example, the composite image generating unit 104. The composite image generating unit 104 changes the aspect of the position information 114 shown in the composite image 115 based on the received information about the first operation. When the position information 114 is displayed as a scale bar, the aspects of the position information 114 that can be changed by receiving the first operation include, for example, the following: - Orientation and tilt of the scale bar - Increasing or decreasing the scale bar scale - Thickness and color of the scale bar - Starting and stopping the display of the scale bar.

[0072] If the orientation of the scale bar can be changed by accepting a first operation, the display control unit 105, in response to the acceptance by the acceptance unit 12 of the first operation, causes the display device 5 to display a scale bar in which the position of the second end has been changed while keeping the position of the first end corresponding to the first position T fixed.

[0073] FIG. 12 is a diagram showing an example of a composite image 115 displayed when a first operation is accepted. As shown in the left diagram of FIG. 12 , the position of the scale bar W4 in the composite image 115 may not be suitable for measuring the size of the object P. In such a case, the doctor performs a first operation on the accepting unit 12. This causes the estimation device 1a to display, on the display device 5, a composite image 115 including a scale bar W5 in which the position of the second end has been changed while the position of the first end corresponding to the first position T is fixed. The right diagram of FIG. 12 is an example of a composite image 115 including a scale bar W5 in which the scale bar W4 has been rotated by an angle θ as a result of accepting the first operation.

[0074] In this way, the estimation device 1a can change the aspect of the position information 114 in the composite image 115 to a desired aspect through the first operation. For example, the doctor can change the aspect of the position information 114 in the composite image 115 to match the position, shape, etc. of the object P in the endoscopic image 111. Therefore, the doctor can easily and accurately measure the size of the object P.

[0075] [Embodiment 3] Another embodiment of the present disclosure will be described below with reference to Figures 13 and 14. For ease of explanation, members having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[0076] 13 is a block diagram showing an example of a schematic configuration of an endoscopic system 100b according to a third embodiment of the present disclosure. The endoscopic system 100b is configured such that the estimation device 1b and the information processing device 7 each have the functions of the estimation device 1. The endoscopic system 100b is introduced in a medical facility Ha, and may include an endoscopic device 4, an image management device 6, the estimation device 1b, the information processing device 7, and a display device 5.

[0077] The endoscopic device 4, image management device 6, estimation device 1b, information processing device 7, and display device 5 may be directly connected, or may be communicatively connected via a local area network (LAN) as shown in Fig. 13. Alternatively, the endoscopic device 4, image management device 6, estimation device 1b, information processing device 7, and display device 5 may be communicatively connected via the Internet. The number of each of the endoscopic device 4, image management device 6, estimation device 1b, information processing device 7, and display device 5 included in the endoscopic system 100b may be one or more.

[0078] FIG. 14 is a functional block diagram showing an example of the configuration of the endoscope system 100b.

[0079] (Configuration of Estimation Device 1b) The estimation device 1b is a computer and includes a control unit 10b and a storage unit 11b. In one example, the control unit 10b may be a CPU. The control unit 10b reads a control program, which is software stored in the storage unit 11b, expands it in a memory such as a RAM, and executes various functions.

[0080] The storage unit 11b may be configured by a non-volatile storage device such as a hard disk, a flash memory, etc. The storage unit 11b may store a detection model 112 and an estimation model 113.

[0081] The control unit 10b includes an acquisition unit 101, a detection unit 102, and an estimation unit 103. A detection model 112 and an estimation model 113 are stored in the storage unit 11b. That is, the estimation device 1b has a function of acquiring an endoscopic image 711 from the endoscope device 4 or the image management device 6, detecting a first position T, and estimating a second position. The estimation device 1b outputs position information 712 including information indicating the first position T and information indicating the second position to the information processing device 7.

[0082] (Configuration of Information Processing Device 7) The information processing device 7 may be a computer used by a medical professional such as a doctor. The information processing device 7 includes a control unit 70 and a storage unit 71. In one example, the control unit 70 may be a CPU. The control unit 70 reads a control program, which is software stored in the storage unit 71, and loads it into a memory such as a RAM to execute various functions.

[0083] The storage unit 71 may be configured with a non-volatile storage device such as a hard disk or flash memory. The storage unit 71 may store an endoscopic image 711 captured inside the body of a subject. The storage unit 71 may also store position information 712 estimated from the endoscopic image 711 by the estimation device 1b and a composite image 713 generated by the information processing device 7.

[0084] The control unit 70 may include an acquisition unit 701 , a composite image generation unit 702 , and a display control unit 703 .

[0085] The acquisition unit 701 acquires an endoscopic image 711 captured by the endoscopic device 4. The acquisition unit 701 may acquire the endoscopic image 711 from the endoscopic device 4 or from the image management device 6. If the endoscopic image 711 is stored in advance in the storage unit 71, the acquisition unit 701 may acquire the endoscopic image 711 by reading it out from the storage unit 71.

[0086] The acquisition unit 701 also acquires position information 712 from the estimation device 1b. The position information 712 includes information indicating a first position T in the endoscopic image 711 and information indicating a second position.

[0087] The composite image generating unit 702 generates a composite image 713 by combining position information 712 including information indicating the first position T and information indicating the second position with the endoscopic image 711. Here, in the composite image 713, the position information 712 may be displayed in the endoscopic image 711 as a scale bar that has the first position T as a first end and indicates the second position on a straight line extending from the first end to the second end.

[0088] The display control unit 703 causes the display device 5 to display the composite image 713. Note that the composite image generation unit 702 is not an essential component of the information processing device 7. If the information processing device 7 does not include the composite image generation unit 702, the information processing device 7 stores the position information 712 in the storage unit 71. The display control unit 703 may then read out the endoscopic image 711 and the position information 712 from the storage unit 71, and cause the display device 5 to display a composite image 713 in which the endoscopic image 711 is combined (superimposed) with the position information 712 (for example, the scale bar described above).

[0089] Here, the estimation device 1b and the information processing device 7 may be configured to instantly acquire the endoscopic image 711 captured by the endoscopic device 4. Alternatively, the endoscopic image 711 may be recorded and stored in the image management device 6 or the storage unit 11b of the estimation device 1b.

[0090] In the endoscope system 100b, separate devices perform the analysis process of the endoscopic image 711 (i.e., the detection of the first position T and the estimation of the second position) and the generation process of the composite image 713. This reduces the load on the control unit 10b of the estimation device 1b that performs the processes using the detection model 112 and the estimation model 113.

[0091] When introducing the endoscopic system 100b into a medical facility Ha, all that is required is to add an existing endoscopic device 4, an information processing device 7 that analyzes endoscopic images 711 captured by the endoscopic device 4, and an estimation device 1b that is communicatively connected to the display device 5.

[0092] When applying a function equivalent to the reception unit 12 (see FIG. 11) provided in the estimation device 1a to the endoscope system 100b, it is sufficient to provide the information processing device 7 with a component equivalent to the reception unit 12 (e.g., a foot pedal, etc.).

[0093] [Embodiment 4] Another embodiment of the present disclosure will be described below with reference to Fig. 15. For ease of explanation, members having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and their description will not be repeated.

[0094] (Configuration of Endoscopic System 100c) Fig. 15 is a block diagram showing an example of a schematic configuration of an endoscope system 100c according to a fourth embodiment of the present disclosure. Fig. 15 shows an example in which the endoscope system 100c is introduced in medical facilities H1 to Hn. In the endoscope system 100c, the number of medical facilities may be one or more. Furthermore, the number of estimation devices 1b included in the endoscope system 100c may be one or more.

[0095] As shown in Fig. 15, in the endoscope system 100c, each of the medical facilities H1 to Hn may include an endoscope device 4, an image management device 6, an information processing device 7, and a display device 5. In each of the medical facilities H1 to Hn, the endoscope device 4, the image management device 6, the information processing device 7, and the display device 5 may be directly connected, or as shown in Fig. 15, may be communicatively connected via a LAN. Furthermore, the endoscope device 4, the image management device 6, the information processing device 7, and the display device 5 may be communicatively connected via the Internet to the estimation device 1b. In other words, the estimation device 1b may be a device that applies cloud computing, or may be a device provided within a specific medical facility or company.

[0096] In the endoscope system 100c, the estimation device 1b acquires endoscopic images 711 captured and managed at medical facilities H1 to Hn. In this case, the estimation device 1b may acquire the endoscopic images 711 together with a facility ID unique to each of the medical facilities H1 to Hn and a subject ID unique to each subject.

[0097] The estimation device 1b then transmits position information 712 obtained by analyzing the acquired endoscopic image 711 to each of the medical facilities H1 to Hn that are the providers of the endoscopic image 711. This allows the endoscopic system 100b to be introduced to the medical facilities H1 to Hn all at once.

[0098] [Example of Implementation by Software] The functions of the estimation devices 1, 1a, and 1b (hereinafter referred to as "estimation devices") included in the endoscope systems 100, 100a, 100b, and 100c can be realized by a program that causes a computer to function as the estimation device, and that causes a computer to function as each control block of the estimation device (particularly, each unit included in the control unit 10, 10a, and 10b). Furthermore, the functions of the information processing device 7 included in the endoscope systems 100b and 100c can be realized by a program that causes a computer to function as the information processing device 7, and that causes a computer to function as each control block of the information processing device 7 (particularly, each unit included in the control unit 70).

[0099] In this case, each of the estimation device and the information processing device 7 includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and the storage device.

[0100] The program may be stored non-transitoryly in one or more computer-readable recording media. The recording media may or may not be included in each of the estimation device and the information processing device 7. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0101] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0102] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0103] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure.

[0104] [Summary] An endoscopic system according to aspect 1 of the present disclosure includes an acquisition unit that acquires an endoscopic image captured by an endoscopic device, a detection unit that detects a first position in the endoscopic image corresponding to the position of a tip of a treatment tool protruding from the tip of an insertion section of the endoscopic device, an estimation unit that estimates a second position in the endoscopic image corresponding to a position that is a first distance from the tip of the treatment tool using a trained estimation model, and a display control unit that causes a display device to display a composite image obtained by combining the endoscopic image with positional information including information indicating the first position and information indicating the second position.

[0105] An endoscopic system according to aspect 2 of the present disclosure is in the above-mentioned aspect 1, wherein the estimation model is generated by learning a correspondence between the position of the tip of the treatment tool in the reference image and the position of the mark, using a plurality of reference images taken by the endoscopic device each time the position of the tip of the treatment tool is changed, of a plate-like member fixed to the tip of the treatment tool protruding from the tip of the insertion section of the endoscopic device, the plate-like member bearing at least a mark indicating a position at which the distance from the tip of the treatment tool is the first distance, and the position of the mark, and the estimation unit may output the position of the mark estimated using the learned estimation model as the second position.

[0106] An endoscopic system according to aspect 3 of the present disclosure is, in the above-mentioned aspect 2, the plate-like member may be fixed to the tip of the treatment tool approximately parallel to a plane perpendicular to the protrusion direction of the treatment tool.

[0107] An endoscopic system according to aspect 4 of the present disclosure may be, in aspect 2 or 3 above, wherein the mark is a line or a point indicating a position at the first distance from the tip of the treatment instrument.

[0108] In an endoscopic system according to aspect 5 of the present disclosure, in any one of aspects 1 to 4 above, the display control unit may cause the display device to display the position information as a scale bar having the first position as a first end and showing the second position on a straight line extending from the first end to the second end.

[0109] An endoscopic system according to aspect 6 of the present disclosure is, in the above-mentioned aspect 5, further provided with a reception unit that receives a first operation from a user, and the display control unit may, in response to the reception unit receiving the first operation, display on the display device the scale bar in which the position of the second end has been changed while the position of the first end is fixed.

[0110] In an endoscopic system according to aspect 7 of the present disclosure, in any one of aspects 1 to 6 above, the display control unit may cause the display device to display the composite image including the position information as a point indicating the first position and a curve indicating the second position.

[0111] In an endoscopic system according to aspect 8 of the present disclosure, in any of aspects 1 to 7 above, the estimation unit may estimate a third position in the endoscopic image corresponding to a position at a second distance from the tip of the treatment instrument that is different from the first distance using the learned estimation model, and the display control unit may display on the display device a composite image obtained by combining the endoscopic image with position information including information indicating the first position, information indicating the second position, and information indicating the third position.

[0112] In an endoscopic system according to aspect 9 of the present disclosure, in any one of aspects 1 to 8 above, the detection unit may use a trained detection model to generate a bounding box surrounding the treatment tool in the endoscopic image, and detect the first position based on the coordinates of the bounding box.

[0113] A control method for an endoscopic system according to aspect 10 of the present disclosure is a control method for an endoscopic system executed by one or more computers, and includes an acquisition step of acquiring an endoscopic image captured by an endoscopic device; a detection step of detecting a first position in the endoscopic image corresponding to the position of the tip of a treatment tool protruding from the tip of an insertion portion of the endoscopic device; an estimation step of estimating a second position in the endoscopic image corresponding to a position that is a first distance from the tip of the treatment tool using a trained estimation model; and a display control step of displaying on a display device a composite image obtained by combining the endoscopic image with positional information including information indicating the first position and information indicating the second position.

[0114] A control program according to aspect 11 of the present disclosure is a control program for causing a computer to function as an endoscopic system described in any one of aspects 1 to 9 above, and is a control program for causing a computer to function as at least the acquisition unit, the estimation unit, and the display control unit.

[0115] The accuracy of a doctor (hereinafter referred to as the subject) with experience using an endoscope when estimating the size of an object using an endoscopic system according to one aspect of the present disclosure (hereinafter referred to as the endoscopic system 100) and the accuracy of a doctor (hereinafter referred to as the subject) when estimating the size of an object using a conventional endoscopic system were investigated.

[0116] When using the endoscope system 100, the subjects estimated the size of the object by utilizing the scale in the endoscopic image (referred to as "virtual scale" in Table 1). On the other hand, when using the conventional endoscope system used in this example, the subjects estimated the size of the object visually.

[0117] The subjects were 10 doctors, five of whom were endoscopic residents and the remaining five were expert endoscopists. In this example, an endoscopic resident is a doctor with less than five years of experience performing endoscopic examinations, and an expert endoscopist is a doctor with more than ten years of experience performing endoscopic examinations.

[0118] The subjects used were silicon models imitating six different sizes of polyps. The silicon models used were "Ip 10 mm," "IIa 20 mm," "Isp 5 mm," "Isp 10 mm," "Is 7 mm," and "IIa 15 mm," which are parts of the gastrointestinal endoscopy simulator "EASY" manufactured by Tanac Co., Ltd. Here, the numbers with "mm" in the names indicate the size (in mm) of the silicon models.

[0119] Each silicone model was attached at equal intervals to the inner wall of the large intestine model. The large intestine model used was a "Colonoscopy Training Model" manufactured by Kyoto Scientific Co., Ltd.

[0120] Each subject first observed the large intestine model using a conventional endoscope system, estimated the size of the silicone model attached to the large intestine model, and reported the results. Each subject also observed the large intestine model using the endoscope system 100, estimated the size of the silicone model attached to the large intestine model, and reported the results.

[0121] For each silicone model, the average value of the numerical values ​​indicating the size reported by the 10 subjects was calculated, and this was used as the "measured value (average)" for each silicone model. In addition, the "error" between the "measured value (average)" calculated for each silicone model and the actual size of each silicone model was calculated. Furthermore, for each silicone model, the "accuracy" was calculated using the following formula: "Accuracy" = {(measured value (average) - |error|) / measured value (average)} * 100 In addition, the average value of the "accuracy" calculated for each silicone model was calculated, and this was used as the "overall accuracy" of each endoscopic system.

[0122] (Results) Table 1 shows the "actual measured value (average)" of each silicon model, the "error" and "precision" in the measurement of each silicon model, and the "overall accuracy" when the size was estimated using the endoscope system 100.

[0123] Table 2 shows the "actual measured value (average)" of each silicon model, the "error" and "precision" in the measurement of each silicon model, and the "overall accuracy" when the size was estimated using a conventional endoscope system.

[0124] For example, when the size of the silicon model "Ip 10 mm" was estimated using the endoscope system 100, the "measured value (average)" was 10.1 (mm), the "error" was +0.1 (mm), and the "accuracy" was 99.0 (%) (Table 1). On the other hand, when a conventional endoscope system was used, the "measured value (average)" was 6.1 (mm), the "error" was -3.9 (mm), and the accuracy was 61.0 (%) (Table 2).

[0125] For example, when the size of the silicon model "IIa 20 mm" was estimated using the endoscope system 100, the "measured value (average)" was 18.1 (mm), the "error" was -1.9 (mm), and the "accuracy" was 90.5 (%) (Table 1). On the other hand, when the conventional endoscope system was used, the "measured value (average)" was 9.9 (mm), the "error" was -10.1 (mm), and the accuracy was 49.5 (%) (Table 2).

[0126] For example, when the size of the silicon model "Isp 5 mm" was estimated using the endoscope system 100, the "measured value (average)" was 4.9 (mm), the "error" was -0.1 (mm), and the "accuracy" was 98.0 (%) (Table 1). On the other hand, when a conventional endoscope system was used, the "measured value (average)" was 3.5 (mm), the "error" was -1.5 (mm), and the accuracy was 70.0 (%) (Table 2).

[0127] For example, when the size of the silicon model "Isp 10 mm" was estimated using the endoscope system 100, the "measured value (average)" was 10.4 (mm), the "error" was +0.4 (mm), and the "accuracy" was 96.0 (%) (Table 1). On the other hand, when a conventional endoscope system was used, the "measured value (average)" was 7.6 (mm), the "error" was -2.4 (mm), and the accuracy was 76.0 (%) (Table 2).

[0128] For example, when the size of the silicon model "Is 7 mm" was estimated using the endoscope system 100, the "measured value (average)" was 7.6 (mm), the "error" was +0.6 (mm), and the "accuracy" was 91.4 (%) (Table 1). On the other hand, when the conventional endoscope system was used, the "measured value (average)" was 5.2 (mm), the "error" was -1.8 (mm), and the accuracy was 74.3 (%) (Table 2).

[0129] For example, when the size of the silicon model "IIa 15 mm" was estimated using the endoscope system 100, the "measured value (average)" was 14.8 (mm), the "error" was -0.2 (mm), and the "accuracy" was 98.7 (%) (Table 1). On the other hand, when the conventional endoscope system was used, the "measured value (average)" was 11.2 (mm), the "error" was -3.8 (mm), and the accuracy was 74.7 (%) (Table 2).

[0130] When the size was estimated using the endoscope system 100, the "overall accuracy" was 94.6% (Table 1), and when the size was estimated using a conventional endoscope system, the "overall accuracy" was 67.6% (Table 2).

[0131] (Discussion) It was found that, regardless of which silicon model was used, the results of size estimation using the endoscope system 100 had higher "precision" and "overall accuracy" than the results of size estimation using a conventional endoscope system. This demonstrates that the endoscope system 100 can measure the size of an object during endoscopic examination with higher precision than a conventional endoscope system.

[0132] When estimating size using a conventional endoscopic system, it was found that experienced endoscopists tended to have higher "accuracy" than endoscopic trainees. On the other hand, when estimating size using the endoscopic system 100, no significant difference in "accuracy" was found between endoscopic trainees and experienced endoscopists (not shown). This indicates that the endoscopic system 100 can assist in estimating the size of an object during endoscopic examination, regardless of the doctor's level of proficiency in endoscopic examination.

[0133] DESCRIPTION OF SYMBOLS 1, 1a, 1b Estimation device 4 Endoscope device 5 Display device 12 Reception unit 41 Treatment tool 42 Insertion unit 100, 100a, 100b, 100c Endoscope system 101 Acquisition unit 102 Detection unit 103 Estimation unit 105, 703 Display control unit 112 Detection model 113 Estimation model L5, L10, L15, L20 Marks R1, R2 Plate-shaped member S1 Acquisition step S2 Detection step S3 Estimation step S4 Display control step

Claims

1. An endoscopic system comprising: an acquisition unit that acquires an endoscopic image captured by an endoscopic device; a detection unit that detects a first position in the endoscopic image corresponding to the position of a tip of a treatment tool protruding from the tip of an insertion unit of the endoscopic device; an estimation unit that estimates a second position in the endoscopic image corresponding to a position that is a first distance from the tip of the treatment tool using a trained estimation model; and a display control unit that displays on a display device a composite image obtained by combining the endoscopic image with positional information including information indicating the first position and information indicating the second position.

2. The endoscopic system described in claim 1, wherein the estimation model is generated by learning the correspondence between the position of the tip of the treatment tool in the reference image and the position of the mark, using a plurality of reference images taken by the endoscopic device each time the position of the tip of the treatment tool is changed, of a plate-like member fixed to the tip of the treatment tool protruding from the tip of the insertion part of the endoscopic device, the plate-like member having at least a mark indicating a position at which the distance from the tip of the treatment tool is the first distance, and the estimation unit outputs the position of the mark estimated using the learned estimation model as the second position.

3. The endoscope system according to claim 2, wherein the plate-like member is fixed to the distal end of the treatment tool substantially parallel to a plane perpendicular to the direction of protrusion of the treatment tool.

4. The endoscope system according to claim 2, wherein the mark is a line or a dot indicating the position at the first distance from the distal end of the treatment tool.

5. The endoscopic system of claim 1, wherein the display control unit causes the display device to display the position information as a scale bar having the first position as its first end and showing the second position on a straight line extending from the first end to the second end.

6. An endoscopic system as described in claim 5, further comprising a reception unit that receives a first operation from a user, and wherein the display control unit, in response to the reception unit receiving the first operation, causes the display device to display the scale bar in which the position of the second end has been changed while the position of the first end is fixed.

7. The endoscope system according to claim 1, wherein the display control unit causes the display device to display the composite image including the position information as a point indicating the first position and a curve indicating the second position.

8. The endoscopic system of claim 1, wherein the estimation unit estimates a third position in the endoscopic image corresponding to a position at a second distance from the tip of the treatment tool that is different from the first distance using the trained estimation model, and the display control unit causes the display device to display the composite image obtained by combining the endoscopic image with position information including information indicating the first position, information indicating the second position, and information indicating the third position.

9. The endoscopic system of claim 1, wherein the detection unit uses a trained detection model to generate a bounding box surrounding the treatment tool in the endoscopic image, and detects the first position based on the coordinates of the bounding box.

10. A control method for an endoscopic system executed by one or more computers, comprising: an acquisition step of acquiring an endoscopic image captured by an endoscopic device; a detection step of detecting a first position in the endoscopic image corresponding to the position of a tip of a treatment tool protruding from the tip of an insertion section of the endoscopic device; an estimation step of estimating a second position in the endoscopic image corresponding to a position that is a first distance from the tip of the treatment tool using a trained estimation model; and a display control step of displaying on a display device a composite image obtained by combining the endoscopic image with positional information including information indicating the first position and information indicating the second position.

11. A control program for causing a computer to function as the endoscope system of claim 1, the control program causing the computer to function as the acquisition unit, the estimation unit, and the display control unit.

Images