Endoscopic inspection assistance method, endoscopic inspection assistance device, and program

Abstract

This endoscopic inspection assistance method comprises: calculating, on the basis of an inspection time previously set in inspection of an organ using an endoscope and an elapsed time that has elapsed from the start of the inspection, a remaining time until the inspection ends; acquiring current position information indicating a position in the organ that is currently being inspected; and generating and outputting, by using the remaining time and the current position information, an inspection time table including a scheduled inspection time for each of a plurality of division sections into which positions in the organ are divided.

Description

Endoscopic examination support method, endoscopy support device, and program

[0001] The present invention relates to an endoscopic examination support method, an endoscopic examination support device, and a program.

[0002] Japanese Patent Laid-Open No. 2009-102663 discloses a system that detects a region of interest including a target of interest from a medical image and prevents the target of interest from being overlooked. If it is determined that the target of interest has been overlooked, the system notifies the user of the occurrence of the oversight.

[0003] International Publication No. 2019 / 078273

[0004] However, if the examination of interest is re-performed following notification of a missed examination, the examination time from start to finish will be longer. Other procedures and changes in circumstances may also increase the time.

[0005] An object of the present invention is to provide an endoscopic examination support method, an endoscopic examination support device, and a program that can output an examination timetable relating to appropriate times for each section of a scheduled examination.

[0006] According to a first aspect of the present invention, an endoscopic examination support method includes the following: calculating the remaining time until the end of an examination of an organ using an endoscope based on a preset examination time and the elapsed time since the start of the examination; acquiring current position information indicating the position in the organ currently being examined; and using the remaining time and the current position information, generating and outputting an examination timetable including the scheduled examination time for each of a plurality of divided sections obtained by dividing positions in the organ.

[0007] According to a second aspect of the present invention, the endoscopic examination support method of the first aspect may include the following: by using the current location information, of the multiple divided sections into which the organ is divided, divided sections in which the examination has been completed are classified as completed examination divided sections, and divided sections in which the examination has not been completed are classified as planned examination divided sections, and the examination timetable including the planned examination time for each planned examination divided section is generated and output according to the remaining time.

[0008] According to a third aspect of the present invention, the endoscopic examination support method of the second aspect may include the following: determining whether an oversight has occurred in the examination in the examination end segment; and, when generating and outputting the examination timetable, reclassifying the examination end segment in which it has been determined that an oversight has occurred into the scheduled examination segment.

[0009] According to a fourth aspect of the present invention, the endoscopic examination support method of the second aspect may include the following: when the endoscope is inserted into the lumen of the organ, a lesion is detected in each of the plurality of divided sections; information about the detected lesion is stored in a memory unit; when the endoscope is withdrawn from the lumen, the information stored in the memory unit is used to determine whether or not a lesion has been overlooked during the examination in the examination end divided section; and when generating and outputting the examination timetable, the examination end divided section including the lesion determined to have been overlooked is reclassified into the planned examination divided section.

[0010] According to a fifth aspect of the present invention, the endoscopic examination support method of the second aspect may include the following: referencing a segmented section of interest that is the examination end segmented section corresponding to the current position information in the examination timetable, and outputting instruction information to instruct the examiner to reexamine the segmented section of interest if the time required for the examination in the segmented section of interest is shorter than the scheduled examination time for the segmented section of interest.

[0011] According to a sixth aspect of the present invention, the method for supporting endoscopic examination in the fifth aspect may include the following: reclassifying the target divided section into the scheduled examination divided sections, and updating and outputting the examination timetable in consideration of the time required for the reexamination.

[0012] According to a seventh aspect of the present invention, the endoscopic examination support method of the second aspect may include the following: acquiring a reference timetable including the scheduled examination times for each of the plurality of divided sections from a storage unit; performing a process on the reference timetable to update the scheduled examination times for the scheduled examination divided sections using the remaining time and the current position information; and outputting the updated reference timetable as the examination timetable.

[0013] According to an eighth aspect of the present invention, the endoscopic examination support method of the seventh aspect may include the following: determining the state of the image acquired by the endoscope in the divided section based on an inference model obtained through machine learning using the image acquired by the endoscope and annotations indicating insufficient observation of the image as training data; generating the reference timetable according to the state determined based on the inference model; and recording the generated reference timetable in the storage unit.

[0014] According to a ninth aspect of the present invention, the endoscopic examination support method of the seventh aspect may include the following: calculating a first sum of the scheduled examination times in the scheduled examination segments among the scheduled examination times included in the reference timetable before being updated; calculating a second sum of the scheduled examination times in the scheduled examination segments among the scheduled examination times included in the reference timetable after being updated; generating speed information indicating the speed of the examination based on a result of comparing the first sum with the second sum; and outputting the examination timetable and the speed information to a display unit.

[0015] According to a tenth aspect of the present invention, the endoscopic examination support method of the first aspect may include: calculating the planned examination time for each planned divided section in accordance with an adapter attached to the tip of the endoscope, and generating and outputting the examination time table including the calculated planned examination time.

[0016] According to an eleventh aspect of the present invention, in the first aspect, the endoscopic examination support method may include outputting the examination time table and the current position information to a display unit.

[0017] According to a twelfth aspect of the present invention, an endoscopic examination support device includes a calculation unit, an acquisition unit, and an output unit. The calculation unit calculates the remaining time until the end of an endoscopic examination of an organ based on a preset examination time and the elapsed time since the examination began. The acquisition unit acquires current position information indicating the position in the organ currently being examined. The output unit uses the remaining time and the current position information to generate and output an examination timetable including the scheduled examination time for each of a plurality of divided sections obtained by dividing the positions in the organ.

[0018] According to a thirteenth aspect of the present invention, there is provided a program for causing a computer to execute the following: Calculate the remaining time until the end of an endoscopic examination of an organ based on a preset examination time and the elapsed time since the examination began; Obtain current position information indicating the position in the organ currently being examined; Use the remaining time and the current position information to generate and output an examination timetable including the scheduled examination time for each of a plurality of divided sections obtained by dividing positions in the organ.

[0019] According to a fourteenth aspect of the present invention, an endoscopic examination support method includes the following: calculating the remaining time until the end of an examination of an organ using an endoscope based on a preset examination time and the elapsed time since the start of the examination; acquiring current position information indicating the position in the organ currently being examined; and generating and outputting an examination timetable that can simultaneously display the remaining time and the current position information.

[0020] According to the 15th aspect of the present invention, in the 14th aspect, the endoscopic examination support method may include displaying the remaining time and the time allocation for examination of each part of the organ in at least the part displaying the remaining time based on the examination timetable.

[0021] According to a sixteenth aspect of the present invention, in the fifteenth aspect, the time allocation may be determined according to the difference in surface area between the test target parts of the organ to be tested, statistics of lesion occurrence between the test target parts, or the difficulty of the test based on the shape of each test target part.

[0022] According to the 17th aspect of the present invention, in the 14th aspect, the endoscopic examination support method may include, when an examination delay exceeding a specific time width with respect to the time allocated for examination of each part of the organ is detected, resetting the time allocation for examination of each part of the organ based on the remaining time.

[0023] According to an 18th aspect of the present invention, in the 15th aspect, the endoscopic examination support method may include the following: By using the current position information, of the multiple divided sections into which the organ is divided, divided sections into which the examination has been completed are classified as end-of-examination divided sections, and divided sections into which the examination has not been completed are classified as planned examination divided sections. When the endoscope is inserted into the organ, a lesion is detected in each of the multiple divided sections. Information about the detected lesion is saved in a memory unit. When the endoscope is withdrawn from the organ, the information saved in the memory unit is used to determine whether or not a lesion was overlooked during the examination in the end-of-examination divided section. When the examination timetable is generated and output, the end-of-examination divided section including the lesion determined to have been overlooked is reclassified into the planned examination divided section, and the remaining time and the time allocation for the examination in the planned examination divided section are displayed.

[0024] According to a 19th aspect of the present invention, in the 15th aspect, the endoscopic examination support method may include the following: by using the current position information, of the multiple divided sections into which the organ is divided, divided sections into which examination has been completed are classified as completed examination divided sections, and divided sections into which examination has not been completed are classified as planned examination divided sections; when the endoscope is inserted into the organ, image features in each of the multiple divided sections are detected; information on the detected image features is stored in a memory unit; and by using the information on the image features for each divided section stored in the memory unit, the allocation of examination time for each part of the organ is switched.

[0025] According to the 20th aspect of the present invention, in the 15th aspect, the endoscopic examination support method may include referring to the examination timetable and, if the time required for examination in a specific divided section among the multiple divided sections into which the organ is divided is short, switching the time allocation for examination of each part of the organ.

[0026] According to the above aspects, the endoscopic examination support method, the endoscopic examination support device, and the program can output an examination timetable relating to appropriate times for each section of the scheduled examination in order to make up for the examination.

[0027] 1 is a block diagram showing an example of the configuration of an endoscopic system according to a first embodiment of the present invention. FIG. 2 is a flowchart showing an example of a processing procedure executed by an endoscopic examination support device according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of a reference timetable in the first embodiment of the present invention. FIG. 4 is a diagram showing an example of a screen of a display unit in the first embodiment of the present invention. FIG. 5 is a diagram showing a method for generating an inference model in the first embodiment of the present invention. FIG. 6 is a diagram showing a state of an endoscope to which an adapter is attached in the second embodiment of the present invention. FIG. 7 is a flowchart showing an example of a processing procedure executed by an endoscopic examination support device according to the second embodiment of the present invention. FIG. 8 is a diagram showing an example of a screen of a display unit in the second embodiment of the present invention. FIG. 9 is a diagram showing a state of an endoscope to which an adapter is attached in the second embodiment of the present invention. FIG. 10 is a diagram showing an example of a screen of a display unit in the second embodiment of the present invention. FIG. 11 is a diagram showing an example of a treatment being performed on a lesion area in the second embodiment of the present invention. FIG. 12 is a diagram showing an example of a screen of a display unit in the second embodiment of the present invention.

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. An example of an endoscope system including an endoscopic examination support device will be described below. In the following description, "plurality of XX" means two or more XXs.

[0029] 1 shows an example of the configuration of an endoscope system 1 according to a first embodiment of the present invention. The endoscope system 1 shown in FIG. 1 includes an endoscope 10 and an endoscopic examination support device 20.

[0030] The endoscope 10 is inserted into an organ of a subject. The organ is often located within a lumen. For example, the organ is a digestive tract such as the large intestine or stomach. The endoscope 10 has an imaging unit 11. The imaging unit 11 is disposed at the tip of the endoscope 10. The imaging unit 11 is an image sensor that generates multiple images at multiple positions within the digestive tract of the subject. The imaging unit 11 outputs the multiple images to the endoscopic examination support device 20. The examiner (doctor) inserts the endoscope 10 into the digestive tract of the subject and advances the tip of the endoscope 10 (hereinafter, the tip may be omitted) to a predetermined position. The examiner then performs the examination while slowly pulling back the endoscope 10.

[0031] The endoscopic examination support device 20 has an image acquisition unit 21, an information reception unit 22, a timetable generation unit 23, a memory unit 24, an area detection unit 25, an examination position determination unit 26, a time calculation unit 27, a timetable update unit 28, an examination support information generation unit 29, and a display unit 30.

[0032] The image acquisition unit 21 sequentially acquires the plurality of images output from the imaging unit 11. The image acquisition unit 21 outputs the acquired plurality of images to the storage unit 24.

[0033] The information accepting unit 22 accepts information specified by the examiner. For example, the information accepting unit 22 accepts examination condition information indicating examination conditions, and outputs the examination condition information to the timetable generating unit 23. For example, the examination condition information includes the sex of the subject, the age of the subject, the examination target, and anesthesia information. The examination target indicates the type of digestive tract (such as the large intestine or stomach). The anesthesia information indicates whether anesthesia is used during the examination.

[0034] The timetable generation unit 23 generates a standard timetable based on the examination condition information. The standard timetable includes a plurality of divided sections into which positions in an organ are divided, and a planned examination time required for the examination in each divided section. The divided sections indicate the parts through which the tip of the endoscope 10 passes during the examination. When the examination target is the large intestine, the parts are the cecum, ascending colon, transverse colon, descending colon, etc. The timetable generation unit 23 outputs the generated standard timetable to the memory unit 24. Details of the standard timetable will be described later.

[0035] The storage unit 24 is a memory. The storage unit 24 stores a plurality of images output from the image acquisition unit 21 and a reference timetable output from the timetable generation unit 23. When the reference timetable is updated by the timetable update unit 28, the storage unit 24 stores the updated reference timetable. The plurality of images stored in the storage unit 24 are sequentially output to the area detection unit 25, the inspection position determination unit 26, and the display unit 30. The reference timetable stored in the storage unit 24 is output to the time calculation unit 27 and the timetable update unit 28.

[0036] The region detection unit 25 detects characteristic regions in each of multiple sites in the digestive tract based on the multiple images output from the storage unit 24. An example in which the region detection unit 25 detects a missed region as a characteristic region will be described below. A region in the digestive tract may not appear in the image, or the region may have poor visibility in the image. Alternatively, a region in the digestive tract may appear in the image, but the examiner may not perform detailed observation or treatment on that region. The region detection unit 25 detects these regions as missed regions. When a missed region is detected, the region detection unit 25 outputs missed region information indicating the occurrence of the missed region to the timetable update unit 28.

[0037] The examination position determination unit 26 determines the position currently being examined based on the multiple images output from the storage unit 24, and generates current position information indicating that position. The position indicates the position in the digestive tract currently being imaged by the imaging unit 11.

[0038] The "position under examination" may be in units of organ name (part name) or a subdivided position within an organ. For example, the organ name may be the esophagus or stomach in the case of the upper digestive tract, and the rectum or transverse colon in the case of the lower digestive tract. For example, a subdivided position may be a position about 3 cm after passing the sigmoid colon and entering the transverse colon. The "position under examination" may be a position within each lumen, a name within the lumen, or a classified position within the lumen. The "position under examination" may be a combination of the name of the lumen and the luminal position, depth, length, or distance within the lumen. A direction perpendicular to the continuous direction (axial direction) of the lumen may also be used as needed.

[0039] In the case of a pouch-shaped organ such as the stomach, the "position during examination" may be the name of a region or part, such as the cardia, fundus, body, pyloric antrum, or pylorus, from the entrance, or may be information expressing any position of that region in more detail, such as coordinates. Such position information may be detected from image features, such as tissue or blood vessels, specific to that region. The position may be determined by using changes in image features accompanying changes in image frames acquired continuously during insertion of the endoscope 10, or by using patterns of blood vessels or tissues. The endoscopic examination support device 20 may detect the speed (cm / sec) of the endoscope 10 during insertion and convert the insertion position using time information. However, without detecting this speed, the endoscopic examination support device 20 may use, as a constant, the average insertion speed when a typical physician inserts an endoscope. The endoscopic examination support device 20 may determine the shape of the lumen using three-dimensional (3D) reconstruction using image information, etc., and estimate the position during examination based on the results of this determination.

[0040] In the following example, the position indicates a specific region within a simplified digestive tract. If the endoscope 10 has a position sensor at its tip, the examination position determination unit 26 may determine the region currently being examined based on information output from the position sensor. For example, one method is to detect the region using magnetism. Alternatively, the region may be detected using other medical equipment such as a CT or MRI. The examination position determination unit 26 generates region information (current position information) indicating the determined region and outputs the region information to the timetable update unit 28.

[0041] Furthermore, the expression "position in an organ" may refer to the name of a part within an organ, such as the cardia, fundus, body, pyloric antrum, or pylorus, but "position in an organ" may also be used to mean the difference between organs of different classifications, such as the esophagus and the stomach.

[0042] The time calculation unit 27 calculates the remaining time based on a preset examination time and the time that has passed since the start of the examination (elapsed time). The remaining time is the time required for the examination until it is completed. The examination time is the estimated time required for the entire examination. The time calculation unit 27 calculates the remaining time by subtracting the elapsed time from the examination time. The time calculation unit 27 outputs the remaining time to the timetable update unit 28.

[0043] The timetable update unit 28 acquires the reference timetable from the storage unit 24. When the missed program information is output from the area detection unit 25, the timetable update unit 28 updates the reference timetable based on the part information output from the examination position determination unit 26 and the remaining time output from the time calculation unit 27.

[0044] If a missed region occurs in a specific region, it is assumed that the examiner will return the tip of the endoscope 10 from the position during the examination (the current position in the organ, i.e., the region of the organ where the tip of the endoscope 10 is currently located) to a position where the missed region can be observed, and perform the examination again at that position. Since the examination will be repeated, the time required for the examination at that region will increase. In order to complete the examination before the effects of the anesthesia wear off, the entire examination time must be kept within the specified time. However, if the time required for the examination at a specific region increases, there is a possibility that the entire examination time will not fall within the specified time.

[0045] Such revisions or necessary corrective measures are implemented. Examples of corrective measures include removing residue, removing air bubbles, injecting water, supplying water, removing polyps, or stopping bleeding. The examination may be delayed due to the subject's complaint or a change in a vital sign being detected during the procedure. However, even if a delay occurs, the remaining areas that require examination must be examined without fail, so it is important to be aware of timetable management. In the present application, the examiner can perform the examination with this awareness. The timetable update unit 28 adjusts the scheduled examination times for areas that have not yet been examined so that the overall examination time falls within the specified time. In this way, the timetable update unit 28 uses the remaining time and area information to update the scheduled examination times for areas that have not yet been examined in the reference timetable.

[0046] When missed object information is output from the area detection unit 25, the timetable update unit 28 outputs the updated reference timetable (actual examination timetable) to the storage unit 24 and the examination support information generation unit 29. When missed object information is not output from the area detection unit 25, the timetable update unit 28 outputs the unupdated reference timetable to the examination support information generation unit 29. In addition, the timetable update unit 28 outputs the missed object information output from the area detection unit 25 and the part information output from the examination position determination unit 26 to the examination support information generation unit 29.

[0047] The examination support information generation unit 29 generates examination support information including the reference timetable and site information output from the timetable update unit 28. The examination support information generation unit 29 also determines the status of the examination at each site. If it is determined that the examination at a specific site is not complete despite the tip of the endoscope 10 having passed through that site, the examination support information generation unit 29 generates examination support information including instruction information for instructing the examiner to reexamine that site. For example, if missed area information is output from the timetable update unit 28, the examination support information generation unit 29 generates examination support information including a message for urging the examiner to reexamine the site where the missed area occurred. The examination support information generation unit 29 outputs the examination support information to the display unit 30.

[0048] The display unit 30 is a liquid crystal monitor or the like. The display unit 30 sequentially displays a plurality of images output from the storage unit 24. The display unit 30 also displays the examination assistance information output from the examination assistance information generation unit 29 together with the images.

[0049] At least one of the image acquisition unit 21, the information reception unit 22, the timetable generation unit 23, the area detection unit 25, the examination position determination unit 26, the time calculation unit 27, the timetable update unit 28, and the examination support information generation unit 29 may be configured by a processor such as a CPU (Central Processing Unit).

[0050] A computer may load a program and execute the loaded program. The program includes instructions that define the operation of at least one of the image acquisition unit 21, the information reception unit 22, the timetable generation unit 23, the area detection unit 25, the inspection position determination unit 26, the time calculation unit 27, the timetable update unit 28, and the inspection support information generation unit 29. In other words, at least one function of the image acquisition unit 21, the information reception unit 22, the timetable generation unit 23, the area detection unit 25, the inspection position determination unit 26, the time calculation unit 27, the timetable update unit 28, and the inspection support information generation unit 29 may be realized by software.

[0051] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope system 1 via a transmission medium or by transmission waves in the transmission medium. The "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by combining a program already recorded on the computer with the difference program.

[0052] These processors and recording media do not necessarily have to be contained in a single device, but may be configured as multiple devices with distributed functions working together. Also, these processors and recording media may be located on a cloud (network).

[0053] An example of the operation of the endoscopic examination support device 20 will be described with reference to Fig. 2. Fig. 2 shows an example of the procedure of the process executed by the endoscopic examination support device 20.

[0054] (Step S100) The information receiving unit 22 receives examination condition information indicating examination conditions, and outputs the examination condition information to the timetable generating unit 23. As described above, the examination condition information includes the sex of the subject, the age of the subject, the subject of the examination, and anesthesia information.

[0055] (Step S101 ) The timetable generating unit 23 generates a reference timetable based on the inspection condition information, and outputs the generated reference timetable to the storage unit 24 .

[0056] Fig. 3 shows an example of a reference timetable. In the example shown in Fig. 3, the object of examination is the large intestine. The examiner inserts the endoscope 10 into the large intestine and advances the tip of the endoscope 10 to the cecum. After the tip of the endoscope 10 reaches the cecum, the examiner performs the examination while slowly retracting the endoscope 10. The reference timetable TB10 shown in Fig. 3 includes the planned time required for inserting the endoscope 10 and the planned time required for examination of each part of the large intestine.

[0057] If anesthesia is used, the examination must be completed before the effects of the anesthesia wear off. Therefore, a standard examination time (standard examination time from the beginning to the end of the examination) is preset based on the duration of the anesthesia effect. Anesthesia is not required. The "anesthesia time" in the figure is a guideline, as the use of anesthesia may be withheld or the amount of anesthesia may be increased or decreased based on the patient's wishes or constitution, or the doctor's judgment. In addition, a standard planned time (insertion time) required for inserting the endoscope 10 is preset. The timetable generation unit 23 determines the planned time for each site so that the total planned time for each site is equal to or less than the standard examination time excluding the insertion time. The timetable generation unit 23 determines the planned time for each site according to the priority of each site of the large intestine. The priority of each site is preset based on the probability of lesion occurrence in each site and the length of each site. The priority (observation priority) of each site and the planned time for each site are shown in Figure 3. The timetable generation unit 23 generates a standard timetable including the insertion time and the planned time for each site.

[0058] When examination assistance information including this reference timetable is output to the display unit 30, the display unit 30 has the characteristic of displaying the remaining time and the time allocation for examination of each part of the organ based on the reference timetable in at least the part that displays the remaining time. The time allocation indicates the time previously allocated for examination of each part of the organ. This reference timetable serves as a reference for doctors or medical professionals when performing an endoscopic examination to manage the time required for each part to be examined. The time allocation for the part other than the remaining time indicates the time previously allocated for examination of parts that have already been examined. This time allocation is not useful for future reference and does not need to be displayed, but it may be used in reports, etc., together with the actual examination time.

[0059] The endoscopic examination support method, endoscopic examination support, and program calculate the remaining time until the end of an endoscopic examination of an organ based on a preset examination time and the elapsed time since the examination began, obtain current position information indicating the position currently being examined in each organ specified in the examination menu, and generate and output an examination timetable that can simultaneously display the remaining time and the current position information. The preset examination time may be configured to be adjustable depending on the subject's profile (height, weight, sex, race, age, constitution, etc.) or the duration of the anesthesia effect. The examination menu may be determined based on a health checkup or tests for a specific disease, or the doctor may decide the examination menu depending on the situation.

[0060] By managing the time spent during an examination, it is possible to ensure that there are no gaps in time allocation and that the examination is efficient and does not disrupt the schedule. Even if the examination of a particular area takes too long, the examiner can check the planned time allocation for the rest of the examination and calmly make up for it. Such time allocation guidelines are presented based on the doctor's experience, etc. The current position is displayed as it changes moment by moment as the tip of the endoscope 10 moves, making it easy for the examiner to visualize the pace of the examination. Therefore, even if a procedure is required during the examination and takes time, the examiner can look at the time allocation guide and determine at what pace they should proceed with the subsequent examination.

[0061] The endoscopic examination support device 20 determines this time allocation based on the difference in surface area of ​​each examination target region in the organ being examined, the statistics of lesion occurrence in each examination target region, or the difficulty of the examination based on the shape of each examination target region. The predetermined examination time for an organ examination is also determined based on a similar concept. If the examination is progressing smoothly within the scheduled time without any particular oversights, the initial time allocation may be maintained. When an expected time exceeding the initial time allocation is detected, i.e., when an examination delay exceeding a specific time range is detected with respect to the time allocated for examination of each part of the organ, the endoscopic examination support device 20 resets the time allocation for examination of each part of the organ based on the remaining time until the examination is completed.

[0062] In the case of gastrointestinal endoscopy, organ examinations often involve examinations of the organ's lumen, and the divided regions are often segments along the length of the lumen. Of course, even a pouch-like organ such as the stomach may be considered a lumen with a complex shape. Such a reference timetable is also useful for determining whether the examination of each region was performed within the appropriate time. Therefore, the endoscopic examination support device 20 may refer to this reference timetable and, if the time required for examination of a particular region is short, determine that the expected time allocation has not been met and change the time allocation for examination of each region of the organ. This is because the region that required a short examination time may require re-examination or a review of the entire procedure, but the examination time is limited. Although the reference timetable is a standard for the examination timetable, the standard may be changed depending on the situation, and this may also be called an examination timetable.

[0063] Each part in the reference timetable and the actual examination timetable is classified as either an examination-completed part (examination-completed division section), which is a part (division section) where the examination has been completed, or a planned examination part (examination-scheduled division section), which is a part (division section) where the examination has not been completed. A planned examination part indicates a part where the examination has not started or a part where the examination is being performed and not completed. All parts in the reference timetable are planned examination parts. The reference timetable and the actual examination timetable include part type information that indicates the completed examination part or the planned examination part.

[0064] The length of each body part or the probability of lesion occurrence in each body part may differ depending on the gender and age of the subject. The timetable generating unit 23 may change the scheduled time for each body part in the reference timetable depending on the gender and age included in the examination condition information.

[0065] The examination condition information may include the test results of the subject's previous examination. There is a possibility that a lesion will be detected again in a region where a lesion was previously detected. Since the time required to examine that region may be longer, the timetable generating unit 23 may change the scheduled time for each region in the reference timetable according to the previous examination results.

[0066] (Step S102) When the subject is anesthetized and the examination is started, step S103, which will be described later, is executed. The clock in the endoscopic examination support device 20 starts measuring time and generates time information indicating the measured time. If the examination has not started, the endoscopic examination support device 20 waits until the examination is started.

[0067] When the examination is started, the examiner may notify the start of the examination to the endoscopic examination support device 20 by operating a button or the like. When the start of the examination is notified by the examiner, the endoscopic examination support device 20 may execute step S103.

[0068] (Step S103) The image acquisition unit 21 acquires the image output from the imaging unit 11. The image acquisition unit 21 outputs the image to the storage unit 24. After being output to the storage unit 24, the image output from the image acquisition unit 21 is output to the area detection unit 25, the inspection position determination unit 26, and the display unit 30.

[0069] (Step S104) The examination position determination unit 26 determines the region under examination based on the image output from the storage unit 24 and generates region information indicating the region. If the tip or the like of the endoscope 10 has a position sensor, the examination position determination unit 26 may determine the region under examination based on information output from the position sensor. The examination position determination unit 26 outputs the region information to the timetable update unit 28.

[0070] (Step S105) The timetable updating unit 28 acquires the reference timetable from the storage unit 24 and outputs the reference timetable to the examination support information generating unit 29. When step S113 or step S115 described below is executed, the timetable updating unit 28 acquires from the storage unit 24 the reference timetable that was updated in step S113 or step S115 and stored in the storage unit 24, and outputs the reference timetable to the examination support information generating unit 29. The timetable updating unit 28 also outputs the region information output from the examination position determining unit 26 to the examination support information generating unit 29. The examination support information generating unit 29 generates examination support information including the reference timetable and region information output from the timetable updating unit 28, and outputs the examination support information to the display unit 30.

[0071] (Step S106) The display unit 30 displays the image output from the storage unit 24 and the examination assistance information output from the examination assistance information generation unit 29. Examples of the examination assistance information displayed on the display unit 30 will be described later.

[0072] (Step S107) If the region determined by the examination position determination unit 26 has changed, step S108, which will be described later, is executed. In other words, if the region under examination has changed, step S108 is executed. If the region under examination has not changed, step S116, which will be described later, is executed. For example, while an examination is being performed on the cecum, the region under examination does not change. When the tip of the endoscope 10 passes through the cecum and enters the ascending colon, the region under examination changes. In other words, the examination of the cecum ends and the examination of the ascending colon begins. At this time, step S108 is executed.

[0073] (Step S108) The time calculation unit 27 acquires time information from the clock in the endoscopic examination support device 20 and calculates the time that has passed since the examination started (elapsed time). The time calculation unit 27 also calculates the time required to examine the region for which the examination has been completed. Immediately after the examination of the first region is completed, the time required to examine that region is the same as the elapsed time. Immediately after the examination of the Nth region is completed, the time required to examine that region is the same as the time obtained by subtracting the total time required to examine the first through (N-1)th regions from the elapsed time. The time calculation unit 27 stores the time required to examine that region in the memory unit 24.

[0074] (Step S109) The time calculation unit 27 obtains the reference timetable from the memory unit 24 and calculates the examination time for the endoscopic examination of a specific organ or health checkup. The examination time is the sum of the insertion time included in the reference timetable, the scheduled time for each part, and other leeway time, etc. The examination start time calculation unit 27 calculates the remaining time by subtracting the elapsed time from the examination time. The time calculation unit 27 also calculates the standard examination time for the remaining parts by calculating the sum of the scheduled times for parts that have not yet been examined. The time calculation unit 27 outputs the remaining time and the standard examination time to the timetable update unit 28.

[0075] (Step S110) The area detection unit 25 detects a missed area in the area inspected before the area currently being inspected, based on the image output from the storage unit 24. This allows the area detection unit 25 to determine whether an oversight occurred in the inspection of the area for which inspection has been completed. The area detection unit 25 outputs oversight information indicating the occurrence of a missed area to the timetable update unit 28.

[0076] For example, the area detection unit 25 uses a technique called Visual SLAM (Simultaneous Localization and Mapping) to estimate the camera position and perform three-dimensional (3D) reconstruction using image information, etc. A 3D model is restored through the 3D reconstruction. Positions on the 3D model and positions on the image are associated with each other.

[0077] If some areas of the digestive tract are not captured in the image or the image is not suitable for 3D reconstruction, some areas of the 3D model will not be restored. The region detection unit 25 detects these areas as missed regions. As will be described later, the region detection unit 25 may detect missed regions by using artificial intelligence (AI).

[0078] (Step S111) Step S112, which will be described later, is executed if the area detection unit 25 detects a missed area. If the area detection unit 25 does not detect a missed area, step S116, which will be described later, is executed.

[0079] (Step S112) The timetable update unit 28 determines whether the remaining time is within 1.1 times the standard examination time calculated in step S109. If the remaining time is within 1.1 times the standard examination time, step S113, which will be described later, is executed. If the remaining time is not within 1.1 times the standard examination time, step S114, which will be described later, is executed. The number 1.1 is an example and is not limited to this.

[0080] (Step S113) If the remaining time is within 1.1 times the standard examination time, it is possible to re-examine the area where the missed area was detected. The timetable update unit 28 obtains the reference timetable from the storage unit 24 and updates the reference timetable based on the area information output from the examination position determination unit 26 and the remaining time output from the time calculation unit 27.

[0081] Details of step S113 will be described. By using the part information, the timetable update unit 28 classifies, among the multiple parts, parts for which inspection has been completed as parts for which inspection has been completed, and classifies parts for which inspection has not been completed as parts for which inspection is scheduled. As described above, the reference timetable includes part type information indicating parts for which inspection has been completed or parts for which inspection is scheduled. The timetable update unit 28 changes the part type information for parts for which inspection has been completed. The changed part type information indicates parts for which inspection has been completed.

[0082] Furthermore, the timetable update unit 28 reclassifies the examination-completed region in which a missed region has been detected into a region scheduled for examination. As a result, the timetable update unit 28 reclassifies the examination-completed region in which it has been determined that a missed region has occurred into a region scheduled for examination. The timetable update unit 28 changes the region type information of the region in which a missed region has been detected. The changed region type information indicates the region scheduled for examination.

[0083] It is assumed that a re-examination of a region in which a missed region has been detected will require the same time as the scheduled time for that region. The timetable update unit 28 calculates the time required to examine the remaining regions scheduled for examination (actual remaining time) by subtracting the time required to re-examine that region from the remaining time. The timetable update unit 28 distributes the actual remaining time to each region scheduled for examination according to the observation priority of each region of the large intestine. In this way, the timetable update unit 28 updates the scheduled time for each region in the standard timetable. The timetable update unit 28 may also distribute the actual remaining time to each region regardless of the observation priority. In this way, the timetable update unit 28 updates the standard timetable taking into account the time required to re-examine a region in which a missed region has been detected. The timetable update unit 28 stores the updated standard timetable (actual examination timetable) in the memory unit 24.

[0084] (Step S114) If the remaining time is not within 1.1 times the standard examination time, the examination assistance information generating unit 29 generates information for instructing the examiner to administer additional anesthesia. The information is included in the examination assistance information generated in step S105.

[0085] (Step S115) If anesthesia is added, the available examination time is extended. Therefore, the timetable update unit 28 increases the actual remaining time. The timetable update unit 28 obtains the reference timetable from the memory unit 24 and distributes the actual remaining time to each part of the large intestine according to the observation priority of each part. As a result, the timetable update unit 28 updates the scheduled time for each part in the reference timetable. The timetable update unit 28 saves the updated reference timetable (actual examination timetable) in the memory unit 24.

[0086] (Step S116) If the test is interrupted or completed, the process shown in Fig. 2 is completed. If the test is not interrupted or completed, step S103 is executed.

[0087] 4 shows an example of the screen of the display unit 30. The display unit 30 has a screen 30A.

[0088] An image IMG10 output from the imaging unit 11 is displayed on the screen 30A. Examination status information ST10 indicating the status of the examination is displayed on the screen 30A. The examination status information ST10 is included in the examination assistance information. The examination status information ST10 indicates a region R10 including a part where the examination has been performed, a region R11 including a part where the examination has not been completed, and a missed region R12.

[0089] Instruction information IN10 and a message MS10 for prompting the examiner to perform a re-examination are displayed on the screen 30A. The instruction information IN10 and the message MS10 are included in the examination support information. The instruction information IN10 intuitively prompts the examiner to return to the missed area and perform a re-examination. The message MS10 prompts the examiner in text to return to the missed area and perform a re-examination.

[0090] The actual examination timetable TB11 is displayed on the screen 30A. The actual examination timetable TB11 is included in the examination support information. The timetable update unit 28 generates the actual examination timetable TB11 by updating the reference timetable TB10 shown in FIG. 3. The actual examination timetable TB11 shows an example in which a missed area occurs in the ascending colon. When the examination of the ascending colon is completed, the remaining time in the reference timetable TB10 is the sum of 0.9 minutes, 0.5 minutes, 3.4 minutes, 3.5 minutes, and 0.5 minutes (8.8 minutes).

[0091] Because a missed region was found in the ascending colon, the timetable update unit 28 subtracts the time required to reexamine the ascending colon from the remaining time. Assume that the same time required to reexamine the ascending colon as the scheduled time for the ascending colon in the standard timetable is required. Because the scheduled time for the ascending colon in the standard timetable is 1.1 minutes, the timetable update unit 28 subtracts 1.1 minutes from 8.8 minutes to obtain the actual remaining time (7.7 minutes). The timetable update unit 28 distributes the actual remaining time (7.7 minutes) to each part of the large intestine according to the observation priority of each part. As a result, the timetable update unit 28 calculates the updated scheduled time for each part. The scheduled time for the ascending colon, where the reexamination will be performed, is updated to 2.2 minutes.

[0092] Region information PI10 indicating the region under examination is displayed on screen 30A. The region information PI10 is included in the examination support information. The region under examination indicated by the region information PI10 is the transverse colon.

[0093] The instruction information IN11 and the instruction information IN12 are displayed on the screen 30A. The instruction information IN11 and the instruction information IN12 are included in the examination support information. The instruction information IN11 is the same as the instruction information IN10. The instruction information IN12 urges the examiner to hurry up a little and carry out the remainder of the examination.

[0094] The examination support information generating unit 29 calculates a first sum of the scheduled times (standard examination times) for the scheduled examination sites among the scheduled times included in the reference timetable before the update. For example, the examination support information generating unit 29 refers to the reference timetable TB10 shown in FIG. 3 and calculates a first sum (8.8 minutes) of the scheduled times from the transverse colon to the anus. The examination support information generating unit 29 also calculates a second sum of the scheduled times for the scheduled examination sites among the scheduled times included in the reference timetable (actual examination timetable) after the update. For example, the examination support information generating unit 29 refers to the actual examination timetable TB11 shown in FIG. 4 and calculates a second sum (7.7 minutes) of the scheduled times from the transverse colon to the anus.

[0095] The examination support information generation unit 29 generates speed information indicating the speed of the examination according to the result of comparing the first total with the second total. The speed information corresponds to the instruction information IN12. In the above example, the second total is smaller than the first total. In other words, after the reference timetable is updated, the scheduled time for examining the remaining parts is shorter than the standard scheduled time. Therefore, it is necessary to urge the examiner to speed up the examination. In such a case, the examination support information generation unit 29 generates speed information indicating that the speed of the examination should be increased. The examination support information generation unit 29 generates examination support information including the actual examination timetable and the speed information, and outputs the examination support information to the display unit 30.

[0096] The endoscopic examination support device 20 may execute the following process. The examination support information generation unit 29 refers to a region of interest (region of interest segment), which is a region where the examination has ended and which corresponds to the region currently being examined in the reference timetable or the actual examination timetable. For example, if the region currently being examined indicated by the region information is the ascending colon, the region of interest is the cecum, which was examined immediately before the examination of the ascending colon. The examination support information generation unit 29 compares the scheduled time for the region of interest in the actual examination timetable with the time required for the examination of the region of interest. The time required for the examination of the region of interest is stored in the memory unit 24 in step S108.

[0097] If the time required to examine the region of interest is shorter than the scheduled time for the region of interest, the examination support information generation unit 29 generates examination support information including instruction information for instructing the examiner to reexamine the region of interest. For example, the examination support information generation unit 29 generates examination support information including a message for urging the examiner to reexamine the region of interest because the examination time is shorter than the scheduled time. The examination support information generation unit 29 outputs the examination support information to the display unit 30.

[0098] As described above, the reference timetable includes region type information indicating a region to be examined or a region to be examined. The timetable update unit 28 changes the region type information of the region of interest. The changed region type information indicates a region to be examined.

[0099] The timetable generation unit 23 may determine the observation priority by using statistics (missed area probability) of areas that are likely to be overlooked by the endoscope 10. Information published at academic conferences or other such events may be used as the statistics. When regional differences (such as the gender, age, and race of the subject, or the characteristics of the examiner performing the examination, such as the examination method) or differences in the model of the endoscope 10 affect the missed area probability, the timetable generation unit 23 may perform deep learning or other techniques based on the subject's profile or the characteristics of the equipment or examiner used during the examination to set the missed area probability. In other words, the timetable generation unit 23 may determine areas with many lesions in the video images acquired by the endoscope 10, or areas that have been overlooked or have poor visibility, and statistically calculate the missed area probability for each area throughout the entire examination, for example. The timetable generation unit 23 may also detect areas with many lesions using an inference model.

[0100] The timetable generation unit 23 does not need to associate the missed area with a part of the body. The timetable generation unit 23 can use an inference model to infer that there is a lesion, that there is an image frame for which 3D reconstruction fails, or that there is an image frame with reduced visibility. Visibility decreases when the illumination light is too strong or when the tip of the endoscope 10 gets caught on the object of examination. The timetable generation unit 23 may determine whether an area in the image is a missed area based on the inference result.

[0101] To generate the inference model, the timetable generation unit 23 may generate training data by adding annotations to frames (frames of interest) in which visibility is impaired or specific image features (such as lesions) are present among video data including a series of frames obtained from the start to the end of the examination. The timetable generation unit 23 may also generate an inference model by performing learning so as to output frames of interest when a series of frames of a video image acquired by an endoscope are input.

[0102] That is, the timetable generation unit 23 may generate an inference model through machine learning using images acquired by the endoscope 10 and annotations indicating insufficient observation of the images as training data. The timetable generation unit 23 may determine the state of the images acquired by the endoscope 10 at the site scheduled for examination based on the inference model. The timetable generation unit 23 may generate a reference timetable according to the state determined based on the inference model. The timetable generation unit 23 may record the generated reference timetable in the storage unit 24.

[0103] FIG. 5 shows a method for generating an inference model. The timetable generation unit 23 obtains the inference model MO10 by performing machine learning using, as training data, images of the endoscope 10 from multiple cases and annotations resulting from performing a determination of missed areas on the images. The inference model MO10 is, for example, a neural network and is generated through deep learning. The inference model MO10 is not limited to a neural network and may be another machine learning model that can output information for an input image. The inference model MO10 is stored in the storage unit 24.

[0104] It is assumed that the training data is obtained using video (a series of still images) obtained during an endoscopic examination. The training data is a combination of training images and annotations resulting from performing a judgment on the images regarding missed areas. The inference model MO10 is a model trained to output annotations corresponding to input images. For example, annotations such as "There is an area that is easily overlooked in the third frame" or "There is an area containing unclear parts in the third frame" are used. This makes it possible to predict missed areas three frames prior. Alternatively, an annotation such as "There is a continuous area containing unclear parts" may be used. In examples where this annotation is used, it is easy to select images to use as training data.

[0105] Good inference can be performed if abundant training data can be collected. In the above example, a good feature of the training data is that specific images can be easily selected from a group of images using conventional lesion detection techniques or image degradation assessment techniques.

[0106] In the above example, annotations are used for areas that are easy to miss or unclear areas. Annotations may also be used to warn the examiner not to miss images that show lesions in images following an image with a particular feature. Alternatively, annotations may be used to warn the examiner not to miss images that are difficult to use for 3D reconstruction.

[0107] The timetable generation unit 23 may input multiple images output from the imaging unit 11 into the inference model MO10 and identify images determined to be missed regions. The timetable generation unit 23 may calculate the miss probability of each region based on the images identified for each region. The timetable generation unit 23 may set an observation priority according to the miss probability of each region and generate a reference timetable according to the observation priority.

[0108] When images acquired by the endoscope 10 while the examiner is inserting the endoscope 10 into the digestive tract are used as training data, the timetable generation unit 23 performs inference while the examiner is performing the insertion and generates a reference timetable. Alternatively, the timetable generation unit 23 may perform inference while the examiner is removing the endoscope 10 from the digestive tract. In this case, in order to perform predictive inference, annotations are added to images acquired immediately after the start of removal of the endoscope 10, and learning is performed. After removal has started and some time has passed, the timetable generation unit 23 can generate a reference timetable.

[0109] Deep learning is a multilayered version of the machine learning process that uses neural networks. A typical example is a forward propagation neural network, which sends information from front to back and makes a decision. The simplest example requires three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons determined by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights. Adding bias values ​​to the hidden and output layers facilitates the formation of logic gates. While three layers may be used for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, hidden layers with 9 to 152 layers have become practical in terms of training time, decision accuracy, and energy consumption. Alternatively, a convolutional neural network, which compresses image features and uses minimal processing, is also useful for recognizing motion and patterns. Alternatively, a "recurrent neural network" (fully connected recurrent neural network) can be used, which can handle more complex information and allows information analysis in which the meaning changes depending on the order, with information flowing bidirectionally. Other pattern recognition models that use supervised learning include support vector machines or support vector regression, in which the learning involves calculating the weights, filter coefficients, and offsets of the classifier. Another method uses logistic regression processing.

[0110] To perform learning, a general-purpose arithmetic processing circuit such as a CPU or FPGA (Field Programmable Gate Array) may be used, but because much of the processing in neural networks involves matrix multiplication, circuits specialized for matrix calculations called GPUs (Graphic Processing Units) or Tensor Processing Units (TPUs) may also be used. In recent years, such dedicated artificial intelligence (AI) hardware "neural network processing units (NPUs)" have been designed to be integrated and embeddable with circuits such as CPUs, and may even become part of the processing circuit.

[0111] For example, if an inference model is generated through learning using only images acquired through general photography and aligned horizontally and vertically, and the model is then used, it may not be able to perform correct inference on images with differences in vertical and horizontal orientation. Therefore, accurate judgment is possible by reading the above information from the inference information and performing inference on acquired images taking into account horizontal and vertical information from the orientation sensor. It is effective to incorporate horizontal and vertical information into image judgment before performing inference using the inference model. It is preferable that the endoscope system 1 retains information on these conditions and has a sensor for correcting images. The specifications and performance of the inference engine change depending on whether or not such constraints are included during learning. Therefore, such trial and error may be performed in parallel with the annotation process, and the trial and error results may be displayed.

[0112] Similarly, if learning is performed using only images captured at a specific distance from the object, accurate inference cannot be performed for images captured at distances other than the specific distance. In inference using an inference model generated based on such images, accuracy can be improved by enlarging the image of the distant object and using a pseudo-image of the closer object to compensate for the difference in distance. In this case, a distance sensor or the like is used in combination, and correction is performed during image inference to compensate for the difference between the actual enlargement or reduction of the image and the state of the learning data. The endoscopic examination support device 20 may be equipped with a memory for storing information indicating the learning data used to generate the inference model. When inference using the inference model is performed, the above information may be used to correct the image so that the inference model can correctly perform inference. The user may notice whether such correction is necessary during annotation work. A technique that allows verification of provisional learning during annotation work, as in the present embodiment, is significant.

[0113] It is difficult to train a compact inference engine installed in an information terminal product such as a camera or a mobile device to make highly accurate judgments with a small number of layers. Furthermore, since training takes time, innovations are needed to perform accurate annotation and training. When an inference model is generated, the specifications of the inference model change depending on the images used for training, so efficient training may be performed in conjunction with information at the time of training. Therefore, information indicating the type of training performed may be set during annotation work, and this information may be recorded as part of the inference information in a recording unit of an information acquisition device.

[0114] As explained above, in "supervised learning," the "relationship between input and output" is learned using training data in which the output is defined by annotations, and highly reliable inference is performed under specific conditions. In contrast, the endoscopic examination support device 20 may acquire an inference model that can handle more complex situations by using an "unsupervised learning" technique that learns the "structure of data."

[0115] The endoscopic examination support device 20 may also use a technique called "reinforcement learning," which learns "behaviors that maximize value and effectiveness." In this technique, learning is performed to find a law that increases state-action value. The learning is performed by estimating and increasing the value of the next state, not the current one, or by trial and error until a specific reward is obtained. Training data may be used to verify the learning results. In this technique, the correct answer output obtained by annotation is not learned as is, but learning is performed to obtain an even better answer. This makes it possible to respond to unknown situations.

[0116] These may be used in conjunction with supervised learning, or inference may be performed based on unsupervised learning and then based on supervised learning. The annotation data can also be used as validation data for such "unsupervised learning" and "reinforcement learning."

[0117] When a machine is asked to make a judgment, a human must teach the machine how to make the judgment. Here, we used a method to perform image judgment using machine learning, but a rule-based method in which a human applies rules acquired through experience or heuristics to the judgment may also be used.

[0118] As described above, the endoscopic examination support device 20 can output an examination time table (actual examination time table) relating to the time required for each section (site) of the scheduled examination in order to make up for the examination.

[0119] Second Embodiment A second embodiment of the present invention will be described below. In the second embodiment, an endoscope system 1 shown in FIG.

[0120] An adapter (endocuff) for turning up the folds in the large intestine is attached to the tip of the endoscope 10. Figures 6A and 6B show the state of the endoscope 10 with the adapter 12 attached. Figure 6A shows a state in which the examiner inserts the endoscope 10 into the large intestine and the endoscope 10 advances in the depth direction within the large intestine. Figure 6B shows a state in which the examiner performs an examination while pulling the endoscope 10. In areas where folds F10 exist, the adapter 12 slowly turns up the folds F10 as the tip of the endoscope 10 moves, which takes time. Therefore, the endoscopic examination support device 20 modifies the reference timetable that was generated assuming that the adapter 12 will not be used.

[0121] An example of the operation of the endoscopic examination support device 20 will be described using Fig. 7. Fig. 7 shows an example of the procedure of processing executed by the endoscopic examination support device 20. Processing that is the same as the processing shown in Fig. 2 will not be described. Processing that differs from the processing shown in Fig. 2 will be described.

[0122] (Step S101a) Step S101 shown in FIG. 2 is changed as follows. When the adapter 12 is used, the timetable generation unit 23 generates a reference timetable by executing the same process as step S101 shown in FIG. 2. Thereafter, the timetable generation unit 23 updates the reference timetable. When the adapter 12 is used, the time required to insert the endoscope 10 and the time required to examine a specific region increase. Therefore, the timetable generation unit 23 increases those times in the reference timetable and decreases the time required to examine other regions. The total increased time is the same as the total decreased time. The timetable generation unit 23 saves the updated reference timetable in the memory unit 24.

[0123] The endoscopic examination support device 20 may detect image change characteristics (image features) during insertion of the endoscope 10 and, based on the image features, change the weighting of the importance of the examination when the endoscope 10 is removed. When the endoscope 10 is inserted into the lumen of an organ and an examination plan is used in which the examination is performed when the endoscope 10 is removed, the endoscopic examination support device 20 detects changes in image features in each of multiple divided sections, stores information on the detected image features in the memory unit 24, and uses the image feature information stored in the memory unit 24 to enable the allocation of examination time to each part of the organ to be changed. For example, the image feature information indicates whether a lesion is present or whether visibility is good. This is based on the idea that it is possible to determine, to some extent, which sections require examination even during insertion of the endoscope 10.

[0124] (Step S120) After step S106 is executed, the examination position determination unit 26 determines the traveling direction of the endoscope 10 based on the multiple images output from the memory unit 24. When the examiner is moving the endoscope 10 toward the object of examination in the digestive tract, the traveling direction of the endoscope 10 is the depth direction. When the examiner is moving the endoscope 10 in the direction of removal from the digestive tract, the traveling direction of the endoscope 10 is the retreat direction. When the traveling direction of the endoscope 10 is the retreat direction, step S121 is executed. When the traveling direction of the endoscope 10 is the depth direction, step S122 is executed.

[0125] (Step S121) The endoscopic examination support device 20 executes a process for removal. The details of this process will be described later. After step S121 is executed, step S107 is executed.

[0126] (Step S122) The endoscopic examination support device 20 executes processing at the time of insertion. For example, the region detection unit 25 determines whether or not a region shown in the image output from the storage unit 24 is a fold region. The result of the determination is stored in the storage unit 24. The region detection unit 25 also detects a lesion such as a polyp shown in the image output from the storage unit 24. Information on the detected lesion is associated with the site information generated in step S104 and stored in the storage unit 24. As a result, information on the lesion at each site is saved in the storage unit 24. After step S122 is executed, step S107 is executed.

[0127] 8 shows an example of the procedure of the process executed in step S121 by the endoscopic examination support device 20. An example of the operation of the endoscopic examination support device 20 in step S121 will be described.

[0128] (Step S1210) The area detection unit 25 determines whether or not the joint of the adapter 12 is folded, based on the image output from the storage unit 24. If the joint is folded, step S1211, which will be described later, is executed. If the joint is not folded, step S107 shown in FIG. 7 is executed.

[0129] (Step S1211) When the joint is turning up the pleats, the imaging unit 11 captures images at a high frame rate. The imaging unit 11 outputs two or more images. The image acquisition unit 21 acquires the two or more images output from the imaging unit 11 and outputs the acquired two or more images to the storage unit 24. The storage unit 24 stores the two or more images.

[0130] (Step S1212) The storage unit 24 outputs the two or more images generated in step S1211 to the display unit 30. The display unit 30 displays the two or more images side by side.

[0131] (Step S1213) The region detection unit 25 determines whether or not the joint of the adapter 12 is about to separate from the fold, based on the image output from the storage unit 24. If the joint is about to separate from the fold, step S1214, which will be described later, is executed. If the joint is not about to separate from the fold, step S107 shown in FIG. 7 is executed.

[0132] (Step S1214) Based on the image output from the storage unit 24, the region detection unit 25 detects a lesion in a region that is newly shown in the image as the joint of the adapter 12 gradually moves away from the fold.

[0133] 9 shows an example of a screen 30A of the display unit 30. When the adapter 12 is used, the timetable generating unit 23 changes the reference timetable TB12 in step S101a shown in FIG. 7. The changed reference timetable TB12 is displayed on the screen 30A. The reference timetable TB12 is included in the examination support information.

[0134] The timetable generating unit 23 generates a reference timetable TB12 from the reference timetable TB10 shown in Fig. 3. For example, the timetable generating unit 23 increases the insertion time (3 minutes) to 3.5 minutes and the estimated time required for examining the sigmoid colon (3.4 minutes) to 3.9 minutes. Because the total of the insertion time and the estimated time for the sigmoid colon has increased by 1 minute, the timetable generating unit 23 decreases the total estimated time for areas other than the sigmoid colon by 1 minute.

[0135] A message MS11 is displayed on the screen 30A. The message MS11 is included in the examination support information. The message MS11 indicates that the reference timetable TB10 has been changed in response to the use of the adapter 12.

[0136] If the area shown in the image acquired when the endoscope 10 is removed is the same as the area shown in the image acquired when the endoscope 10 is inserted, the endoscope 10 may pass through that area quickly. When a re-examination is performed at a site where a missed area has occurred, the examination can be made up by having the examiner move the endoscope 10 quickly in areas that have not changed.

[0137] 10A and 10B show the state of the endoscope 10 with the adapter 12 attached. Fig. 10A shows a state in which the examiner inserts the distal end of the endoscope 10 into the large intestine and advances the distal end of the endoscope 10 in the depth direction within the large intestine. Fig. 10B shows a state in which the examiner is conducting an examination while pulling the endoscope 10 back. In a region where a fold F11 is present, the portion of the fold F11 shown in the image in Fig. 10A is different from the portion of the fold F11 shown in the image in Fig. 10B. Meanwhile, the portion other than the fold F11 shown in the image in Fig. 10A is the same as the portion other than the fold F11 shown in the image in Fig. 10B. Because the portion other than the fold F11 is observed during insertion of the endoscope 10, the examiner can move the distal end of the endoscope 10 more quickly through that region.

[0138] Fig. 11 shows an example of a screen 30A of the display unit 30. Portions that are the same as those shown in Fig. 4 will not be described. Portions that differ from those shown in Fig. 4 will be described.

[0139] An image IMG11 output from the imaging unit 11 is displayed on the screen 30A. An actual examination timetable TB13 is displayed on the screen 30A. The actual examination timetable TB13 is included in the examination support information. The timetable update unit 28 generates the actual examination timetable TB13 by updating the reference timetable TB12 shown in FIG. 9. The actual examination timetable TB13 shows an example in which a missed area occurs in the ascending colon.

[0140] Because a missed region was found in the ascending colon, the scheduled time for the ascending colon in the actual examination timetable TB13 is changed from 1.0 minute to 2.0 minutes. Because the scheduled time for the ascending colon has increased by 1 minute, the timetable update unit 28 reduces the total scheduled time for the remaining parts whose examinations have not yet been completed by 1 minute.

[0141] The same region information PI10 as the region information PI10 shown in FIG. 4 is displayed on the screen 30A. The region being examined indicated by the region information PI10 is the descending colon. Because the portion of the descending colon that appears in the image when the endoscope 10 is removed is the same portion of the descending colon that appears in the image when the endoscope 10 is inserted, the examiner may move the tip of the endoscope 10 faster in that portion. Therefore, instruction information IN13 and a message MS12 are displayed on the screen 30A. The instruction information IN13 and the message MS12 are included in the examination support information. The instruction information IN13 intuitively prompts the examiner to move the endoscope 10 faster. The message MS12 prompts the examiner in text to move the endoscope 10 faster.

[0142] Information about the lesion detected in step S122 when the endoscope 10 is inserted is stored in the storage unit 24. When the endoscope 10 is removed, the examiner performs treatment on the lesion. If the lesion detected in step S122 is not treated, the region detection unit 25 detects the lesion as a missed region.

[0143] 12 shows an example in which treatment is performed on a lesion. The imaging unit 11 generates an image IMG1 at imaging time t1, an image IMG2 at imaging time t2, and an image IMG3 at imaging time t3. When the endoscope 10 is moving in the depth direction, the imaging unit 11 generates the image IMG1. When the endoscope 10 is moving in the backward direction, the imaging unit 11 generates the images IMG2 and IMG3.

[0144] When the endoscope 10 is moving in the depth direction, the region detection unit 25 detects the lesion L1 and the lesion L2 that appear in the image IMG1. Information about the lesion L1 and the lesion L2 is stored in the storage unit 24.

[0145] When the endoscope 10 is moving backward, the region detection unit 25 detects the lesion L1 and the lesion L2 shown in the image IMG2. The examiner performs treatment on the lesion L2 but forgets to perform treatment on the lesion L1. The treatment on the lesion L2 is detected from the image generated by the imaging unit 11. Information indicating the treatment on the lesion L2 and information indicating the non-treatment on the lesion L1 are stored in the memory unit 24. The region detection unit 25 detects the lesion L1 shown in the image IMG3. The lesion L2 is hidden by the folds and is not shown in the image IMG3.

[0146] In step S110, the region detection unit 25 acquires information indicating whether or not treatment has been performed on the lesion from the storage unit 24, and detects a missed region based on the information. In the above example, the region detection unit 25 determines that the lesion L2 is not a missed region based on the information indicating that treatment has been performed on the lesion L2. Furthermore, the region detection unit 25 determines that the lesion L1 is a missed region based on the information indicating that treatment has not been performed on the lesion L1.

[0147] Fig. 13 shows an example of a screen 30A of the display unit 30. Portions that are the same as those shown in Fig. 4 will not be described. Portions that differ from those shown in Fig. 4 will be described.

[0148] An image IMG12 output from imaging unit 11 is displayed on screen 30A. A frame FR10 indicating the position of lesion L10 detected by region detection unit 25 is superimposed on image IMG12.

[0149] When the endoscope 10 is withdrawn from the lumen of the organ, the region detection unit 25 detects a lesion area where no treatment has been performed as a missed region. The timetable update unit 28 uses the information stored in the memory unit 24 to determine whether a lesion area has been missed during the examination of the examination-completed region. The timetable update unit 28 reclassifies the examination-completed region including the lesion area where it has been determined that a missed lesion has occurred into a region scheduled for examination. As described above, the standard timetable includes region type information indicating the examination-completed region or the region scheduled for examination. The timetable update unit 28 changes the region type information of the region including the lesion area where it has been determined that a missed lesion has occurred. The changed region type information indicates the region scheduled for examination. When the examination support information generation unit 29 outputs the updated standard timetable to the display unit 30, the display unit 30 displays the remaining time and the time allocation for the examination of the region scheduled for examination.

[0150] The timetable update unit 28 generates an actual examination timetable by updating the reference timetable in step S113 shown in Fig. 7. The actual examination timetable TB14 is displayed on the screen 30A. The actual examination timetable TB14 is included in the examination support information.

[0151] The timetable update unit 28 generates an actual examination timetable TB14 from the reference timetable TB12 shown in Fig. 9. The actual examination timetable TB14 shows an example in which an overlooked region occurs in the descending colon.

[0152] Because a missed region was found in the descending colon, the scheduled time for the descending colon in the actual examination timetable TB14 is changed from 0.4 minutes to 1.4 minutes. Because the scheduled time for the descending colon has increased by 1 minute, the timetable update unit 28 reduces the total scheduled time for the remaining parts whose examinations have not yet been completed by 1 minute.

[0153] A message MS13 is displayed on the screen 30A. The message MS13 is included in the examination assistance information. The message MS13 prompts the examiner in text to perform treatment on the lesion.

[0154] As described above, when the adapter 12 is attached, the reference timetable is updated. The reference timetable is also updated when a lesion is not treated and the lesion is detected as a missed area. Therefore, the endoscopic examination support device 20 can output an examination timetable indicating the time required for each section (site) of the scheduled examination in order to make up for the missed examination.

[0155] When a re-examination is to be performed on a site where a missed region has occurred, the display unit 30 displays information urging the examiner to quickly move the tip of the endoscope 10. The examiner can make up for the missed region by quickly moving the tip of the endoscope 10 in a region where no change has occurred.

[0156] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims.

[0157] According to each embodiment of the present invention, the endoscopic examination support method, endoscopic examination support device, and program can output an examination timetable relating to appropriate times for each section of the scheduled examination in order to make up for the examination.

[0158] REFERENCE SIGNS LIST 1 Endoscope system 10 Endoscope 11 Imaging unit 20 Endoscopic examination support device 21 Image acquisition unit 22 Information reception unit 23 Time table generation unit 24 Storage unit 25 Area detection unit 26 Examination position determination unit 27 Time calculation unit 28 Time table update unit 29 Examination support information generation unit 30 Display unit

Claims

1. A method for supporting endoscopic examination, comprising: calculating the remaining time until the end of an examination of an organ using an endoscope, based on a preset examination time and the elapsed time since the examination began; obtaining current position information that indicates the position in the organ currently being examined; and using the remaining time and the current position information, generating and outputting an examination timetable including the scheduled examination time for each of a plurality of divided sections obtained by dividing positions in the organ.

2. The endoscopic examination support method according to claim 1, further comprising: using the current location information to classify, among the multiple divided sections into which the organ is divided, divided sections into which the examination has been completed as divided sections into completed examination, and classifying divided sections into which the examination has not been completed as planned examination divided sections; and generating and outputting the examination timetable including the planned examination time for each of the planned examination divided sections according to the remaining time.

3. The endoscopic examination support method according to claim 2, further comprising: determining whether or not an oversight has occurred during the examination in the examination end division section; and, when generating and outputting the examination timetable, reclassifying the examination end division section in which it has been determined that an oversight has occurred into the examination planned division section.

4. An endoscopic examination support method as described in claim 2, further comprising the steps of: when the endoscope is inserted into the lumen of the organ, detecting a lesion in each of the plurality of divided sections; storing information of the detected lesion in a memory unit; when the endoscope is withdrawn from the lumen, using the information stored in the memory unit to determine whether or not a lesion has been overlooked during the examination in the examination end divided section; and when generating and outputting the examination timetable, reclassifying the examination end divided section including a lesion determined to have been overlooked into the examination planned divided section.

5. The endoscopic examination support method according to claim 2, further comprising: referencing a division section of interest, which is the examination end division section corresponding to the current position information in the examination timetable; and outputting instruction information for instructing an examiner to reexamine the division section of interest if the time required for the examination in the division section of interest is shorter than the scheduled examination time of the division section of interest.

6. The endoscopic examination support method according to claim 5, further comprising the steps of: reclassifying the target division section into the planned examination division section; and updating and outputting the examination time table in consideration of the time required for the reexamination.

7. The endoscopic examination support method according to claim 2, further comprising the steps of: acquiring from a memory unit a reference timetable including the scheduled examination time for each of the plurality of divided sections; using the remaining time and the current position information, performing a process on the reference timetable to update the scheduled examination time for each of the plurality of divided sections; and outputting the updated reference timetable as the examination timetable.

8. The endoscopic examination support method according to claim 7, further comprising the steps of: determining the state of the image acquired by the endoscope in the divided section based on an inference model obtained through machine learning using the image acquired by the endoscope and annotations indicating insufficient observation of the image as training data; generating the reference timetable according to the state determined based on the inference model; and recording the generated reference timetable in the memory unit.

9. The endoscopic examination support method according to claim 7, further comprising the steps of: calculating a first sum of the scheduled examination times in the scheduled examination divided sections among the scheduled examination times included in the reference timetable before being updated; calculating a second sum of the scheduled examination times in the scheduled examination divided sections among the scheduled examination times included in the reference timetable after being updated; generating speed information indicating the speed of the examination based on a result of comparing the first sum with the second sum; and outputting the examination timetable and the speed information to a display unit.

10. The endoscopic examination support method according to claim 1, further comprising the steps of: calculating the scheduled examination time for each divided section according to an adapter attached to the tip of the endoscope; and generating and outputting the examination timetable including the calculated scheduled examination times.

11. The endoscopic examination support method according to claim 1, wherein the examination time table and the current position information are output to a display unit.

12. An endoscopic examination support device comprising: a calculation unit that calculates the remaining time until the end of an examination of an organ using an endoscope, based on a preset examination time and the elapsed time since the examination began; an acquisition unit that acquires current position information that indicates the position in the organ currently being examined; and an output unit that uses the remaining time and the current position information to generate and output an examination timetable including scheduled examination times for each of a plurality of divided sections obtained by dividing positions in the organ.

13. A program for causing a computer to execute the steps of: calculating the remaining time until the end of an examination using an endoscope, based on a preset examination time and the elapsed time since the examination began; acquiring current position information indicating the position in the organ currently being examined; and generating and outputting an examination timetable including the scheduled examination time for each of a plurality of divided sections into which positions in the organ are divided, by using the remaining time and the current position information.

14. An endoscopic examination support method, comprising: calculating the remaining time until the end of an examination of an organ using an endoscope, based on a preset examination time and the elapsed time since the examination began; obtaining current position information indicating the position in the organ currently being examined; and generating and outputting an examination timetable that can simultaneously display the remaining time and the current position information.

15. The endoscopic examination support method according to claim 14, further comprising displaying, at least in the portion displaying the remaining time based on the examination timetable, the remaining time and the time allocation for examination of each part of the organ.

16. The endoscopic examination support method according to claim 15, wherein the time allocation is determined according to the difference in surface area between the examination target parts of the examination target organ, the statistics of the occurrence of lesions between the examination target parts, or the difficulty of the examination based on the shape of each examination target part.

17. The endoscopic examination support method according to claim 14, wherein when an examination delay exceeding a specific time width is detected with respect to the time allocated for examination of each part of the organ, the time allocation for examination of each part of the organ is re-set based on the remaining time.

18. The endoscopic examination support method according to claim 15, further comprising: using the current location information to classify, of the multiple divided sections into which the organ is divided, those divided sections into which the examination has been completed as examination end divided sections, and classifying those divided sections into which the examination has not been completed as examination planned divided sections; detecting a lesion in each of the multiple divided sections when the endoscope is inserted into the organ; storing information on the detected lesion in a memory unit; and using the information stored in the memory unit to determine, when the endoscope is withdrawn from the organ, whether or not a lesion has been overlooked during the examination in the examination end divided section; and, when generating and outputting the examination timetable, reclassifying the examination end divided section including the lesion determined to have been overlooked into the examination planned divided section, and displaying the remaining time and the time allocation of the examination in the examination planned divided sections.

19. An endoscopic examination support method as described in claim 15, further comprising the steps of: using the current position information, classifying, of the multiple divided sections into which the organ is divided, divided sections in which the examination has been completed as divided sections in which the examination has been completed, and classifying divided sections in which the examination has not been completed as divided sections in which the examination has been completed; detecting image features in each of the multiple divided sections when the endoscope is inserted into the organ; storing information on the detected image features in a memory unit; and switching the allocation of examination time for each part of the organ by using the information on the image features for each divided section stored in the memory unit.

20. The endoscopic examination support method according to claim 15, further comprising referring to the examination timetable and switching the allocation of examination time for each part of the organ when the time required for examination in a specific section among the multiple sections into which the organ is divided is short.

Images