Endoscope system and method of operation thereof

Abstract

To provide an endoscope system which performs a detection of an abnormality of an apparatus due to a failure without using a special sensor for detecting the failure and a determination of the type of the failure and efficiently analyzes a cause of the failure by recording a detection result for each failure, and an operation method thereof.SOLUTION: A detection of a failure of an endoscope and a determination of the type of the failure are performed from a medical moving image photographed by the endoscope, and a failure moving image including a failure scene at the timing when the failure is detected from the medical moving image is extracted. The extracted contents of the failure moving image are determined according to a storage mode. The endoscope system is equipped with a storage mode for storing the entire extracted failure moving image and a storage mode for selecting the failure moving image to be stored by temporarily storing. The endoscope system stores failure information including at least the type of the failure and the failure moving image in a main storage region.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an endoscope system and a method of operation thereof. [Background technology]

[0002] In medical devices such as endoscopes, it is common for systems to automatically detect abnormalities such as equipment failures and save them as error logs. The error log records the type of abnormality, the date and time of occurrence, as well as the device settings, program processing status, and communication status between modules at the time the abnormality occurred. The error log makes it possible to understand the fact that an abnormality such as a failure occurred and the circumstances under which it occurred. However, when detecting failures using the error log, it is difficult to identify the specific nature of the failure or its cause. For this reason, it is necessary to understand the specific nature of the failure using data other than the error log.

[0003] Specifically, in Patent Document 1, inspections are performed on multiple status signals that indicate the operating status of the equipment. When a status signal is detected to be outside an allowable range, the time of detection, the time elapsed since startup, and the status thereafter are recorded. The type of abnormality is also displayed using a light, sound, etc. In Patent Document 2, an abnormality is detected from data indicating the operating status of the equipment, and the operating status is recorded along with the type of abnormality detected. It is determined whether the abnormality has been confirmed as a malfunction, and the recorded data is erased when an abnormality that has not been confirmed as a malfunction is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 3207874 [Patent Document 2] Patent No. 3549505 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, abnormalities are detected using equipment status signals, while in Patent Document 2, the type of abnormality is determined as either one that occurs temporarily during extremely high loads, etc., and is likely to self-recover over time, or one that is not. In Patent Documents 1 and 2, signals from the equipment's operating status are acquired, and the type of abnormality is determined by detecting, recording, and judging the abnormality. Even if it can be determined that the abnormality is not due to a malfunction, such as a temporary high load caused by external noise, it is difficult to determine that it is a malfunction or to identify its cause.

[0006] Furthermore, in devices with size constraints such as endoscopes, it is not possible or costly to install dedicated sensors for detecting faults. Furthermore, analyzing all the data obtained from the inspection results requires a lot of effort from the user and is inefficient. Therefore, there is a need to efficiently confirm the nature of the fault and identify the cause of the fault when detecting and recording the faults from the inspection results obtained from conventional inspection equipment.

[0007] The present invention aims to provide an endoscope system and an operating method thereof that can detect abnormalities in equipment caused by failures, determine the type of failure, and record the detection results for each failure, thereby efficiently analyzing the cause of the failure, without using a dedicated sensor for detecting the failure. [Means for solving the problem]

[0008] The endoscopic system of the present invention includes a processor that detects an endoscope malfunction from a medical video captured by the endoscope, determines the type of malfunction, extracts a malfunction video from the medical video that includes a malfunction scene at the time the malfunction was detected, and stores the malfunction video in a main memory area together with malfunction information that includes at least the type of malfunction.

[0009] It is preferable to extract a failure video having a plurality of frame images that are not failure scenes at a timing before, around, or after a failure scene.

[0010] It is preferable to determine the extraction range of the failure video to be extracted based on the failure scene and the failure information corresponding to the failure scene.

[0011] It is preferable to determine the extraction range of the failure video to be extracted according to the failure scene and the operation history at the timing around the failure scene.

[0012] The failure video is temporarily stored in a temporary storage area, and the failure video is selected based on the failure information. It is preferable to store the selected failure video in the main storage area.

[0013] It is preferable to calculate the importance of the failure video and select it based on the importance.

[0014] It is preferable to perform the selection based on the chronological order of the failure scenes.

[0015] It is preferable to store at least one fault video for each type of fault in the main storage area.

[0016] It is preferable that the selection be performed by a user operation.

[0017] It is preferable to acquire shape information of the endoscope and, when saving the failure video, save the shape information in the main storage area together with the failure video and the failure information.

[0018] Preferably, the shape information is used to extract the failure animation.

[0019] It is preferable to detect a fault during an endoscopy and extract a video of the fault.

[0020] It is preferable to temporarily store the failure video in non-volatile memory, select the temporarily stored failure video regardless of the completion status of the endoscopic examination, and store the selected failure video from the non-volatile memory in the main memory area.

[0021] When extracting the failure video, it is preferable to set a different identifier for each failure video, check the identifier of the failure video stored in the main memory area, and delete the failure video having the same identifier as the checked identifier from the non-volatile memory.

[0022] The operating method of the endoscopic system of the present invention includes the steps of detecting an endoscope failure from a medical video captured by the endoscope, determining the type of failure, extracting a failure video containing a failure scene at the time the failure was detected from the medical video, and storing the failure video in a main memory area together with failure information including at least the type of failure. [Effects of the Invention]

[0023] According to the present invention, abnormalities in equipment caused by failures can be detected and the type of failure determined without using a dedicated sensor for detecting failures, and the detection results for each failure can be recorded to efficiently analyze the cause of the failure. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 2 is an explanatory diagram showing connected devices of the endoscope system. [Figure 2] FIG. 2 is a functional block diagram of a processor device. [Figure 3] FIG. 10 is an explanatory diagram of a medical video in which a fault detection process is performed. [Figure 4] 4A, 4B, and 4C are explanatory diagrams of the frame images detected in the fault detection process. [Figure 5] FIG. 10 is an explanatory diagram of an extraction range for a failure scene when extracting a failure video. [Figure 6] FIG. 2 is an explanatory diagram of each storage mode. [Figure 7] FIG. 10 is an explanatory diagram illustrating a case where extracted failure videos are stored without being selected. [Figure 8] FIG. 10 is an explanatory diagram illustrating a case where extracted failure videos are selected and saved. [Figure 9] FIG. 10 is an image diagram showing a playback screen of a failure video. [Figure 10] FIG. 10 is an image diagram showing a thumbnail display of a video of a failure. [Figure 11] 1 is a flowchart showing a series of processing steps according to the present invention. [Figure 12] 12A, 12B, and 12C are explanatory diagrams illustrating how shape information of an endoscope insertion portion is acquired in the second embodiment. [Figure 13] FIG. 11 is an explanatory diagram illustrating temporary storage of a failure video in a nonvolatile memory in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] [First embodiment] 1 is a diagram showing connected devices of an endoscope 11 in an endoscope system 10 according to an embodiment of the present invention. The endoscope system 10 includes an endoscope 11, a light source device 12, a processor device 13, a database 14, a display 15, and a user interface (UI) 16. The processor device 13 is electrically connected to the light source device 12, the database 14, the display 15, and the user interface 16. The endoscope 11 has an elongated shape and includes an insertion section 11a that is inserted into a patient's body to acquire images, and an operation section 11b that accepts user operations such as bending the insertion section 11a and zooming when acquiring images. The tip of the insertion section 11a is provided with a tip section 11c that has an imaging function and that emits illumination light and projects treatment tools.

[0026] The light source device 12 is optically connected to the endoscope 11 and supplies illumination light to the endoscope 11 during endoscopic examination. The database 14 is a device that stores images and can send and receive data to and from the processor device 13, and may be a recording medium such as a USB (Universal Serial Bus) or an HDD (Hard Disc Drive). The display 15 displays images acquired by the processor device 13. The user interface 16 is an input device for inputting data to the processor device 13, and may be a foot pedal, a gesture recognizer, or a voice recognizer in addition to or instead of a keyboard or mouse. Input may also be made using input means provided in the medical device, such as a switch on the endoscope 11, rather than the user interface 16. Note that images include moving images, still images, and frame images that make up moving images.

[0027] The database 14 stores video and still images of medical examinations created by the processor device 13 or other medical devices. Unless otherwise specified, white light is used as illumination light, a video signal of 60 frames per second (60 fps) is acquired, and the imaging time is recorded. When the video signal is 60 fps, it is preferable to count the time in units of 1 / 100 of a second.

[0028] As shown in FIG. 2, in the endoscope system 10, a central control unit (not shown) configured with an image control processor runs programs in a program memory, thereby realizing the functions of an image acquisition unit 21, an input receiving unit 22, a main memory area 23, a display control unit 24, and a failure video creation unit 30. In addition, with the realization of the function of the failure video creation unit 30, the functions of a failure detection unit 31, an extraction range determination unit 32, an extraction unit 35, a temporary storage unit 36, and a selection unit 37 are also realized. The extraction range determination unit 32 further includes the functions of a mode control unit 33 and an image information management unit 34, and the selection unit 37 further includes the function of an importance calculation unit 38. The image acquisition unit 21 receives data such as medical videos captured by the endoscope 11 and medical videos stored in the database 14. The acquired medical videos are sent to the failure video creation unit 30 as medical videos 40.

[0029] The input receiving unit 22 is connected to the user interface 16. The main memory area 23 stores acquired medical videos or fault videos extracted by the fault detection unit 31. Instead of the main memory area 23, a database 14 connected to the processor device 13 may have that function. The display control unit 24 controls the display of images on the display 15. The processor device 13 has programs related to processes such as image processing stored in a program memory (not shown).

[0030] As shown in Figure 3, when creating a failure video, the failure video creation unit 30 determines the type of each frame that makes up the medical video through failure detection processing. Medical videos 40 input to the failure video creation unit 30 are continuous in chronological order of in-vivo imaging, i.e., they are continuous in time during the same examination. Each frame image that makes up the medical video 40 undergoes failure detection processing in the failure detection unit 31, and is sorted into normal frame images 41 with no failure, failure frame images 42 with an abnormality due to a failure, and non-fault abnormal frame images 43 that are abnormal but not a failure. The type of failure is also determined.

[0031] In a medical video 40 that has undergone the fault detection process, fault frame images 42 that have the same type of fault and are detected continuously or with a certain frequency or higher are determined to be the same fault. A group of frame images that constitute the same fault is detected as a fault scene. The medical video 40 undergoes a fault video extraction process for each fault scene. Note that each rectangle shown in Figure 3 is considered one frame that constitutes the medical video, and three consecutive frames are considered a fault scene, but the number of frame images containing fault scenes that constitute actual medical videos and fault videos is enormous.

[0032] As shown in FIG. 4, the fault detection process for the medical video 40 detects either a normal frame image 41, a fault frame image 42, or a fault-free abnormal frame image 43. The frame images shown as rectangles in FIG. 3 were actually captured under the conditions shown in FIGS. 4A to 4C. As shown in FIG. 4A, the normal frame image 41 is a normal frame image in which no abnormality was detected in the medical video 40. As shown in FIG. 4B, the fault frame image 42 is a frame image in which a fault in the endoscope 11 was detected in the medical video 40. It is a frame image in which a "break" C appears as a vertical or horizontal line in the image due to a "break," or a frame image in which a portion of the image is filled in due to "scope contamination" regardless of the observation target, and a non-lesion abnormal region T is included. Furthermore, as shown in FIG. 4C, the fault-free abnormal frame image 43 is a frame image that is abnormal but not a fault. For example, it is a frame image in which part or all of the image is obscured due to "external noise," or a frame image in which a lesion region R appears as a severe "lesion." The no-fault abnormal frame image 43 is not a fault, and is therefore treated in the same way as the normal frame image 41 when extracted after the fault detection process.

[0033] The function of the failure video creation unit 30 is as follows: The failure video creation unit 30 receives medical videos 40, which are videos captured by the endoscope 11 and acquired by the image acquisition unit 21 from the database 14 or the endoscope 11. The failure detection unit 31 performs a failure detection process on the received medical videos 40. The failure detection process determines whether or not there is a failure scene, which is one or more frame images in which a failure is detected, and, if there is a failure scene, determines the type of failure. The extraction unit 35 extracts a failure video 44, which is a short-time failure video, for each failure scene, which is one or more frame images showing the same failure, detected by the failure detection process. The failure video 44 is saved in the main memory area 23 together with failure information that includes at least information about the failure.

[0034] A failure scene is not limited to consecutive failure frame images 42, but may also include normal frame images 41 or non-failure abnormal frame images 43 at a rate below a certain level. However, consecutive failure frame images with different types of failure are different failure scenes. In other words, a failure scene is composed of failure frame images 42 of the same type that are detected consecutively or at a frequency above a certain level. Also, even if only one failure frame image 42 is detected in the failure detection process, it may be considered a failure scene.

[0035] If the failure video 44 consists only of failure scenes, it is difficult to compare the frame images with normal conditions, and it may not be possible to accurately analyze the cause of the failure. Therefore, when extracting the failure video 44 from the medical video 40, it is preferable to also acquire normal frame images 41 before or after the failure scene.

[0036] As shown in Figure 5, in extracting a failure video 44, the failure video is extracted from the medical video 40 based on a set extraction range. One of the following extraction ranges is set in advance before extraction: a first extraction range that extracts multiple normal frame images 41 at the timing of the failure scene and before and after the failure scene, a second extraction range that extracts multiple normal frame images 41 at the timing of the failure scene and before the failure scene, or a third extraction range that extracts only the failure scene. Regarding extraction, a no-fault abnormal frame image 43 is treated the same as a normal frame image 41 because it does not represent a failure.

[0037] When checking for a failure through on-screen display or investigating the cause of the failure through image analysis, it is preferable to set the first extraction range for the failure video 44, which includes multiple frame images that are not failure scenes, such as normal frame images 41, before the failure scene, allowing comparison before and after the failure. If the failure is not temporary and continues from its occurrence until the end of the endoscopic examination, an upper limit is set for the failure scenes to be extracted, such as by stopping detection as a failure scene after a certain period of time has elapsed. In this case, since there are no normal frame images 41 after the failure occurs, it is preferable to use the second extraction range. When prioritizing data volume reduction, such as when saving a large number of failure videos 44, it is preferable to extract the failure video 44 using the third extraction range. Regarding the extraction of the failure video 44 in each storage mode described below, unless otherwise specified, the extraction process is performed using the first extraction range.

[0038] The fault detection unit 31 executes fault detection processing for the received medical video 40, detecting the presence or absence of faults such as image distortion or dirt for each frame. The fault detection unit 31 has, for example, the function of a trained model required for the fault detection processing. That is, it is a computer algorithm consisting of a neural network that performs machine learning, and detects the presence or absence of a fault for each input medical video 40 according to the learning content, and if a fault is present, performs specific inference on the type of fault and determines the type of fault. In addition to the type of fault, it is preferable that the inference result also acquire information such as the degree of match with the previously learned learning content.

[0039] The fault detection process involves detecting fault frame images 42 from frame images in the acquired medical video 40, and fault determination, which determines the type of fault for the fault frame images 42, such as "disconnection," "dirty scope," or "damaged lens." The determination result is linked to the fault frame image 42 as fault information. Note that the fault detection process may also involve abnormality detection and fault extraction to determine the presence or absence of a fault in the medical video 40. Abnormality detection detects frame images that are not normal frame images 41 and have abnormalities such as image distortion or noise, and fault extraction determines whether the abnormality is a fault frame image 42 due to a "fault" caused by an endoscope fault, or a "no fault" non-fault abnormal frame image 43 due to a "no fault" that is not an endoscope fault.

[0040] The extraction range determination unit 32 determines the range to be extracted from the medical video 40, i.e., the configuration of the failure video 44, by combining the failure scene detected in the failure detection process, the setting of the storage mode controlled by the mode control unit 33, and information such as the operation history and failure information referenced by the image information management unit 34. This may also be set by user operation. For example, it determines whether to keep normal image frames before and after the selected failure scene, and the time range of the failure video 44 to be extracted from the type of failure scene and image information.

[0041] The mode control unit 33 receives and controls a mode selection regarding the setting of the extracted content of the failure video 44. The mode control unit 33 sets the normal mode or the storage mode, and further sets the storage mode to any one of the first to sixth storage modes. The mode selection is performed, for example, by a user operation. In the normal mode, the medical video 40 in which a failure has been detected is at least either stored in the main memory area 23 or transmitted to the display control unit 24 for screen display.

[0042] As shown in FIG. 6, there are six types of storage modes, which can be categorized by the conditions and methods for extracting and saving failure videos 44 from the medical videos 40. In the first and second storage modes, the failure videos 44 extracted for each failure scene are not selected, and all are saved in the main memory area 23. In the third to sixth storage modes, the failure videos 44 extracted for each failure scene are temporarily saved in a temporary storage area, and the failure videos 44 to be saved are selected. The selected failure videos 44 are saved in the main memory area 23. Details of extraction and selection in each storage mode will be described later.

[0043] The image information management unit 34 acquires and manages failure information including the type of failure acquired by the failure detection unit 31, the operation history of the endoscope 11, etc. The failure information includes, for example, the severity of the failure, the area of ​​the abnormal area caused by the failure, the duration of the failure scene, etc. The operation history of the endoscope 11 includes angle operation, use of treatment tools, zoom operation, etc.

[0044] The extraction unit 35 extracts failure videos 44 that include at least failure scenes from the medical videos 40 based on the extraction range determined by the extraction range determination unit 32. The extracted failure videos 44 are sent to the main memory area 23 or the temporary storage unit 36 ​​depending on the storage mode. If the storage mode selects the failure videos 44 to be stored, the failure videos 44 are sent to the temporary storage unit 36, and if the storage mode selects the failure videos 44 to be stored, the failure videos 44 are sent to the temporary storage unit 36.

[0045] When the failure video storage mode is set to the first storage mode or the second storage mode, selection is not performed, and the extracted failure video 44 is sent to and stored in the main memory area 23. When the failure video storage mode is set to the third storage mode to the sixth storage mode, selection of the failure video 44 to be stored is performed, and the extracted failure video 44 is sent to the temporary storage unit 36 ​​and temporarily stored.

[0046] When selecting the extracted failure videos 44 in the creation of the failure videos, the temporary storage unit 36 ​​temporarily stores each failure video 44 that is a candidate for storage. The selection is preferably performed by the selection unit 37 after all the extracted failure videos 44 have been temporarily stored. The temporary storage is realized by the temporary storage unit 36 ​​storing the videos in a temporary storage area (not shown) which is a volatile memory or nonvolatile memory in the processor device 13. Note that if the temporary storage area is a nonvolatile memory, it may be physically the same device as the main storage area 23.

[0047] The selection unit 37 selects the failure videos 44 to be saved as failure videos according to the selection criteria defined by the third to sixth storage modes. By performing the selection, it is possible to narrow down the failure scenes that are likely to be useful for analyzing the cause of the failure. The selection is performed by at least one of comparing the failure scenes or failure information in the failure videos 44 with each other, or determining whether the failure scenes meet the criteria defined for each storage mode. For example, in the third storage mode, selection is performed based on the importance of the failure scenes.

[0048] The importance calculation unit 38 calculates the importance of the failure scene detected by the failure detection unit 31. The importance can be expressed, for example, as a percentage (%) or a graded rating (high, medium, low). The importance is calculated from at least one of the order in which the failure occurred, the type of failure, the frequency of failure occurrence, the severity of the failure, the certainty of failure detection, the visibility of the failure, and the impact. The higher the importance of the failure, the more likely it is that the video will be useful for failure analysis, etc. The image information used to calculate the importance is acquired by the image information management unit 34.

[0049] The selected failure video 44 is sent to and stored in the main memory area 23. The stored failure video 44 may also be sent to the display control unit 24 and displayed on the screen so that it can be checked. The failure video 44 may also be stored in the database 14 instead of or in addition to the main memory area 23.

[0050] The display control unit 24 controls the screen display of the medical video 40 or the failure video 44 received from the failure video creation unit 30. When the failure video 44 is displayed, the display control unit 24 accepts the user's editing of the configuration of the failure video and image information including failure information via the user interface 16. When the failure video 44 is displayed, the failure scene can be confirmed and the failure information can be referenced.

[0051] The determination of the storage mode will now be described. The storage mode is preferably set by the user based on the content of the endoscopic examination, etc. The storage modes include first and second storage modes in which the failure videos 44 extracted for each failure scene are directly stored in the main memory area 23, and third to sixth modes in which the extracted failure videos 44 are sent to the temporary storage unit 36 ​​and temporarily stored in the temporary memory area, and the failure videos 44 to be stored in the main memory area 23 are selected.

[0052] As shown in FIG. 7, a case in which the extracted failure video 44 is not selected, which is common to the first and second storage modes, will be described. The medical video 40 undergoes a failure detection process to detect different failure scenes, namely, the failure scene in the failure frame image 42, the failure scene in the failure frame image 42a, and the failure scene in the failure frame image 42b, and these are transmitted to the extraction unit 35 along with the detected failure information. The extraction unit 35 extracts a failure video 44 for each failure scene. A failure video 44 having the failure scene in the failure frame image 42, a failure video 44a having the failure scene in the failure frame image 42a, and a failure video 44b having the failure scene in the failure frame image 42b are created. The created failure videos 44, 44a, and 44b are saved in the main memory area 23 in the order they were extracted.

[0053] In the first storage mode, the extraction range of the failure video 44 to be extracted is determined based on the failure scenes contained in the medical video 40 and the failure information corresponding to the failure scenes. Specifically, the extraction range of the failure video 44 for each failure scene is determined based on the type of failure, the duration of the failure, and the average area of ​​the abnormal part caused by the failure in the frame image detected in the failure detection process. For example, for failure scenes that last a long time or for failure scenes where the type of failure is "disconnection," the duration of the normal frame images 41 to be extracted, i.e., the range of the failure video 44 to be extracted, is lengthened. The extraction range may be determined by the number of frame images, or may be determined as a time range based on the number of frame images per second.

[0054] In the second storage mode, the extraction range of the failure video 44 to be extracted is determined according to the failure scene contained in the medical video 40 and the operation history of the endoscope 11 at the timing around the failure scene. Specifically, this includes angle operation, treatment tool operation, zoom operation, shooting mode switching operation, etc. This leads to the identification of failures related to the operation of the endoscope 11. There are cases where a disconnection failure occurs during angle operation, and in other cases, a failure related to the treatment tool can be confirmed by capturing the treatment tool in an image. Because failures can occur several seconds after the endoscope operation, the operation history at the timing before the failure scene is also included.

[0055] The range of the failure video 44 to be extracted according to the operation history is determined by, for example, when a treatment tool is operated, the treatment tool can be extended or retracted with a single operation, so that in the case of a failure related to the treatment tool, a failure scene due to a change in the extension or retraction of the treatment tool and normal frame images 41 that can be compared to the failure scene are extracted in a relatively short range. When an angle operation is performed, the angle operation may be performed once over several seconds, or may be performed multiple times to gradually bend the insertion section 11a of the endoscope 11. Therefore, by extracting many normal frame images 41, it becomes easier to confirm a failure that is so slight that it is not detected as a failure, or a change in the area of ​​an abnormal area due to a failure.

[0056] A case where the failure videos 44 extracted based on the set extraction range are selected, which is common to the third to sixth storage modes, will be described below. When selecting, the failure videos 44 temporarily stored in the temporary storage unit 36 ​​are selected based on the failure information, and the selected failure videos 44 are stored in the main memory area 23.

[0057] As shown in FIG. 8 , for example, a medical video 40 undergoes a failure detection process, whereby different failure scenes, failure frame images 42, 42a, and 42b, are detected and transmitted to the extraction unit 35 along with the failure information. The extraction unit 35 extracts a failure video 44 having the failure scene of the failure frame image 42, a failure video 44a having the failure scene of the failure frame image 42a, and a failure video 44b having the failure scene of the failure frame image 42b. The extracted failure videos 44, 44a, and 44b are temporarily stored in the temporary storage unit 36 ​​in order. After all the extracted failure videos 44 are temporarily stored, the selection unit 37 compares and selects the failure scenes. If the failure video 44 is selected, the failure videos 44a and 44b are deleted, and the failure video 44 is stored in the main memory area 23.

[0058] In the third storage mode, the importance of the failure video 44 is calculated, and the failure videos 44 are sorted based on the importance. When importance is used for sorting, the importance calculation unit 38 calculates the importance for each failure scene and links it to the failure information of the failure scene. The importance indicates the possibility of a serious failure or its usefulness in analyzing the cause of the failure. For example, a calculation result with an importance of 70% or more, or "high," is saved as a failure video 44, and a calculation result with an importance of less than 70% or "low" is deleted. The method of calculating the importance will be described later. Also, instead of when the importance is evaluated to be equal to or greater than a certain value, a certain number or percentage of failure videos 44 may be sorted in order of increasing importance.

[0059] In the fourth storage mode, the failure videos to be stored in the main memory area 23 are selected based on the chronological order of the failure scenes detected from the medical video 40. For example, among the multiple failure scenes detected in the medical video 40, the first or last N failure scenes, or a failure video 44 containing one failure scene selected at regular intervals, is saved. The first failure scene is the one in which the time elapsed since the start of the endoscopic examination is short, making it easy to detect failures other than those caused by the endoscopic examination, such as scope contamination. In the last failure scene, if multiple failures occur, it is easy to detect multiple types of failures from a single failure scene. Furthermore, by selecting at regular intervals, it is possible to check whether there are any failures that occur throughout the entire endoscopic examination.

[0060] In the fifth storage mode, failure videos are sorted based on the type of detected failure scene. When multiple failures occur, at least one failure video 44 for each type is saved in the main memory area 23. Even if multiple failures, including a type that occurs frequently and has a large abnormal area and a type that occurs infrequently and has a small abnormal area, are detected in the medical video 40, each type of failure scene can be confirmed without missing when reviewing the saved failure videos. Furthermore, when different types of failure scenes are detected simultaneously or at short intervals, a failure video 44 including both types of failure scenes may be created as a separate type of failure scene.

[0061] In the sixth storage mode, the failure video 44 to be stored in the main memory area 23 is selected by the user. Specifically, the temporarily stored failure video 44 and image information including at least the failure information are displayed on the display 15, and the user checks the played back failure video 44 and the image information to determine the failure video 44 to be stored in the main memory area 23. The image information to be displayed may be information on the failure video 44 or information on each displayed frame image.

[0062] 9, the display 15 displays an image display area 50 in which the failure video 44 is played back and the details of the failure are confirmed, and an image information display field 51 in which image information including the failure information of the played back failure video 44 is displayed. In the image display area 50, for example, extracted failure videos 44 are played back in order, and the user can select whether or not to store each of them in the main memory area 23. The failure information includes, for example, the type of failure, the time of the failure scene, the average area of ​​the abnormal area, and the operation history.

[0063] User operations in the selection process are performed via the user interface 16, and include methods of executing play and save commands provided in the command area 52 using a keyboard, mouse, foot pedal, etc. Image information may also be edited by user operations.

[0064] As shown in FIG. 10, instead of playing and checking the failure videos 44 in order, it is also possible to select them by displaying a list using thumbnails in the image display area 50. A thumbnail image of one of the multiple displayed failure videos 44 is selected, and image information for the selected failure video 44 is displayed in the image information display field 51. Whether or not to use the command area 52 may be determined based on the image information. When playing back the selected failure video 44, it may be played back at the thumbnail display size, or it may be temporarily enlarged and played back as shown in FIG. 9.

[0065] The failure video storage mode is not limited to the contents of the first to sixth storage modes, and the extraction criteria and selection conditions of each storage mode may be combined to store the failure video 44. After the failure video 44 in each storage mode is stored in the main memory area 23, the stored failure video 44 may be displayed on the display 15 for confirmation.

[0066] A series of operations for controlling the storage of the failure video 44 in this embodiment will be described with reference to the flowchart shown in FIG. 11. The endoscope system 10 acquires, from the database 14 or the endoscope 11, medical videos 40 captured during medical examinations, from which the failure video 44 is extracted (step ST110). One of the storage modes for extracting and storing the failure video 44 is selected depending on the examination details and expected failure details of the acquired medical video 40 (step ST120). A failure detection process is executed to detect failure scenes from the medical video 40 (step ST130). If no failure scenes are detected in the medical video 40 by the failure detection process (N in step ST140), it is determined that there is no failure. To extract the failure video 44, another medical video 40 is acquired (step ST110). If a failure scene is detected in the medical video 40 by the failure detection process (Y in step ST140), the type of failure for each detected failure scene is determined (step ST150). According to the selected storage mode and the extraction range according to the result of the failure detection process, the failure video 44 including at least a failure scene is extracted from the medical video 40 (step ST160).

[0067] The extracted failure videos 44 may be all saved as they are, or the failure videos 44 to be saved may be selected. Whether or not selection is performed is determined by the save mode. If the save mode is the first save mode or the second save mode, that is, if selection of the failure videos 44 is not performed (N in step ST170), all of the extracted failure videos 44 are saved in the main memory area 23 (step ST200).

[0068] When the storage mode is one of the third to sixth storage modes, that is, when selecting a failure scene (Y in step ST170), the failure video 44 is temporarily stored together with image information including failure information (step ST180). The temporarily stored failure video 44 is divided into the following categories for each storage mode: The failure videos 44 are selected as the failure videos to be stored by determining whether they satisfy certain criteria and comparing the failure videos 44 with each other (step ST190). The selected failure videos 44 are stored in the main memory area 23 (step ST200).

[0069] When analyzing the cause of the failure, if the number of failure videos 44 is insufficient or if the saving mode when extracting from the same medical video 40 is changed (N in step ST210), it is preferable to acquire the medical video 40 again, detect the failure, extract the failure video 44, and save it (step ST110). If the number of failure videos 44 sufficient for analyzing the cause of the failure can be saved (Y in step ST210), the series of steps ends.

[0070] [Second embodiment] In the first embodiment, a failure scene is detected from each frame constituting the medical video 40 in the failure detection process, and a failure video 44 is extracted and saved for each detected failure scene. In the second embodiment, a form will be described in which, in addition to that, shape information of the endoscope 11 is acquired, and when saving the extracted failure video 44, both the shape information and the failure information are saved. Note that a description of content common to the above embodiments will be omitted.

[0071] When saving the failure video 44, the failure video 44 in which the shape information and the failure information are linked is saved in the main memory area 23. The shape information may be used to analyze the cause of the failure of the failure video 44. Furthermore, the shape information of the endoscope 11 is used to extract the failure video 44. Furthermore, the shape information may be used not only for extraction but also for selection.

[0072] As shown in FIG. 12, in the fault detection process for the medical video 40, a fault may or may not be detected depending on the physical shape of the endoscope 11. For example, in FIG. 12(A), no disconnection occurs when the insertion section 11a is bent at approximately 45 degrees relative to the control section 11b at a certain distance from the distal end 11c of the endoscope 11. In contrast, in FIG. 12(B), a disconnection occurs when the insertion section 11a is bent at approximately 90 degrees relative to the control section 11b at a certain distance from the distal end 11c of the endoscope 11. In such cases, a disconnection does not occur if the angle is adjusted only once or twice. Furthermore, the angle adjustment may be performed gradually over time rather than in a short period of time, so the operation history may not always be sufficient to track the fault. Furthermore, in FIG. 12(C), when the insertion section 11a is bent at approximately 180 degrees relative to the control section 11b at a certain distance from the distal end 11c of the endoscope 11, this may be detected as a more serious fault than a 90-degree bend. Therefore, the fault detection process that adds the shape information of the insertion portion 11a of the endoscope 11 can perform fault detection process with higher accuracy than fault detection process for the medical video 40 that uses machine learning or the like.

[0073] Specifically, by using an endoscope system 10 equipped with a shape information sensor for the scope, the fault detection process is combined with fluctuations in the shape information of the endoscope 11 on a frame-by-frame basis. It is preferable to store the shape information triggered by the detection of a fault scene. This prevents faults, such as wire breaks, that occur during endoscopic examinations from being judged to be caused by external noise or from being calculated with low importance, enabling highly accurate detection. Furthermore, by finding in the analysis of the cause of a fault that a specific shape is the condition for the occurrence of the fault, repair of the fault becomes easier.

[0074] It is preferable to use a function that acquires shape information by generating magnetism from an external source, such as an endoscope insertion shape observation device (Coronavi), as the shape information sensor of the insertion section 11a of the endoscope 11. Furthermore, in addition to the shape information sensor, the endoscope 11 may acquire shape information using a function that is not intended for detecting failures in the tip, such as a motion sensor or an insertion depth meter.

[0075] [Third embodiment] The first and second embodiments are embodiments in which a medical video 40 capturing the entire endoscopic examination is acquired, and a failure video 44 is extracted and saved. In this embodiment, a failure detection process is performed on frame images received by a processor device while an endoscopic examination is being performed in real time, and a failure video 44 is extracted. Note that a description of content common to the above embodiments will be omitted.

[0076] Fault detection processing is performed in real time during an endoscopic examination. When an endoscopic examination begins, the image acquisition unit 21 receives one or a small number of frame images and inputs them to the fault video creation unit 30. The frame images input to the fault video creation unit 30 are sequentially subjected to fault detection processing by the fault detection unit 31, and fault scenes are detected.

[0077] A failure video 44 is extracted for each failure scene. In a storage mode that does not perform selection, the extracted failure video 44 is stored in the main memory area 23. In a storage mode that performs selection, the extracted failure video 44 is temporarily stored in the temporary storage unit 36. The temporarily stored failure video 44 is sorted according to the storage mode, and after selection, is stored in the main memory area 23. It is not necessary to create a medical video 40 that records the entire medical examination.

[0078] The function of the temporary storage unit 36 ​​may be realized by a nonvolatile memory that maintains stored data even when the processor device 13 is not powered. It is possible that the power of the endoscope 11 or the processor device 13 is turned off during an endoscopic examination, or that the power of the processor device 13 is turned off after an endoscopic examination is completed before the temporarily stored failure video 44 is completely saved in the main memory area 23. Even in such cases, the data temporarily stored in the temporary storage unit 36, which is a nonvolatile memory, can be prevented from being lost when the power is turned off.

[0079] In the storage mode for sorting, the failure videos 44 extracted for each failure are sequentially temporarily stored in the temporary storage unit 36, which is a non-volatile memory. If the power of the endoscope 11 or the processor device 13 is turned off during an endoscopic examination, the endoscopic examination ends at that point, and when the power is turned on again, sorting of the failure videos 44 begins. After the endoscopic examination ends, sorting is performed, and the selected failure videos 44 are stored in the main memory area 23.

[0080] As shown in Figure 13, a case where the power is turned off during an examination in real-time detection in which data is temporarily saved in a temporary storage area that is non-volatile memory will be described. Examination A is an examination in which the endoscopic examination is completed without the power being turned off, and Examination B is an examination in which the endoscope 11 or the processor device 13 is turned off during real-time detection. In Examination A in which the user operates to end the endoscopic examination, the temporarily saved failure videos 44 are selected and saved after the endoscopic examination is completed. In Examination B in which the end was forcibly terminated by the power being turned off rather than by the user's operation, when the power is turned on again, the failure videos 44 saved in the non-volatile memory at the time the power was turned off are selected and saved. Temporarily saving data in non-volatile memory allows the failure videos 44 to be selected regardless of the end status of the endoscopic examination, such as when the power is turned off during the examination.

[0081] After the selection is complete, the selected failure videos 44 are saved from the non-volatile memory to the main memory area 23, but the power to the processor device 13 may be turned off during the saving process, or the main memory area 23 may be disconnected if it is detachable from the processor device 13. If the saving process is interrupted, it may be impossible to distinguish between failure videos 44 saved in the main memory area 23 and failure videos 44 that are not saved. For this reason, an identifier such as a flag is set when extracting the failure videos 44 to enable automatic discrimination.

[0082] When temporarily saving failure videos 44, a different identifier is set for each failure video 44 when the failure videos are extracted. The identifier is preferably a flag or the like and is set as image information, but is not particularly limited as long as it allows for distinguishing between the failure videos 44. When resuming the interrupted saving process, the identifier of the failure video 44 saved in the main memory area 23 is checked, and it is determined whether or not there is a failure video 44 with the same identifier in the temporary storage area, which is non-volatile memory. If a failure video 44 has the same identifier as the confirmed identifier, it is not saved and is deleted from the non-volatile memory.

[0083] If the failure video 44 for which the same identifier has not been confirmed is in the temporary storage area, it is determined that the failure video 44 is not stored in the main memory area 23, and a storage process is performed. Note that the identifier may be used not only when the storage process is interrupted, but also during normal storage process to confirm whether the storage has been performed reliably.

[0084] In the third embodiment (real-time failure detection), the selection of the failure videos 44 does not need to wait until all the failure videos 44 are collected, i.e., until the end of the inspection, but may be performed immediately after the failure videos 44 are extracted. For example, the importance of the mth extracted failure video 44 that has already been temporarily stored and the newly extracted (m+1)th failure video 44 are compared, and the one with the lower importance is deleted. This makes it possible to save memory space when the area for temporary storage is limited.

[0085] In the above embodiment, the hardware structure of processing units that perform various processes, such as the central control unit, the image acquisition unit 21, the input receiving unit 22, the display control unit 24, and the fault detection unit 31, extraction range determination unit 32, mode control unit 33, image information management unit 34, extraction unit 35, temporary storage unit 36, selection unit 37, and importance calculation unit 38 included in the fault video creation unit 30, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for performing various processes.

[0086] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0087] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit formed by combining circuit elements such as semiconductor elements, and the hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). [Explanation of symbols]

[0088] 10 Endoscopy System 11 Endoscopy 11a Insertion part 11b Operation section 11c Tip 12 Light source device 13 Processor unit 14 Database 15 Display 16 User Interface 21 Image acquisition unit 22 Input receiver 23 Main storage area 24 Display control unit 30. Fault Video Creation Department 31 Fault detection unit 32 Extraction range determination unit 33 Mode control section 34 Image Information Management Department 35 Extraction part 36 Temporary storage section 37 Sorting Department 38 Importance calculation part 40 Medical Videos 41 Normal frame image 42 Faulty frame images 42a Failure frame image 42b Failure frame image 43 No fault abnormal frame image 44 Malfunction Video 44a Breakdown video 44b breakdown video 50 Image display area 51 Image information display area 52 Command Area C. Disconnection R Lesion area T Non-lesional abnormal area

Claims

1. a processor; The processor: Detecting a malfunction of an endoscope from a medical video taken by the endoscope; determining the type of the fault; extracting a failure video having a failure scene at a timing when the failure is detected from the medical video; The endoscope system stores the failure video in a main storage area together with failure information including at least the type of the failure.

2. The processor: The endoscope system according to claim 1 , wherein the failure video includes a plurality of frame images that are not failure scenes, and the frame images are extracted before, around, or after the failure scene.

3. The processor: The endoscope system according to claim 1 or 2, wherein an extraction range of the failure video to be extracted is determined based on the failure scene and the failure information corresponding to the failure scene.

4. The processor: The endoscope system according to claim 1 , wherein an extraction range of the failure video to be extracted is determined according to the failure scene and an operation history at a timing around the failure scene.

5. The processor: Temporarily storing the failure video in a temporary storage area; Selecting the failure video based on the failure information; The endoscope system according to claim 1 or 2, wherein the selected failure video is stored in a main storage area.

6. The processor: Calculating the importance of the failure video; The endoscope system according to claim 5 , wherein the selection is performed based on the importance.

7. The processor: The endoscope system according to claim 5 , wherein the selection is performed based on a chronological order of the failure scenes detected from the medical video.

8. The processor: The endoscope system according to claim 5, wherein at least one video of the failure is stored in a main storage area for each type of the failure.

9. The processor: The endoscope system according to claim 5 , wherein the selection is performed by a user operation.

10. The processor: acquiring shape information of the endoscope; 10. The endoscope system according to claim 1, wherein when the failure video is saved, the shape information is saved in the main storage area together with the failure video and the failure information.

11. The processor: The endoscope system according to claim 10 , wherein the failure video is extracted using the shape information.

12. The processor: The endoscope system according to claim 1 , wherein the failure is detected during an endoscopy, and the failure video is extracted.

13. The processor: Temporarily storing the failure video in a temporary storage area; Regardless of the endoscopy completion status, the temporarily stored video of the malfunction is selected, The endoscope system according to claim 12, wherein the selected failure video is saved from the temporary storage area to the main storage area.

14. The processor: When extracting the failure video, a different identifier is set for each of the failure videos; Check the identifier of the failure video stored in the main storage area; The endoscope system according to claim 13 , wherein the failure video having the same identifier as the confirmed identifier is deleted from the temporary storage area.

15. detecting a malfunction of the endoscope from a medical video captured by the endoscope; determining the type of the fault; extracting a failure video having a failure scene at a timing when the failure is detected from the medical video; and storing the failure video in a main memory area together with failure information including at least the type of the failure.

Images