Endoscope processor device

Abstract

An endoscope processor device capable of performing examination assistance in accordance with steps determined in accordance with guidelines is provided. An observation scene information acquisition section (22) acquires observation scene information related to a photographed observation scene. An observation scene comparison section (25) compares the observation scene information with reference scene information (31) related to a reference scene determined in advance in accordance with guidelines. In the case where the observation scene is not a recommended observation scene, a recommended observation scene notification section (26) displays recommended observation scene information on a display (18).

Description

Technical Field

[0001] This invention relates to an endoscope processor device that performs recognition processing using AI (Artificial Intelligence). Background Technology

[0002] In the medical field, diagnostic procedures using endoscopic systems equipped with endoscopes, light sources, and processors are becoming increasingly common. Endoscopic systems generate images of the object being examined based on signals captured by the endoscope and display them on a monitor. The user (doctor) can then perform a diagnosis while simultaneously observing the images displayed on the monitor.

[0003] Furthermore, in recent years, AI technology has begun to be incorporated into endoscopic systems to effectively assist in diagnosis. Such endoscopic systems are particularly useful in screening examinations, methods for quickly determining the likelihood of a disease. The following patent documents support this diagnostic aid technology.

[0004] Patent Document 1 discloses a technique that compares an inspection image with a reference image, prompts the operator to take a picture when the overlap is high, and provides assistance to acquire images according to the instructions. Patent Document 2 discloses an assistance technique that detects completed tasks based on the content of the processing and displays them on a display differently from unperformed tasks, thereby enabling inspections that comply with inspection protocols. Patent Document 3 discloses a technique that issues an alarm when the number of images of a specified inspection area does not match the predetermined number of images taken, to notify the user of image loss.

[0005] Previous technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2018-139846

[0008] Patent Document 2: Japanese Patent Application Publication No. 2017-99509

[0009] Patent Document 3: Japanese Patent Application Publication No. 2012-70936 Summary of the Invention

[0010] The technical problem to be solved by the invention

[0011] The guidelines for endoscopic examinations include not only the location of the areas to be observed but also the sequence of photographs taken. This is because secondary image readings ensure the continuity of the organ's region and suppress unnecessary movement of the camera during the examination, preventing omissions or false negatives and shortening the examination time.

[0012] However, in the methods of Patent Document 1 or Patent Document 3, in order to obtain images according to the instructions, the user must operate the endoscope to position it in the appropriate imaging position. If the complex operation required to obtain appropriate images is taken into account, it is not an effective method for rapid screening. Furthermore, in the method of Patent Document 2, although auxiliary information is displayed to enable examination according to predetermined steps, if images are missed due to the presence of lesions in the observed object or the lighting conditions, even experts may sometimes find it difficult to take images in the appropriate position without specific operational information.

[0013] The object of the present invention is to provide an auxiliary endoscope processor device capable of performing endoscopic examinations according to steps determined by guidelines.

[0014] means for solving technical problems

[0015] The endoscope processor device of the present invention includes a processor that acquires captured medical images, and when performing specific processing, uses the medical images to acquire observation scene information related to the captured observation scene, stores the observation scene information in an observation scene information memory, stores reference scene information related to a reference scene predetermined according to guidance in a reference scene information memory, compares the reference scene information and the observation scene information, determines whether the observation scene is a recommended observation scene that should be observed according to guidance, and displays recommended observation scene information related to the recommended observation scene on a display if the observation scene is not a recommended observation scene.

[0016] Preferably, the processor stores the observation scene and the shooting order for specific processing in the observation scene information memory, stores the reference scene and the shooting order of the reference scene determined according to the guidance in the reference scene information memory, compares the observation scene with the reference scene corresponding to the shooting order of the observation scene, and determines whether the observation scene is a recommended observation scene.

[0017] Preferably, if the observation scene is determined to be not a recommended observation scene, the processor displays reference scene information of the reference scene for the shooting order on the display as recommended observation scene information.

[0018] Preferably, when the observation scene is determined to be a recommended observation scene, the processor displays reference scene information of the next order of shooting sequence on the display as the recommended observation scene information.

[0019] Preferably, the specific processing is still image acquisition processing for acquiring still images of medical images.

[0020] Preferably, the processor displays guidelines for guiding the endoscopic observer to the recommended observation scenario.

[0021] Preferably, the processor overlays the guide onto the medical image.

[0022] Preferably, the guidelines include operational steps for shooting recommended observation scenes based on reference scene information.

[0023] Preferably, the processor displays information from the observation scene information that is at least related to the part of the observation scene and the location of the observation scene as character information.

[0024] Preferably, after all the shooting of the observed scene has ended, the processor detects the observed scenes that need to be reviewed and displays the information of the observed scenes that need to be reviewed.

[0025] Invention Effects

[0026] According to the present invention, it is possible to perform endoscopic examinations in accordance with the steps determined according to the instructions. Attached Figure Description

[0027] Figure 1 This is an external view of the endoscope system.

[0028] Figure 2 This is a block diagram illustrating the functions of an endoscope processor device.

[0029] Figure 3 This diagram illustrates the endoscopic procedures used during screening examinations of the upper digestive tract.

[0030] Figure 4 This is a diagram that illustrates information about a reference scene.

[0031] Figure 5 This is an example of displaying captured medical images and related information on a monitor.

[0032] Figure 6 This diagram illustrates a comparison between the observation scene information stored in the memory and the reference scene information stored in the memory when shooting according to the instructions.

[0033] Figure 7 This is an illustration of the screen that appears when you take photos according to the instructions.

[0034] Figure 8 This diagram illustrates a comparison between the observation scene information stored in the observation scene information storage and the reference scene information when shooting is not done according to the instructions.

[0035] Figure 9 This image illustrates the scene that appears when the instructions are not followed.

[0036] Figure 10This diagram illustrates the instructions for operating the endoscopic observer in order to capture recommended observation scenes.

[0037] Figure 11 This is a diagram illustrating the first example of guidelines for overlaying medical images.

[0038] Figure 12 This is the second example illustrating the guidelines for overlaying medical images.

[0039] Figure 13 This diagram illustrates the operational steps of using an endoscope to capture images of a reference scene defined in the guidelines.

[0040] Figure 14 This is a block diagram representing the functionality of an endoscope processor device that has a scene detection section that requires re-examination and observation.

[0041] Figure 15 This is a flowchart explaining the function of the recommended observation scenario reporting department. Detailed Implementation

[0042] like Figure 1 As shown, the endoscope system 10 includes an endoscope 11, a light source device 14, an endoscope processor device 16, a display 18, and a user interface 19. The endoscope 11 is optically connected to the light source device 14 and electrically connected to the endoscope processor device 16. The endoscope 11 illuminates the object being observed with light supplied from the light source device 14. Additionally, the endoscope 11 captures images of the object being observed to obtain image signals. The endoscope processor device 16 acquires the image signals from the endoscope 11 and performs various image processing on the image signals. Furthermore, the endoscope processor device 16 displays various images on the display 18 based on the image signals that have undergone various image processing.

[0043] like Figure 2 As shown, the endoscope processor device 16 includes a medical image acquisition unit 21, an observation scene information acquisition unit 22, an observation scene information memory 23, a reference scene information memory 24, an observation scene comparison unit 25, and a recommended observation scene notification unit 26. Each function of the endoscope processor device 16 is implemented by a control unit (not shown) composed of a processor executing a program for identifying lesions, etc., stored in a program memory.

[0044] The endoscope processor 16 is electrically connected to the display 18 and the user interface 19. The display 18 displays an image of the object being observed, or information accompanying the image. The user interface 19 receives input operations such as designating a region of interest (ROI) or setting functions.

[0045] The endoscope system 10 has three observation modes: a normal light mode, a special light mode, and an examination assistance mode. The medical image acquisition unit 21 generates medical images based on image signals obtained by capturing the observed object under different environments corresponding to each observation mode.

[0046] When the observation mode is normal light mode, ordinary light such as white light is emitted, and a normal light image with natural tones is generated based on the image signal obtained by photographing the observed object under illumination using this ordinary light. Alternatively, when the observation mode is special light mode, special light with a different emission spectrum than ordinary light is emitted, and a special light image with emphasis on lesions or structures is generated based on the image signal obtained by photographing the observed object under illumination using this special light.

[0047] Additionally, when the observation mode is the examination assistance mode, an examination illumination light is emitted. Then, an examination image is generated based on the image signal obtained by capturing the observed object within the illuminated examination area using the examination illumination light. The examination assistance mode is a mode that assists in reliably capturing images determined according to diagnostic guidelines during screening. Furthermore, the examination illumination light can be either ordinary light or special light.

[0048] For example, when screening the upper gastrointestinal tract according to guidelines, such as Figure 3 As shown, firstly, the insertion part 12 of the endoscope 11 is pressed into the lumen, passing through the esophagus Es and stomach St, and inserted to the deepest examination position P0 of the duodenum. Still images are captured upon reaching the deepest examination position P0. First, the user operates the operation part 13 of the endoscope 11 to capture images of the examination area determined according to the instructions at the deepest examination position P0. In the upper digestive tract, the insertion part 12 of the endoscope 11 is withdrawn in the order of duodenum, stomach St, and esophagus Es, and images of the examination areas determined according to the instructions are captured for each area. Once images of all examination areas are captured, the insertion part 12 of the endoscope 11 is withdrawn out of the body cavity. Furthermore, it is preferable to perform the same screening procedure for the lower digestive tract, such as the large intestine.

[0049] Additionally, a measuring scale (not shown) is provided on the outer peripheral surface of the insertion portion 12 of the endoscope 11 for measuring the length of insertion of the insertion portion 12 into the body, i.e., the insertion length. The measuring scale is engraved along the length direction of the insertion portion 12 at predetermined intervals (e.g., every 1 cm). The insertion length is measured by reading the value of the measuring scale using a scale detection sensor (not shown). The scale detection sensor is preferably located on the suction nozzle that the patient holds in their mouth when the insertion portion 12 of the endoscope 11 is inserted. The scale detection sensor is connected to the endoscope processor device 16 via wired or wireless means, and the information detected by the scale detection sensor is sent to the endoscope processor device 16.

[0050] Reference scene information 31 is stored in reference scene information memory 24. For example... Figure 4 As shown, reference scene information 31 includes, in association, reference scenes and the photographic order of the reference scenes determined according to the guidelines. For example, as reference scene information 31, the first reference scene in the photographic order is scene A of a small area of ​​part P. Similarly, the second reference scene is scene B of a small area of ​​part Q, and the third reference scene is scene C of a small area of ​​part Q. In this way, all reference scenes determined according to the guidelines, along with the photographic order, are included in reference scene information 31.

[0051] Figure 5 This is an example of a display screen 41 shown on monitor 18 during screening and inspection. The monitor 18 displays observation images 42 obtained from photographs of the observed object in real time. In addition to scene recognition results 43 of the observed scene, the relevant information display area 44 also displays information related to the observed scene, such as the possibility of lesions. Figure 5 In the image 43, the scene recognition result 43 for the observation scene "Scene C" is displayed, and the similarity between the observation scene and Scene C is "80%". The similarity is calculated by comparing the pre-stored image of the observation scene with the observation image 42. Furthermore, the observation image 42 is displayed as an examination image taken using examination illumination light, which can be either a normal light image or a special light image. Alternatively, in the case of taking still images while operating the endoscope 11, the captured still images can also be displayed.

[0052] The observation scene information acquisition unit 22 operates on the freeze button 13a of the operation unit 13 of the endoscope 11 (see reference). Figure 1 When a still image acquisition process (specific processing) is performed to capture a still image of the observation scene to be displayed on the display 18, observation scene information related to the captured observation scene is acquired. Furthermore, the observation scene information memory 23 stores the acquired observation scene information corresponding to the photographing order indicating the order in which the still image acquisition process is performed. The observation scene information includes information such as the location of the photographed area or its specific position, and whether there are lesions in the photographed area. This information is acquired as a result of processing performed using a machine learning-based learning model, known as AI (Artificial Intelligence). Preferably, the similarity between the observation scene and a reference scene, described later, is also processed by AI. Furthermore, a CNN (Convolutional Neural Network) can be used as the machine learning method.

[0053] The observation scene comparison unit 25 compares the observation scene information stored in the observation scene information memory 23 with the reference scene information 31 stored in the reference scene information memory 24, and determines whether the observation scene is a recommended observation scene that should be observed according to the guidelines.

[0054] Specifically, the scene comparison unit 25 compares the observed scene with a reference scene whose shooting order corresponds to the observed scene, and determines whether the observed scene is a recommended observation scene. For example... Figure 6 As shown, for example, the observation scene stored in the observation scene information memory 23 with the first shooting order is "Scene A". It is compared with "Scene A" corresponding to the first shooting order in the reference scene information 31. In this way, the observation scene comparison unit 25 compares the scenes with each other in the same shooting order.

[0055] If the observation scene comparison unit 25 determines that the observation scene is a recommended observation scene, the recommended observation scene notification unit 26 displays the reference scene information 31 of the next order of the observation scene in the relevant information display area 44 of the display 18.

[0056] exist Figure 6 In the observation scene information storage 23, if "Scene C" is stored in the observation scene that is the third in the photography order, and it is the same as the third reference scene in the reference scene information 31, then the observation scene is determined to be a recommended observation scene. At this time, if... Figure 7 As shown, the information of the fourth "Scene D" is displayed in the relevant information display area 44 of the display 18 as a reference scene information 31 for the next sequence of "Scene C".

[0057] On the other hand, if the observation scene comparison unit 25 determines that the observation scene is not a recommended observation scene, the recommended observation scene notification unit 26 displays reference scene information 31 of the shooting order of the observation scene in the relevant information display area 44 of the display 18 as recommended observation scene information.

[0058] like Figure 8 As shown, the observation scene information memory 23 stores "Scene D" in the observation scene with the third photography order. When compared with the reference scene in the reference scene information 31 with the third photography order, they are different since the third reference scene in the reference scene information 31 is "Scene C". At this time, the observation scene comparison unit 25 determines that the observation scene is not a recommended observation scene.

[0059] In this case, the recommended observation scene notification unit 26 displays the reference scene "Scene C," which is the third scene in the photography sequence, as the recommended observation scene information in the relevant information display area 44. As a method for displaying the recommended observation scene information, for example... Figure 9As shown, in order to inform that "Scene C" was not photographed, the relevant information display area 44 displays "Scene C was not photographed".

[0060] When all observation scenes corresponding to the reference scene determined according to the guidelines have been captured, if there is an image in the captured images that is judged to have a high probability of being lesion, information related to the observation scene of the image or operation information for moving to the observation scene is preferably displayed as the recommended observation scene to be captured next.

[0061] In addition, such as Figure 10 As shown, the recommended observation scene notification unit 26 displays a guide for guiding the endoscope 11's observer (not shown) to a position where the recommended observation scene can be captured (e.g., via a user operation guide display button 45 (see reference)). Figure 9 For example, the relevant information display area 44 of the display 18 displays "Up / Down Angle -15 Degrees" and "Insert 5cm" as operation information to guide the position of the shooting scene C.

[0062] Furthermore, the guide can also be overlaid on the observation image 42 displayed on the monitor 18. In this case, in addition to displaying a graphic indicating the direction or angle of the observer moving the endoscope 11, the operation information can also be indicated by changing the color of the frame portion of the observation image 42 to a color corresponding to the operation information to indicate the direction of movement, etc.

[0063] In the case of overlap with the observed image 42, for example, as Figure 11 As shown, the downward arrow and "-15 degrees" indicate that the angle of the endoscope 11's observer is adjusted by 15 degrees in the negative direction (downward). Furthermore, as... Figure 12 As shown, the display "Insert 5cm" indicates the operation of inserting the endoscope 11's observer 5cm. Thus, when displayed in conjunction with the observation image 42, it is preferable to switch the display for each operation.

[0064] like Figure 13 As shown, the reference scene information 31 includes operation steps for moving the endoscope 11's observer from the position of capturing the previous reference scene to the position of capturing the next reference scene. These operation steps are pre-stored in correspondence with information related to height and physique. The aforementioned guide is displayed according to these operation steps. For example, when moving the endoscope 11's observer to return it from scene D to scene C, the guide is displayed in the reverse order of the operation steps from scene C to scene D. That is, the guide is displayed in a manner that allows for insertion of 5cm after setting the vertical angle to -15 degrees.

[0065] like Figure 14As shown, the endoscope processor device 16 may also include a scene detection unit 27 that requires re-examination. After all observation scenes corresponding to the reference scene have been captured, the scene detection unit 27 detects observation scenes that require re-examination from the observation scenes stored in the observation scene information memory 23. For example, the scene detection unit 27 detects scene B (reference scene) in the stored observation scenes. Figure 6 If a lesion area suspected of being a lesion is detected in the endoscope (e.g.), scenario B is designated as a scenario requiring further observation. In this case, it is recommended that the observation scenario notification unit 26 display information related to scenario B and instructions for operating the endoscope 11 on the display 18.

[0066] Figure 15 This is a flowchart illustrating the function of the recommended observation scene notification unit 26 when a user performs a screening examination using the endoscope system 10 of the present invention. First, when the user captures a still image of the observation scene, the observation scene information acquisition unit 22 stores the observation scene information associated with the observation scene and the shooting sequence into the observation scene information memory 23 (step S1).

[0067] The observation scene comparison unit 25 compares the observation scene stored in the observation scene information memory 23 with the reference scene included in the reference scene information 31, and determines whether the observation scene is a recommended observation scene that should be observed according to the guidance (step S2). If it is determined that the stored observation scene is a recommended observation scene, the recommended observation scene notification unit 26 displays the reference scene information 31 of the next order of the observation scene's photography sequence on the display 18 as recommended observation scene information (step S3). If it is determined that the stored observation scene is not a recommended observation scene, it is determined that the photography was not done according to the guidance. In this case, the reference scene information 31 of the observation scene's photography sequence is displayed on the display 18 as recommended observation scene information (step S4). In step S4, a guide for guiding the endoscope 11's observer to a position where the recommended observation scene can be photographed is also displayed on the display 18, and the user follows the guide to photograph the unphotographed observation scene. When the photography of the recommended observation scene ends, the recommended observation scene notification unit 26 displays a guide to return to the original route (step S5).

[0068] In the same steps, the user follows the instructions to perform screening checks until the final observation scene. If the last observed scene captured is the last scene of the reference scene information 31, the observation scene comparison unit 25 determines it to be the final observation scene (step S6). After capturing all observation scenes corresponding to the reference scene, the observation scene re-examination detection unit 27 detects the observation scenes that need to be re-examined in the observation scenes stored in the observation scene information memory 23 (step S7). The recommended observation scene notification unit 26 displays information related to the observation scenes that need to be re-examined and instructions for operating the endoscope 11 on the display 18 (step S8). After completing all checks, including the re-examination, the user pulls the endoscope 11 out of the body cavity, ending the examination.

[0069] Furthermore, in this embodiment, when the user captures a still image of the observed scene, it is determined whether the shooting can be done according to the instructions. However, for images captured continuously like animation, the following method can also be used to determine whether the shooting is in accordance with the instructions. For example, the observation scene comparison unit 25 checks whether the reference scene in the reference scene information 31 is included in the image captured as animation. When checking according to the instructions, the specific scene (reference scene) is included in the animation in the same order as the predetermined shooting order. Therefore, if the order of the observed scene captured in the animation is different from the order of the reference scene, the observation scene comparison unit 25 determines that the shooting is not in accordance with the instructions. In the case of determining that the shooting is not in accordance with the instructions, the observation scene notification unit 26 is recommended to display operation information to guide the user to shoot according to the instructions.

[0070] In this embodiment, the hardware structure of the processing unit that performs various processes, such as the medical image acquisition unit 21, the observation scene information acquisition unit 22, the observation scene comparison unit 25, and the recommended observation scene notification unit 26, is as shown below. These processors include general-purpose processors that execute software (programs) and function as various processing units, such as CPUs (Central Processing Units), processors whose circuit structure can be modified after manufacturing, such as FPGAs (Field Programmable Gate Arrays), which are programmable logic devices (PLDs), and processors with circuit structures specifically designed for performing various processes, such as dedicated circuits.

[0071] A processing unit can be composed of one of these various processors, or it can be composed of a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, a single processor can constitute multiple processing units. As examples of a single processor constituting multiple processing units, firstly, there are computer-like devices such as client or server computers, where a combination of one or more CPUs and software is used to construct a single processor, which functions as multiple processing units. Secondly, there are devices such as System-on-Chip (SoC), where a single IC (Integrated Circuit) chip implements the overall system functionality including multiple processing units. In this way, various processing units are constructed using one or more different processors as hardware structures.

[0072] Moreover, more specifically, the hardware structure of these various processors is a circuit that combines circuit elements such as semiconductor components.

[0073] Symbol Explanation

[0074] 10 Endoscopic Systems

[0075] 11. Endoscope

[0076] 12 Insertion section

[0077] 13 Operations Department

[0078] 13a Freeze button

[0079] 14 Light source device

[0080] 16 Endoscopic Processor Device

[0081] 18 monitors

[0082] 19 User Interface

[0083] 21 Medical Image Acquisition Department

[0084] 22 Observation Scene Information Acquisition Department

[0085] 23 Observation Scene Information Storage

[0086] 24. Reference Scene Information Storage

[0087] 25. Observation Scene Comparison Section

[0088] 26. Recommended observation scenario to inform the department.

[0089] 27. The scene detection department needs to be re-examined.

[0090] 31 Reference Scene Information

[0091] 41 Display screen

[0092] 42. Observe the image.

[0093] 43 Scene Recognition Results

[0094] 44. Related Information Display Area

[0095] 45 Guide Display Button

Claims

1. An endoscope processing device, comprising a processor, The processor performs the following processing: Acquire captured medical images; When specific processing is performed, the medical images are used to acquire observation scene information related to the captured observation scene; The observed scene information is stored in the observed scene information storage device; The reference scenario information related to the reference scenario predetermined according to the guidelines is stored in the reference scenario information storage device; By comparing the reference scene information and the observation scene information, it is determined whether the observation scene is a recommended observation scene that should be observed according to the guidance; If the observed scenario is not the recommended observed scenario, information about the recommended observed scenario related to the recommended observed scenario is displayed on the monitor. The observed scene and the photographic sequence in which the specific processing is performed are stored together in the observed scene information memory; The reference scene and the photographing order of the reference scene determined according to the guidance are stored in the reference scene information storage; By comparing the observed scene with the reference scene corresponding to the photographic order of the observed scene, it is determined whether the observed scene is the recommended observed scene; If the observed scene is determined to be the recommended observed scene, at least one image of the reference scene from the next sequence of reference scene information in the shooting order is displayed on the display as the recommended observed scene information.

2. The endoscope processor device according to claim 1, wherein, If it is determined that the observation scene is not the recommended observation scene, the processor displays the reference scene information of the reference scene in the shooting order on the display as the recommended observation scene information.

3. The endoscope processor device according to claim 1 or 2, wherein, The specific processing is still image acquisition processing for obtaining still images of the medical image.

4. The endoscope processor device according to claim 1 or 2, wherein, The processor displays guidelines for guiding the endoscopic observer to the recommended observation scenario.

5. The endoscope processor device according to claim 3, wherein, The processor displays guidelines for guiding the endoscopic observer to the recommended observation scenario.

6. The endoscope processor device according to claim 4, wherein, The processor overlays the guide onto the medical image.

7. The endoscope processor device according to claim 4, wherein, The guidelines include operational steps for photographing the recommended observation scene based on the reference scene information.

8. The endoscope processor device according to claim 6, wherein, The guidelines include operational steps for photographing the recommended observation scene based on the reference scene information.

9. The endoscope processor device according to claim 1 or 2, wherein, The processor displays at least the information in the observation scene information that is related to the part of the observation scene and the location of the observation scene as character information.

10. The endoscope processor device according to claim 1 or 2, wherein, In a case where the photographing of the observation scene is completely finished, the processor detects an observation scene requiring review in the observation scene, and displays information of the observation scene requiring review.

Images