Diagnostic device, diagnostic system, diagnostic method, and program

Abstract

A diagnostic device is provided with a processor for an endoscopic observer having a tip section of an optical system capable of irradiating light into the body and provided with a camera capable of capturing an image of the body. The processor performs diagnosis of the optical system by performing image analysis using AI on an inner surface image obtained by photographing the inner surface by the camera in a state in which the inner surface is irradiated with light in a state in which the cover is attached to the tip section. And a cover which covers the optical system and the irradiation window by being attached to the front end portion and is formed inside which the light can be reflected. The processor outputs a diagnosis result obtained by performing the diagnosis.

Description

Technical Field

[0001] This invention relates to a diagnostic device, a diagnostic system, a diagnostic method, and a procedure. Background Technology

[0002] Japanese Patent No. 5162374 discloses an endoscope image offset measuring device. The endoscope image offset measuring device described in Japanese Patent No. 5162374 includes a test pattern depicting a test pattern, a positioning mechanism, an image synthesis mechanism, and an offset acquisition mechanism. The positioning mechanism positions either the front end of the insertion portion of an electronic endoscope inserted into a body cavity or one of the test pattern relative to the other. After positioning by the positioning mechanism, the image synthesis mechanism synthesizes an endoscope image obtained by taking a picture of the test pattern with the electronic endoscope and a reference image. The reference image is a mask image with an exposed portion that hides invalid areas of the endoscope image and only exposes the valid areas, and the exposed portion has a reference pattern. The offset acquisition mechanism acquires the offset of the test pattern relative to the reference pattern from the synthesized image synthesized by the image synthesis mechanism.

[0003] Japanese Patent Application Publication No. 2021-182950 discloses an information processing apparatus comprising an evaluation unit that evaluates the state of a medical device based on sound signals generated by the device. The evaluation unit compares the sound signals with past sound signals from the medical device and evaluates the state of the medical device based on the comparison result. The evaluation unit evaluates the state of the medical device and detects or predicts malfunctions of the medical device. Furthermore, Japanese Patent Application Publication No. 2021-182950 discloses a method for determining medical device malfunctions via the cloud. Summary of the Invention

[0004] One embodiment of the present invention provides a diagnostic device, diagnostic system, diagnostic method, and procedure capable of highly accurate diagnosis of the optical system installed in an endoscope.

[0005] means for solving technical problems

[0006] The first aspect of the present invention is a diagnostic device comprising a processor for an endoscope, the endoscope having a front end portion capable of irradiating light into the body and equipped with a camera capable of capturing images of the body; the processor performing the following processing: performing AI-based image analysis on an image of the interior obtained by the camera capturing light into the interior while a cover is installed on the front end portion; the cover covers the optical system by being installed on the front end portion and is formed in the interior portion capable of reflecting light; and outputting the diagnostic result obtained by performing the diagnostic.

[0007] The second aspect of this invention is the diagnostic device described in the first aspect, wherein,

[0008] Diagnosis includes determining whether there are any defects in the optical system and whether those defects have been addressed.

[0009] The third aspect of this invention is a diagnostic device as described in the first or second aspect, wherein,

[0010] The diagnosis includes determining the type of defect in the optical system and the corresponding treatment. The types of defects include Type 1, where the optical system does not need to be recycled, and Type 2, where the optical system needs to be recycled.

[0011] The fourth aspect of this invention is the diagnostic device described in the third aspect, wherein,

[0012] The classification process includes classifying adverse conditions into Category 1 and Category 2.

[0013] The fifth aspect of this invention is a diagnostic device as described in the third or fourth aspect, wherein...

[0014] The first type is dirt in the optical system, and the second type is malfunction in the optical system.

[0015] The sixth aspect of this invention refers to a diagnostic device according to any one of the first to sixth aspects, wherein,

[0016] Image analysis is achieved by generating diagnostic results or information that forms the basis of diagnostic results by inputting images into the learned model. The learned model generates hypothetical diagnostic results or information that forms the basis of diagnostic results by inputting images into the model.

[0017] The seventh aspect of this invention refers to a diagnostic device according to any one of the third to fifth aspects, wherein...

[0018] AI includes: Category 1 determination AI, which determines the first category based on the image inside; and Category 2 determination AI, which determines the second category based on the image inside.

[0019] The eighth aspect of this invention refers to a diagnostic device according to any one of the first to seventh aspects, wherein,

[0020] AI is enhanced through relearning based on diagnostic results.

[0021] The ninth aspect of this invention refers to a diagnostic device according to any one of the first to eighth aspects, wherein,

[0022] Diagnosis includes location determination processing to pinpoint the location of defects in the optical system.

[0023] The tenth aspect of this invention refers to a diagnostic device according to any one of the first to ninth aspects, wherein,

[0024] The images inside are obtained through noise removal and / or edge extraction processing.

[0025] The eleventh aspect of this invention is the diagnostic device described in the tenth aspect, wherein,

[0026] The image inside is obtained by edge extraction after noise removal.

[0027] The 12th aspect of this invention refers to a diagnostic device according to any one of the 1st to 11th aspects, wherein,

[0028] It contains a first region that reflects light and a second region with a lower reflectivity than the first region.

[0029] The 13th aspect of this invention is the diagnostic device involved in the 12th aspect, wherein,

[0030] The area of ​​region 1 is larger than that of region 2, and the spatial frequency of region 2 is higher than that of region 1.

[0031] The 14th aspect of this invention is a diagnostic device as described in the 12th or 13th aspect, wherein...

[0032] The second area is the test image.

[0033] The 15th aspect of this invention refers to a diagnostic device according to any one of the 1st to 14th aspects, wherein,

[0034] The inside is formed into a curved surface.

[0035] The 16th aspect of this invention refers to a diagnostic device according to any one of the 1st to 15th aspects, wherein,

[0036] The inside is the surface that allows light to diffuse.

[0037] The 17th aspect of this invention refers to a diagnostic device according to any one of the 1st to 16th aspects, wherein,

[0038] The cover is formed by a double-layer structure.

[0039] The 18th aspect of this invention is the diagnostic device involved in the 17th aspect, wherein,

[0040] The double-layer structure consists of an inner layer and an outer layer, with a hollow area between the inner and outer layers.

[0041] The 19th aspect of this invention refers to a diagnostic device involved in any of the 1st to 18th aspects, wherein,

[0042] The outer surface of the lid is black.

[0043] The 20th aspect of this invention refers to a diagnostic device according to any one of the 1st to 19th aspects, wherein,

[0044] The cover is installed on the front end via an accessory, and the diameter of the connecting cover in the accessory and the portion of the front end is adjustable.

[0045] The 21st aspect of this invention is a diagnostic system comprising: a terminal for an endoscope, the endoscope having a front end portion capable of irradiating light into the body and equipped with a camera capable of capturing images of the body; and a server, wherein the terminal sends an image of the interior obtained by the camera capturing images of the interior while light is irradiated into the interior, with a cover installed on the front end portion, to the server, the cover covering the optical system and formed on the interior portion capable of reflecting light, and the server performing the following processing: performing an image analysis of the interior image using AI to diagnose the optical system; sending the diagnostic result obtained by performing the diagnostic to the terminal, and the terminal receiving the diagnostic result.

[0046] The 22nd aspect of the present invention is a diagnostic method for an endoscope, the endoscope having a front end of an optical system capable of irradiating light into the body and equipped with a camera capable of capturing images of the body. The diagnostic method includes the following steps: performing AI-based image analysis on an image of the interior obtained by the camera capturing light into the interior while a cover is installed on the front end, thereby performing a diagnosis of the optical system, wherein the cover covers the optical system by being installed on the front end and is formed in the interior capable of reflecting light; and outputting the diagnostic result obtained by performing the diagnosis.

[0047] The 23rd aspect of the present invention is a program for causing a computer to perform processing. The computer is used for an endoscope, which has a front end of an optical system capable of irradiating light into the body and equipped with a camera capable of capturing images of the body. The processing includes the following steps: performing AI-based image analysis on an image of the interior obtained by the camera capturing images of the interior while light is irradiating it through the front end with a cover installed, thereby performing a diagnosis of the optical system. The cover covers the optical system by being installed on the front end and is formed in the interior that is capable of reflecting light; and outputting the diagnostic results obtained by performing the diagnosis. Attached Figure Description

[0048] Figure 1 This is a conceptual diagram illustrating an example of how a doctor uses an endoscopic device included in a diagnostic system.

[0049] Figure 2 This is a conceptual diagram representing an example of the overall structure of a diagnostic system.

[0050] Figure 3 This is a block diagram illustrating an example of the hardware structure of the electrical system of a diagnostic system.

[0051] Figure 4 This is an example of a display image showing a frame generated by a camera capturing images of the large intestine and displayed in the first display area.

[0052] Figure 5 This is a conceptual diagram representing a structural example of a cover and its accessories.

[0053] Figure 6 This is a conceptual diagram illustrating an example of how a camera can photograph the inside of a cover while the cover is mounted on the front end of an endoscope via an accessory.

[0054] Figure 7 This is a block diagram illustrating an example of the main functions of a processor included in a medical support device and an example of information stored in a storage device.

[0055] Figure 8 This is a conceptual diagram illustrating an example of the processing capabilities of a medical support device and processing equipment.

[0056] Figure 9 This is a conceptual diagram illustrating an example of the processing content of the control unit and an example of the content displayed on the screen of the display device.

[0057] Figure 10 This is a flowchart illustrating an example of a diagnostic procedure.

[0058] Figure 11 This is a conceptual diagram representing the first variation of the processing content of the processing device.

[0059] Figure 12 This is a conceptual diagram representing a second variation of the processing content of the processing device.

[0060] Figure 13 This is a conceptual diagram representing the third variation of the processing content of the processing device.

[0061] Figure 14 This is a conceptual diagram representing the fourth variation of the processing content of the processing device.

[0062] Figure 15This is a conceptual diagram illustrating an example of how the execution unit performs noise removal processing, edge extraction processing, identification processing, comprehensive judgment processing, and / or generation processing. Detailed Implementation

[0063] Hereinafter, an example of an embodiment of the diagnostic device, diagnostic system, diagnostic method and procedure involved in the present invention will be described with reference to the accompanying drawings.

[0064] First, the terminology used in the following explanation will be explained.

[0065] CPU stands for Central Processing Unit. GPU stands for Graphics Processing Unit. GPGPU stands for General-Purpose computing on Graphics Processing Units. APU stands for Accelerated Processing Unit. TPU stands for Tensor Processing Unit. RAM stands for Random Access Memory. NVM stands for Non-volatile memory. EEPROM stands for Electrically Erasable Programmable Read-Only Memory. ASIC stands for Application Specific Integrated Circuit. PLD stands for Programmable Logic Device. FPGA stands for Field-Programmable Gate Array. SoC stands for System-on-a-chip. SSD stands for Solid State Drive. USB stands for Universal Serial Bus. HDD stands for Hard Disk Drive. EL stands for Electro-Luminescence. CMOS stands for Complementary Metal Oxide Semiconductor. CCD stands for Charge Coupled Device. AI stands for Artificial Intelligence. BLI stands for Blue Light Imaging. LCI stands for Linked Color Imaging. I / F stands for Interface. SSL stands for Sessile Serrated Lesion.LAN stands for "Local Area Network". WAN stands for "Wide Area Network". 5G stands for "5th Generation Mobile Communication System". IC stands for "Integrated Circuit".

[0066] In the following description, a processor (hereinafter referred to as a "processor") can be a single physical or virtual computing device, or a combination of multiple physical or virtual computing devices. Furthermore, a processor can be a single computing device or a combination of multiple computing devices. Examples of computing devices include CPUs, GPUs, GPGPUs, APUs, or TPUs.

[0067] In the following description, the marked memory refers to memory such as RAM that temporarily stores information and is used as working memory by the processor.

[0068] In the following description, the symbolic storage device refers to one or more non-volatile storage devices that store various programs and parameters. Examples of non-volatile storage devices include flash memory, hard disks, or magnetic tapes. Furthermore, cloud storage devices are another example of a storage device.

[0069] In the following embodiments, the marked external I / F is responsible for the transmission and reception of various information between multiple interconnected devices. An example of an external I / F is a USB interface. External I / Fs can also include communication I / Fs that include communication processors and antennas. Communication I / Fs are responsible for communication between multiple computers. Examples of communication standards applied to communication I / Fs include wireless communication standards such as 5G, Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0070] In the following embodiments, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" can mean only A, only B, or a combination of A and B. Furthermore, in this specification, when three or more cases are connected by "and / or", the same consideration as "A and / or B" can be applied.

[0071] Figure 1 This is a conceptual diagram illustrating an example of using diagnostic system 1. For example... Figure 1As shown, the diagnostic system 1 includes a processing unit 2 and an endoscope device 10, which are communicatively connected via a network 3. The Internet is one example of the network 3. However, the Internet is only one example; other examples of the network 3 include WANs and / or LANs.

[0072] In this embodiment, the diagnostic system 1 is an example of the "diagnostic system" involved in the present invention, the processing device 2 is an example of the "server" involved in the present invention, and the endoscope device 10 is an example of the "diagnostic device" and "terminal" involved in the present invention.

[0073] The endoscope device 10 is used by the doctor 12 during endoscopic examinations. Endoscopic examinations are assisted by staff such as nurses 14.

[0074] Information obtained by the endoscope device 10 is sent to the processing device 2 via the network 3. A cloud server can be cited as an example of the processing device 2. However, a cloud server is only one example; the processing device 2 can be a local server or a personal computer. The processing device 2 receives information sent from the endoscope device 10, performs processing using the received information, and sends the processing result obtained from the processing back to the endoscope device 10.

[0075] The endoscopic device 10 includes an endoscopic observer 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In this embodiment, the endoscopic observer 16 is an example of the "endoscopic observer" involved in the present invention.

[0076] The endoscope device 10 is a medical imaging device used to diagnose and treat the large intestine 28 contained within the body of a subject 26 (e.g., a patient) using an endoscope observer 16. In this embodiment, the large intestine 28 is the object observed by the physician 12.

[0077] The endoscope 16 is used by the doctor 12 and inserted into the body of the subject 26. In this embodiment, the endoscope 16 is inserted into the large intestine 28, which is the lumen organ of the subject 26.

[0078] The endoscope device 10 enables the endoscope observer 16, which is inserted into the large intestine 28 of the subject 26, to take pictures of the large intestine 28 of the subject 26, and to perform various medical procedures on the large intestine 28 as needed.

[0079] The endoscope device 10 acquires and outputs an image representing the interior of the large intestine 28 by photographing the interior of the patient 26. In this embodiment, the endoscope device 10 is an endoscope with an optical imaging function, which captures reflected light obtained by illuminating light 30 inside the large intestine 28 and reflecting it through the intestinal wall 32 of the large intestine 28. Light 30 is an example of "light" as understood in this invention.

[0080] Furthermore, an endoscopic examination of the large intestine 28 is illustrated here, but this is only one example. The present invention also applies to endoscopic examinations of luminal organs such as the esophagus, stomach, duodenum, or trachea.

[0081] A light source device 20, a control device 22, and a medical support device 24 are mounted on a trolley 34. Multiple platforms are arranged vertically within the trolley 34, with the medical support device 24, control device 22, and light source device 20 mounted from the lower platform to the upper platform. A display device 18 is mounted on the top platform of the trolley 34.

[0082] The control device 22 controls the entire endoscope device 10. Under the control of the control device 22, the medical support device 24 performs various image processing operations on images obtained by the endoscope observer 16 capturing images of the intestinal wall 32. Furthermore, the medical support device 24 is communicatively connected to the processing device 2 via the network 3, and receives requested services from the processing device 2 by requesting such services.

[0083] Display device 18 displays various information, including images. Examples of display devices 18 include liquid crystal displays (LCDs) or EL displays. Furthermore, a tablet terminal with a display can be used in place of or in conjunction with display device 18.

[0084] A screen 35 is displayed on the display device 18. The screen 35 includes multiple display areas. These multiple display areas are arranged side-by-side within the screen 35. Figure 1 In the example shown, as one example of multiple display areas, a first display area 35A and a second display area 35B are illustrated. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as the main display area, and the second display area 35B is used as a sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited to this, as long as they fall within the frame 35.

[0085] An endoscopic dynamic image 39 is displayed in the first display area 35A. The endoscopic dynamic image 39 is a dynamic image acquired by the endoscopic observer 16 capturing images of the intestinal wall 32 within the large intestine 28 of the subject 26. Figure 1In the example shown, as an example of endoscopic dynamic image 39, a dynamic image reflecting the intestinal wall 32 is shown.

[0086] In the intestinal wall 32 projected in the endoscopic dynamic image 39, as the area of ​​interest (i.e., the area of ​​observation) observed by the physician 12, there are lesions 42 (e.g., in...). Figure 1 In the example shown, a lesion 42 is present. The physician 12 can visually identify the intestinal wall 32 containing the lesion 42 through the endoscopic dynamic image 39.

[0087] Lesion 42 can be of various types. Examples of lesion 42 include, for example, neoplastic polyps and non-neoplastic polyps. Examples of neoplastic polyps include, for example, adenomatous polyps (e.g., SSL). Examples of non-neoplastic polyps include, for example, mammillary polyps, hyperplastic polyps, and inflammatory polyps. Furthermore, the types exemplified here are pre-conceived as types of lesions 42 for endoscopic examination of the colon 28; the types of lesions 42 will differ depending on the organ being examined.

[0088] In this embodiment, for ease of explanation, an example is given in which one lesion 42 is reflected in the endoscopic dynamic image 39. However, the present invention is not limited to this. The present invention is valid even if multiple lesions 42 are reflected in the endoscopic dynamic image 39.

[0089] In this embodiment, lesion 42 is illustrated, but this is only one example. The area of ​​interest (i.e., the area of ​​observation) observed by the doctor 12 may also be a characteristic area with certain inherent characteristics, such as an organ (e.g., duodenal papilla), a marker, an artificial treatment instrument (e.g., an artificial clamp), or a treated area (e.g., an area with residual traces after the removal of polyps, etc.).

[0090] The image displayed in the first display area 35A is one frame 40 contained in a dynamic image consisting of multiple frames 40 arranged in a time sequence. That is, in the first display area 35A, multiple frames 40 arranged in a time sequence are displayed at a specified frame rate (e.g., tens of frames per second).

[0091] As an example of a moving image displayed in the first display area 35A, a moving image in real-time preview mode can be given. Real-time preview mode is just one example; like moving images in post-view mode, it could also be a moving image that is temporarily stored in memory or the like before being displayed. Furthermore, records stored in memory or the like can be played and displayed on screen 35 (e.g., the first display area 35A) as endoscopic moving images 39, using frames contained in the moving image.

[0092] Within screen 35, the second display area 35B is adjacent to the first display area 35A and is displayed in the lower right of the front view within screen 35. The display position of the second display area 35B can be any position as long as it is within screen 35 of the display device 18, but it is preferred to display it in a position that can be compared with the dynamic image 39 of the endoscope.

[0093] In the second display area 35B, auxiliary information 44 is displayed. This auxiliary information 44 assists the physician 12 and others in making medical judgments during endoscopic examinations and in judging malfunctions of the endoscope observer 16 during endoscope maintenance. The auxiliary information 44 is information referenced by the physician 12 and others. As an example of auxiliary information 44, various information related to the subject 26 inserted into the body by the endoscope observer 16 and / or various information obtained by performing diagnostic procedures on the endoscope observer 16 (e.g., determining whether malfunctions have occurred in the endoscope observer 16 or determining what kind of malfunctions have occurred).

[0094] Figure 2 This is a conceptual diagram illustrating an example of the overall structure of the endoscope device 10. (See diagram for example.) Figure 2 As shown, the endoscope 16 includes an operating section 46 and an insertion section 48. The insertion section 48 is partially bent by being operated by the operating section 46. The insertion section 48 is oriented according to the instructions of the physician 12 (see reference). Figure 1 The operation performed on the operating unit 46 is based on the large intestine 28 (reference). Figure 1 The shape of the tube is bent and inserted into the large intestine 28.

[0095] A camera 52, a lighting device 54, and an opening 56 for a treatment device are provided at the front end 50 of the insertion part 48. The camera 52 and the lighting device 54 are provided on the front end face 50A of the front end 50. In addition, the example given here is that the camera 52 and the lighting device 54 are provided on the front end face 50A of the front end 50, but this is only one example. The camera 52 and the lighting device 54 can also be provided on the side of the front end 50 so that the endoscope observer 16 can be configured as a side view mirror.

[0096] Camera 52 is mounted on endoscope observer 16, inserted into the body cavity of subject 26, and captures images of the observed area, thereby generating frame 40 as an endoscopic image. In this embodiment, camera 52 generates an endoscopic dynamic image 39 containing multiple frames 40 arranged in a time sequence by capturing images inside the body of subject 26 (e.g., inside the large intestine 28). As an example of camera 52, a CMOS camera can be cited. However, this is merely one example, and other types of cameras, such as CCD cameras, could also be used. In this embodiment, front end 50 is an example of a "front end" according to the present invention, and camera 52 is an example of a "camera" according to the present invention.

[0097] The illumination device 54 has an optical system 55. The optical system 55 includes illumination lenses 55A and 55B. Illumination lenses 55A and 55B are lenses including objective lenses, which protrude from the front end face 50A. The optical system 55 is an example of an "optical system" according to the present invention.

[0098] The lighting device 54 illuminates light 30 via the optical system 55 (reference). Figure 1 The type of light 30 emitted from the illumination device 54 can include, for example, visible light (e.g., white light) and non-visible light (e.g., near-infrared light). Furthermore, the illumination device 54 emits special light via the optical system 55. Examples of special light include BLI light and / or LCI light. The camera 52, while the large intestine 28 is illuminated by the light 30 emitted from the illumination device 54, captures images of the large intestine 28 using optical methods.

[0099] The opening 56 for the treatment device is an opening for the treatment device 58 to protrude from the front end 50. Furthermore, the opening 56 for the treatment device can also be used as a suction port for aspirating blood and internal waste, and as a delivery port for discharging fluids.

[0100] An instrument insertion port 60 is formed in the operating section 46, and an instrument 58 is inserted into the insertion section 48 through the instrument insertion port 60. The instrument 58 passes through the insertion section 48 and protrudes outward from the instrument opening 56. Figure 2 In the example shown, the puncture needle protrudes from the opening 56 of the treatment device 58. Here, the puncture needle is shown as the treatment device 58, but this is only one example. The treatment device 58 can also be a holding forceps, a nipple cutter, a snare, a catheter, a guidewire, a cannula, and / or a puncture needle with a guiding sheath, etc.

[0101] The endoscopic observer 16 is connected to the light source device 20 and the control device 22 via a universal plug 62. A medical support device 24 and a receiving device 64 are connected to the control device 22. Furthermore, in addition to the processing device 2, a display device 18 is also connected to the medical support device 24. That is, the control device 22 is connected to the processing device 2 and the display device 18 via the medical support device 24.

[0102] Here, the medical support device 24 is illustrated as an external device used to extend the functions performed by the control device 22. Therefore, an example is given where the control device 22 and the display device 18 are indirectly connected via the medical support device 24, but this is only one example. For example, the display device 18 may also be directly connected to the control device 22. In this case, for example, it is sufficient that the functions of the medical support device 24 are carried over to the control device 22.

[0103] The receiving device 64 receives commands from the doctor 12 and outputs the received commands as electrical signals to the control device 22. Examples of receiving devices 64 include keyboards, mice, touch panels, foot switches, microphones, and / or remote operation devices.

[0104] The control device 22 controls the light source device 20, or transmits and receives various signals with the camera 52, or transmits and receives various signals with the medical support device 24.

[0105] The light source device 20 emits light under the control of the control device 22 and supplies the light to the lighting device 54. A light guide is built into the lighting device 54, and the light supplied from the light source device 20 passes through the light guide and illuminates the lighting lenses 55A and 55B. The light illuminated by the lighting lenses 55A and 55B is light 30 (reference). Figure 1 The control device 22 causes the camera 52 to capture images, obtaining dynamic images 39 of the endoscope from the camera 52 (see reference). Figure 1 It is then output to the designated output destination (e.g., medical support device 24).

[0106] The medical support device 24 provides medical (here, for example, endoscopy) support by performing various image processing on the endoscopic dynamic image 39 input from the control device 22. The medical support device 24 outputs the endoscopic dynamic image 39 with various image processing performed to a designated output destination (e.g., display device 18).

[0107] Furthermore, the example described here is that the endoscopic dynamic image 39 output from the control device 22 is output to the display device 18 via the medical support device 24, but this is only one example. For example, the control device 22 may be connected to the display device 18, and the endoscopic dynamic image 39, for which image processing is performed by the medical support device 24, may be displayed on the display device 18 via the control device 22.

[0108] Figure 3 This is a block diagram illustrating an example of the hardware structure of the electrical system of the endoscope device 10. For example... Figure 3 As shown, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage device 76. The processor 72, memory 74, storage device 76, and external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and storage device 76 are used by the processor 72.

[0109] External I / F 70 is responsible for the transmission and reception of various information between the processor 72 and one or more devices located outside the control device 22 (hereinafter also referred to as "the first external device").

[0110] A camera 52 is connected to an external I / F 70 as one of the first external devices. The external I / F 70 is responsible for transmitting and receiving various information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. Furthermore, the processor 72 acquires information via the external I / F 70 about the large intestine 28 (reference) through the camera 52. Figure 1 Endoscopic dynamic images obtained by taking pictures inside the endoscopic cavity 39 (reference) Figure 1 ).

[0111] A light source device 20 is connected to an external I / F 70 as one of the first external devices. The external I / F 70 is responsible for transmitting and receiving various information between the light source device 20 and the processor 72. Under the control of the processor 72, the light source device 20 supplies light to the lighting device 54. The lighting device 54 illuminates the light supplied from the light source device 20.

[0112] A receiving device 64 is connected to the external I / F 70 as one of the first external devices. The processor 72 obtains the command received by the receiving device 64 via the external I / F 70 and executes the processing corresponding to the obtained command.

[0113] The medical support device 24 includes a computer 78 and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage device 86. The processor 82, memory 84, storage device 86, and external I / F 80 are connected to a bus 88. In this embodiment, the computer 78 is an example of a "computer" according to the present invention, and the processor 82 is an example of a "processor" according to the present invention.

[0114] Furthermore, the hardware structure of computer 78 (i.e., processor 82, memory 84, and storage device 86) is basically the same as that of computer 66, so the description related to the hardware structure of computer 78 is omitted here.

[0115] External I / F 80 is responsible for the transmission and reception of various information between the processor 82 and one or more devices located outside the medical support device 24 (hereinafter also referred to as "second external device").

[0116] A control device 22 is connected to the external I / F80 as one of the second external devices. Figure 3 In the example, an external I / F 70 of the control device 22 is connected to the external I / F 80. The external I / F 80 is responsible for transmitting and receiving various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 acquires the endoscopic dynamic image 39 (see reference) from the processor 72 of the control device 22 via the external I / F 70 and 80. Figure 1), and perform various image processing on the acquired endoscopic dynamic images 39.

[0117] A display device 18 is connected to an external I / F 80 as one of the second external devices. The processor 82 controls the display device 18 via the external I / F 80, thereby causing the display device 18 to display various information (e.g., endoscopic dynamic images 39 that have undergone various image processing, etc.).

[0118] An external I / F 80 is connected via network 3 to a processing device 2 as one of the second external devices. The processor 82 transmits and receives various types of information with the processing device 2 via the external I / F 80. For example, the external I / F 80 sends an endoscopic motion image 39 acquired by the processor 82 from the camera 52 to the processing device 2. The processing device 2 receives the endoscopic motion image 39 sent from the external I / F 80, performs processing using at least one frame 40 contained in the received endoscopic motion image 39, and sends the processing result to the medical support device 24. The external I / F 80 receives the processing result sent from the processing device 2. The processor 82 acquires the processing result received from the external I / F 80.

[0119] As an example, such as Figure 4 As shown, in frame 40 displayed in the first display area 35A, dirt 90 adhering to the optical system 55 (e.g., illumination lenses 55A and / or 55B) may sometimes be reflected, or a faulty part 92 of the optical system 55 (e.g., illumination lenses 55A and / or 55B) may be reflected. If dirt 90 or faulty part 92 is reflected in frame 40, it may hinder the doctor 12 from performing medical identification and / or treatment.

[0120] Dirt 90 adhering to the optical system 55 can be wiped off by hospital staff such as doctors 12 or nurses 14, but malfunctions of the optical system 55 are difficult for hospital staff to handle and require professional work such as replacement or repair of the optical system 55 by professionals who handle optical systems (hereinafter also referred to as "professionals").

[0121] Therefore, a method is required to accurately diagnose whether a malfunction of the optical system 55 is one that does not require professional personnel to recover the optical system 55, or whether a malfunction of the optical system 55 is one that requires professional personnel to recover the optical system 55 (in other words, whether the object reflected in frame 40 is dirt 90 that is easy for hospital staff to handle, or a faulty part 92 that is difficult for hospital staff to handle).

[0122] Therefore, in view of this situation, in the diagnostic system 1 according to this embodiment, optical system diagnosis is performed according to the following principles. Optical system diagnosis refers to the diagnosis of optical system 55.

[0123] exist Figure 5 The diagram shows a conceptual illustration of an example of the structure of the cover 94 and accessory 96 used for optical system diagnostics.

[0124] A cover 94 is attached to the front end 50 via an attachment 96. The cover 94 is formed by a double-layer structure consisting of an inner layer 94A (in other words, the inner cover) and an outer layer 94B (in other words, the outer cover). Polyoxymethylene (POM) can be used as an example of the material for the inner layer 94A and the outer layer 94B. Both the inner layer 94A and the outer layer 94B are bottomed cylindrical covers. The inner layer 94A is white. The outer layer 94B is black, and by fitting the outer layer 94B over the inner layer 94A, external light is absorbed by the outer layer 94B. That is, since the outer surface of the cover 94 is black, external light is absorbed at the outer surface of the cover 94. Furthermore, a hollow region 97 is provided between the inner layer 94A and the outer layer 94B, which suppresses the change in the hue of the inner layer 94A due to the black color of the outer layer 94B. Inner layer 94A is an example of an "inner layer" as described in this invention, outer layer 94B is an example of an "outer layer" as described in this invention, and hollow region 97 is an example of a "hollow region" as described in this invention.

[0125] An annular flange 94B1 is formed on the outer layer 94B. The flange 94B1 has an annular protrusion 94B1a that protrudes in the direction of the inner diameter of the outer layer 94B (in other words, it protrudes perpendicularly to the axial side of the outer layer 94B). An annular groove 94A1a is formed on the outer peripheral surface 94A1 of the inner layer 94A. If the outer layer 94B is covered by the inner layer 94A in such a way that the inner side of the outer layer 94B covers the outer side of the inner layer 94A, the protrusion 94B1a of the flange 94B1 is embedded in the groove 94A1a of the outer peripheral surface 94A1 of the inner layer 94A. Thus, the inner layer 94A is fixed to the outer layer 94B. In addition, here is an example of fixing the inner layer 94A to the outer layer 94B by the protrusion 94B1a being embedded in the groove 94A1a, but this is only one example, and the structure in which the outer layer 94B covers the inner layer 94A and is fixed can also be other structures.

[0126] The inner layer 94A has an annular edge 94A2 with a diameter smaller than that of the flange 94B1. The edge 94A2 protrudes outward from the outer layer 94B along the axial direction of the outer layer 94B. The inner layer 94A has an opening 94A3. The opening 94A3 is formed in a circular shape through the edge 94A2.

[0127] The inner layer 94A has an annular sleeve 94A4. The sleeve 94A4 is the portion of the inner layer 94A that protrudes outward from the outer layer 94B along the axial direction of the inner layer 94A, including the edge 94A2.

[0128] The inner surface 98 of the inner layer 94A (i.e., the bottom of the inner side of the inner layer 94A) is formed as a curved surface that is concave inward when viewed from the opening 94A3 side. As an example of a curved surface, an integral sphere can be given. Here, an example of the inner surface 98 being formed as a curved surface is given, but this is only one example, and the inner surface 98 can also be formed as a planar surface.

[0129] Inside, 98 has a first region 100 and a second region 102. The first region 100 is for reflected light 30 (reference). Figure 1 The region 100 is defined as follows: White light is used as an example of the light 30 reflected in region 100, but this is only one example; other types of light may also be present. Region 102 is a region with a lower reflectivity than region 100. The area of ​​region 100 is larger than the area of ​​region 102. Furthermore, the spatial frequency of region 102 is higher than the spatial frequency of region 100. Region 102 is a test pattern. Figure 5 In the example shown, region 102, a cross-shaped test pattern with a ring mark at the center is displayed.

[0130] The inner surface 98 is a surface for diffusing the light 30 emitted from the endoscope observer 16. In this embodiment, the inner surface 98 is sandblasted to diffuse the light 30.

[0131] Annex 96 includes a cylindrical structure 104 and a sliding component 106. The cylindrical structure 104 has openings on its upper and lower sides, extending from the upper side to the lower side. The cylindrical structure 104 is made of a material capable of radial expansion and contraction (here, as an example, resin). The cylindrical structure 104 is black in color. The cylindrical structure 104 is formed in a cylindrical shape, and the inner diameter of the main body of the cylindrical structure 104 gradually increases from the upper side to the lower side.

[0132] A first annular flange 104A is formed at the upper end of the cylindrical structure 104, and a second annular flange 104B with a smaller diameter than the first flange 104A is formed at the lower end of the cylindrical structure 104.

[0133] A circular opening 104A1 for mounting the cover 94 to the cylindrical structure 104 is formed at the center of the first flange 104A. The diameter of the opening 104A1 is such that the sleeve 94A4 can be inserted and the inner circumferential surface of the first flange 104A can be made to fit tightly with the outer circumferential surface of the sleeve 94A4 inserted into the opening 104A1.

[0134] A circular opening 104B1 for inserting the front end portion 50 of the endoscope 16 is formed in the center of the second flange 104B. The cylindrical structure 104 extends from the opening 104B1 to the opening 104A1.

[0135] At the lower part of the cylindrical structure 104 (i.e., on the side of the second flange 104B), a plurality of notches 104C are formed at certain intervals around the axis of the cylindrical structure 104 (i.e., in the circumferential direction of the cylindrical structure 104). The notches 104C are formed in a linear shape from the side of the second flange 104B toward the side of the first flange 104A along the axial direction of the cylindrical structure 104.

[0136] A sliding member 106 is mounted between the first flange 104A and the second flange 104B on the outer surface of the cylindrical structure 104, in a manner that allows it to slide along the axial direction of the cylindrical structure 104. The sliding member 106 is formed of an elastic material such as resin. By sliding the sliding member 106 along the axial direction of the cylindrical structure 104, the cylindrical structure 104 expands and contracts radially.

[0137] The cover 94 is installed in the cylindrical structure 104. That is, the sleeve 94A4 is inserted into the opening 104A1 by overcoming the pressure from the inner circumferential surface of the first flange 104A (i.e., the sleeve 94A4 is pressed into the first flange 104A) until the flange 94B1 contacts the first flange 104A, thereby installing the cover 94 in the cylindrical structure 104. Here, an example is given in which the outer circumferential surface of the sleeve 94A4 is pressed by the inner circumferential surface of the first flange 104A, so that the outer circumferential surface of the sleeve 94A4 is in close contact with the inner circumferential surface of the first flange 104A. However, this is only one example. For example, one of the internal threads and the external threads may be formed on the outer circumferential surface of the sleeve 94A4, and the other of the internal threads and the external threads may be formed on the inner circumferential surface of the first flange 104A. Using the threaded structure, the outer circumferential surface of the sleeve 94A4 is embedded into the inner circumferential surface of the first flange 104A.

[0138] The front end portion 50 of the endoscope 16 is inserted into the cylindrical structure 104 through the opening 104B1. Then, with the front end portion 50 of the endoscope 16 inserted into the cylindrical structure 104, the diameter of the portion connecting the cover 94 and the front end portion 50 in the accessory 96 is adjusted by sliding the sliding member 106 along the axial direction of the cylindrical structure 104. For example, by sliding the sliding member 106 towards the second flange 104B along the axial direction of the cylindrical structure 104, the diameter of the portion connecting the cover 94 and the front end portion 50 becomes smaller. As a result, for example, Figure 6 As shown, the front end portion 50 of the endoscope observer 16 is clamped by the sliding member 106 via the cylindrical structure 104 surrounding the front end portion 50 and thus fits tightly against the inner surface of the cylindrical structure 104.

[0139] Thus, by attaching the cover 94 to the front end 50 via attachment 96, with the front end face 50A facing the inner surface 98, a light-tight space is formed within the cover 94. In this state, if light 30 is shone from the illumination lenses 55A and 55B of the front end 50, the light 30 is reflected in the inner surface 98 of the cover 94. The reflected light 30A obtained by the reflection of light 30 in the inner surface 98 is captured by the camera 52. That is, the inner surface 98 is captured by the camera 52. In this way, a diagnostic frame 40A is generated by capturing the inner surface 98 by the camera 52.

[0140] Diagnostic frame 40A is used for diagnosing the optical system 55. If dirt 90 adheres to the optical system 55 (e.g., illumination lenses 55A and / or 55B), the dirt 90 is reflected in diagnostic frame 40A. If the optical system 55 (e.g., illumination lenses 55A and / or 55B) malfunctions, the faulty portion 92 is reflected in diagnostic frame 40A. Thus, if dirt 90 and / or faulty portion 92 are reflected in diagnostic frame 40A, the optical system 55 is diagnosed as malfunctioning (i.e., there is a defect). Conversely, if neither dirt 90 nor faulty portion 92 is reflected in diagnostic frame 40A, the optical system 55 is diagnosed as normal. In this embodiment, diagnostic frame 40A is an example of the "internal image" according to the present invention.

[0141] To achieve this optical system diagnostic, in this embodiment, as an example, such as Figure 7 As shown, diagnostic processing is performed by the processor 82 of the medical support device 24. Additionally, Figure 7 This is a block diagram illustrating one example of the main functions of the processor 82 included in the medical support device 24 and one example of the information stored in the storage device 86.

[0142] A diagnostic program 108 is stored in the storage device 86. The diagnostic program 108 is an example of a "program" according to the present invention. The processor 82 performs diagnostic processing by reading the diagnostic program 108 from the storage device 86 and executing the read diagnostic program 108 on the memory 84. The diagnostic processing is implemented by the processor 82 operating as the execution unit 82A and the control unit 82B according to the diagnostic program 108 executed on the memory 84.

[0143] Figure 8 This is a conceptual diagram illustrating an example of the processing performed by the medical support device 24 and the processing device 2 with the cover 94 installed on the front end 50 of the endoscope 16 via attachment 96. Furthermore, here, the state of the front end 50 with the cover 94 installed via attachment 96 refers to the state where the front end face 50A inserted into attachment 96 faces the inner surface 98 and a light-tight space is formed within the cover 94 (i.e., the front end 50 is light-tightly closed by the cover 94 and attachment 96 with the front end face facing the inner surface 98).

[0144] like Figure 8 As shown, the execution unit 82A acquires diagnostic frames 40A generated by the camera 52 shooting at a shooting frame rate (e.g., tens of frames / second) in a time sequence from the camera 52 on a unit of one frame. Here, a diagnostic frame 40A contained in a moving image is shown as an example, but this is only one example, and a diagnostic frame 40A generated as a still image can also be used.

[0145] The execution unit 82A indirectly performs optical system diagnostics using the processing device 2. That is, the execution unit 82A requests the processing device 2 to perform optical system diagnostics and receives from the processing device 2 the results obtained by performing optical system diagnostics by the processing device 2.

[0146] If the execution unit 82A acquires a diagnostic frame 40A from the camera 52, it generates a request message 110 and sends it to the processing device 2 via an external I / F 80. The request message 110 is a request to the processing device 2 to perform optical system diagnostics. The request message 110 includes the diagnostic frame 40A acquired by the execution unit 82A.

[0147] The processing device 2 receives a request message 110 sent from the execution unit 82A. Upon receiving the request message 110, the processing device 2 performs an AI-based image analysis on the diagnostic frame 40A included in the request message 110 to diagnose the optical system 55. The processing device 2 performs an AI-based image analysis, including an identification process 112, a defect presence / absence determination process 114, and a type determination process 116. In this embodiment, based on the request from the execution unit 82A, the processing device 2 performs the identification process 112, the defect presence / absence determination process 114, and the type determination process 116, thereby achieving the diagnosis of the optical system 55. In this embodiment, the identification process 112, the defect presence / absence determination process 114, and the type determination process 116 are examples of "AI-based image analysis" according to the present invention. Furthermore, in this embodiment, the defect presence / absence determination process 114 is an example of "defect presence / absence determination processing" according to the present invention, and the type determination process 116 is an example of "type determination processing" and "location determination processing" according to the present invention.

[0148] The processing device 2 has an identification model 118. In the processing device 2, as identification processing 112, processing using the identification model 118 is performed. The processing device 2 identifies the types of malfunctions of the optical system 55, including whether the optical system 55 is functioning properly, by performing identification processing 112 on the diagnostic frame 40A included in the request information 110, and generates an identification result 120. The types of malfunctions of the optical system 55, including whether the optical system 55 is functioning properly, are identified by the identification model 118 with confidence (e.g., probability).

[0149] Recognition model 118 is a learned model for object recognition using an AI-based bounding box method. Recognition model 118 is optimized by machine learning on a neural network using first training data. The first training data is a dataset containing multiple data (i.e., multiple frames of data) obtained by establishing a correspondence between the first example data and the first correct answer data.

[0150] The first example of the data is an image of a hypothetical diagnostic frame 40A. As a first example of an image of a hypothetical diagnostic frame 40A, an image obtained by a camera (i.e., a camera of the same specifications as camera 52) capturing the interior 98 with a cover 94 attached to the front end (i.e., the portion corresponding to the front end 50) of an endoscope of the same specifications as the endoscope observer 16 via attachment 96. As a second example of an image of a hypothetical diagnostic frame 40A, a virtually created image (e.g., an image generated by a generative AI) can be cited.

[0151] The first correct answer data is the correct answer data (i.e., the annotation) for the first example data. Here, as an example of the first correct answer data, an annotation can be given indicating whether a defect corresponding to the optical system 55 (e.g., a defect corresponding to dirt 90 and / or faulty part 92) is reflected in the image used as the first example data, and an annotation that can determine the type of defect in the optical system 55 and the location of the defect in the optical system 55.

[0152] Examples of defects in the optical system 55 include dirt 90, fogging (e.g., fogging adhering to the lens included in the optical system 55), stains (e.g., stains adhering to the lens included in the optical system 55), peeling (e.g., peeling of the coating on the surface of the lens included in the optical system 55, peeling of the bonding lens included in the optical system 55 and / or peeling of the lens included in the optical system 55 from the peripheral components), and lens scratches (e.g., scratches adhering to the lens included in the optical system 55).

[0153] The defects in the optical system 55 can be broadly categorized into those that do not require recycling and those that do require recycling. Defects that do not require recycling are those in the optical system 55 that do not need to be recycled. Defects that require recycling are those in the optical system 55 that require recycling. For example, dirt 90 can be considered a defect that does not require recycling. For example, defects other than dirt 90, i.e., malfunctions, can be considered. Examples of malfunctions include fogging, stains, peeling, and lens scratches. Here, the defect that does not require recycling is an example of the "first type" according to this invention, and the defect that requires recycling is an example of the "second type" according to this invention.

[0154] Processing device 2 acquires diagnostic frame 40A from request information 110 and inputs the acquired diagnostic frame 40A into recognition model 118. Thereby, recognition model 118, based on the input diagnostic frame 40A, identifies the type of malfunction of optical system 55, including whether optical system 55 is functioning properly, and generates recognition result 120. Recognition result 120 is an example of "information as the basis for diagnostic results" as described in this invention.

[0155] The "Damage Determination Process 114" determines whether the optical system 55 has any defects based on the recognition result 120 generated by the recognition model 118. If the "Damage Determination Process 114" determines that the optical system 55 has no defects, the processing device 2 generates "No Defect Information 122" indicating that the optical system 55 has no defects and sends it to the medical support device 24. The "No Defect Information 122" includes a diagnostic frame 40A input to the recognition model 118 to obtain the recognition result 120 used to determine that the optical system 55 has no defects. The "No Defect Information 122" is received by the external I / F 80 of the medical support device 24. Then, the execution unit 82A acquires the "No Defect Information 122" received from the external I / F 80. Details will be described later, but if the execution unit 82A acquires the "No Defect Information 122," the control unit 82B performs processing based on the "No Defect Information 122." In this embodiment, the "No Defect Information 122" is an example of the "diagnostic result" according to the present invention.

[0156] If the optical system 55 is determined to have a defect through the defect presence / absence determination process 114, the processing device 2 executes the type determination process 116. The type determination process 116 is a process that determines the type of the defect in the optical system 55 based on the recognition result 120 generated by the recognition model 118 and determines the location where the defect in the optical system 55 occurs on the diagnostic frame 40A. The type determination process 116 determines the location of the defect in the optical system 55 on the diagnostic frame 40A by obtaining location determination information 124 from the recognition result 120, which determines the location where the defect in the optical system 55 occurs on the diagnostic frame 40A. The location determination information 124 includes a bounding box BB that determines the location where the defect in the optical system 55 occurs on the diagnostic frame 40A. In the type determination process 116, the type of the defect in the optical system 55 is determined based on the recognition result 120, and defect type information 126 representing the type of defect in the optical system 55 (e.g., dirt 90, fogging, stains, peeling, or lens scratches, etc.) is generated.

[0157] The classification process 116 includes a differentiation process 116A. Differentiation process 116A classifies the defects of the optical system 55 into types that do not require recycling and types that require recycling. This distinction is made based on the defect type information 126. For example, if dirt 90 is identified as a defect type of the optical system 55 based on the defect type information 126, the defect type of the optical system 55 is classified as a type that does not require recycling. If, based on the defect type information 126, a type other than dirt 90 is identified as a defect type of the optical system 55, the defect type of the optical system 55 is classified as a type that requires recycling.

[0158] By performing the type determination process 116 in this way, the type of defect in the optical system 55 is determined, the location where the defect in the optical system 55 occurs is determined on the diagnostic frame 40A, and the type of defect in the optical system 55 is classified into a type that does not need to be recovered or a type that needs to be recovered. The processing device 2 generates defect information 128 as a processing result based on the type determination process 116 and sends it to the medical support device 24.

[0159] The defect information 128 includes location determination information 124 and defect type information 126. Furthermore, the defect information 128 includes either "non-recoverable" type information 130 or "recoverable" type information 132. "Non-recoverable" type information 130 indicates that the defect type of the optical system 55 has been classified as non-recoverable through the differentiation process 116A, while "recoverable" type information 132 indicates that the defect type of the optical system 55 has been classified as recoverable through the differentiation process 116A.

[0160] The adverse condition information 128 sent by the processing device 2 is received by the external I / F 80 of the medical support device 24. Then, the execution unit 82A acquires the adverse condition information 128 received from the external I / F 80. Details will be described later, but if the execution unit 82A acquires the adverse condition information 128, the control unit 82B performs processing based on the adverse condition information 128. In this embodiment, the adverse condition information 128 is an example of the "diagnostic result" involved in the present invention.

[0161] Figure 9 This is a conceptual diagram illustrating an example of display control performed by the control unit 82B on the display device 18 when the execution unit 82A obtains information 122 indicating no malfunction or information 128 indicating a malfunction. (Example) Figure 9As shown, when the execution unit 82A obtains the no-defect information 122, the control unit 82B outputs information that visualizes the no-defect information 122 obtained by the diagnosis performed by the optical system 55 by the processing device 2 (that is, the no-defect information 122 obtained by performing the identification process 112 and the no-defect determination process 114).

[0162] That is, when the execution unit 82A obtains the no-defect information 122, the control unit 82B displays the diagnostic frame 40A included in the no-defect information 122 in the first display area 35A, and displays the no-defect message 134 as one of the auxiliary information 44 in the second display area 35B. The no-defect message 134 is a message indicating that there are no defects in the optical system 55. The diagnostic frame 40A displayed in the first display area 35A and the no-defect message 134 displayed in the second display area 35B are examples of information in which the no-defect information 122 is visualized.

[0163] Furthermore, the example shown here is of a no-defect message 134 being displayed in the second display area 35B, but this is only one example. Marks and / or codes indicating that there are no defects in the optical system 55 may also be displayed in the second display area 35B. Additionally, an audio signal indicating that there are no defects in the optical system 55 may be output from a speaker (not shown). Furthermore, the no-defect information 122 and / or information generated based on the no-defect information 122 (e.g., no-defect message 134, etc.) may be stored in a storage area (e.g., storage device 86, etc.).

[0164] In contrast, when the execution unit 82A obtains the defect information 128, the control unit 82B outputs information that visualizes the defect information 128 obtained by the diagnosis performed by the optical system 55 by the processing device 2 (i.e., the defect information 128 obtained by performing the identification process 112, the defect presence / absence determination process 114, and the type determination process 116).

[0165] That is, when the execution unit 82A obtains the defect information 128, the control unit 82B displays the diagnostic frame 40A contained in the defect information 128 in the first display area 35A, and displays the defect message 136 as one of the auxiliary information 44 in the second display area 35B. The control unit 82B displays the bounding box BB overlaid on the diagnostic frame 40A displayed in the first display area 35A based on the position determination information 124 contained in the defect information 128.

[0166] exist Figure 9The example shown illustrates a bounding box BB displayed on the area where existing dirt 90 is projected onto the diagnostic frame 40A. However, if a fault in the optical system 55 is determined based on the fault type information 126, the control unit 82B displays the faulty area 92 on the diagnostic frame 40A (see reference). Figure 4 The bounding boxes BB are overlapped. In this case, the control unit 82B displays the bounding box BB displayed on the area showing dirt 90 differently from the bounding box BB displayed on the faulty area 92, so that the bounding box BB displayed on the area showing dirt 90 and the bounding box BB displayed on the faulty area 92 can be visually distinguished. For example, the control unit 82B can visually distinguish the bounding box BB displayed on the area showing dirt 90 and the bounding box BB displayed on the faulty area 92 by changing the line thickness, line type, brightness and / or color of the bounding box BB. Furthermore, it is also possible to consider the case where more than one dirt 90 and more than one faulty area 92 are shown in the diagnostic frame 40A. In this case, the same principle can be used to make the display mode of the bounding box BB displayed on the area showing dirt 90 different from the bounding box BB displayed on the faulty area 92.

[0167] Message 136 indicates a malfunction in the optical system 55. The diagnostic frame 40A displayed in the first display area 35A and the message 136 displayed in the second display area 35B are examples of information obtained by visualizing the malfunction information 128.

[0168] The defect message 136 is categorized into dirt message 136A and fault message 136B. When the execution unit 82A receives the defect information 128, either dirt message 136A or fault message 136B is displayed in the second display area 35B. If the defect type information 126 included in the defect information 128 indicates dirt 90, the control unit 82B displays dirt message 136A in the second display area 35B. If the defect type information 126 included in the defect information 128 indicates a fault (e.g., fogging, stains, peeling, or lens scratches), the control unit 82B displays fault message 136B in the second display area 35B.

[0169] The dirt message 136A includes a first message 136A1 indicating that dirt 90 is attached to the optical system 55, a second message 136A2 recommending wiping the dirt 90, and a third message 136A3 indicating that the optical system 55 does not need to be recycled. The first message 136A1 and the second message 136A2 are generated by the control unit 82B based on the defect type information 126 included in the defect information 128. That is, the first message 136A1 and the second message 136A2 can be considered as information obtained by visualizing the defect type information 126. The third message 136A3 is generated by the control unit 82B based on the non-recyclable type information 130 included in the defect information 128. That is, the third message 136A3 can be considered as information obtained by visualizing the non-recyclable type information 130.

[0170] The fault message 136B includes a fourth message 136B1 indicating a fault in the optical system 55, a fifth message 136B2 determining the type of fault in the optical system 55, and a sixth message 136B3 indicating that the optical system 55 needs to be retrieved by a professional. The fourth message 136B1 and the fifth message 136B2 are generated by the control unit 82B based on the fault type information 126 included in the fault information 128. That is, the fourth message 136B1 and the fifth message 136B2 can be considered as information obtained by visualizing the fault type information 126. The sixth message 136B3 is generated by the control unit 82B based on the retrieval requirement information 132 included in the fault information 128. That is, the sixth message 136B3 can be considered as information obtained by visualizing the retrieval requirement information 132.

[0171] Furthermore, while the example shown here is of a malfunction message 136 being displayed in the second display area 35B, this is merely one example. Marks and / or codes indicating a malfunction in the optical system 55 may also be displayed in the second display area 35B. Additionally, sounds indicating the first message 136A1, the second message 136A2, the third message 136A3, the fourth message 136B1, the fifth message 136B2, and / or the sixth message 136B3 may be output from a speaker (not shown). Furthermore, the malfunction information 128 and / or information generated based on the malfunction information 128 (e.g., the first message 136A1, the second message 136A2, the third message 136A3, the fourth message 136B1, the fifth message 136B2, and / or the sixth message 136B3, etc.) may be stored in a storage area (e.g., a storage device 86, etc.).

[0172] The content displayed by the control unit 82B on the first display area 35A and the second display area 35B (i.e., the diagnostic results of the optical system 55) is evaluated by an optical system professional (hereinafter also referred to as "professional"). For example, the professional evaluates the appropriateness of the content displayed on the first display area 35A and the second display area 35B (i.e., the diagnostic results of the optical system 55). The evaluation results performed by the professional are received by the receiving device 64 and the like. The execution unit 82A causes the processing device 2 to perform relearning of the recognition model 118 based on the evaluation results received by the receiving device 64 and the like. As a result, the recognition model 118 is strengthened.

[0173] Next, refer to Figure 10 The function of the parts of the diagnostic system 1 involved in this invention will be explained. Figure 10 The diagnostic process shown is an example of the "diagnostic method" involved in this invention.

[0174] exist Figure 10 In the diagnostic process shown, firstly, in step ST10, the execution unit 82A determines whether, with the cover 94 installed on the front end 50 of the endoscope 16 via the attachment 96, a one-frame image with the interior 98 as the subject has been captured by the camera 52. If, in step ST10, a one-frame image with the interior 98 as the subject has not been captured by the camera 52 with the cover 94 installed on the front end 50 of the endoscope 16 via the attachment 96, the determination is negative, and the diagnostic process proceeds to step ST20. If, in step ST10, a one-frame image with the interior 98 as the subject has been captured by the camera 52 with the cover 94 installed on the front end 50 of the endoscope 16 via the attachment 96, the determination is positive, and the diagnostic process proceeds to step ST12.

[0175] In step ST12, the execution unit 82A acquires the diagnostic frame 40A obtained by the camera 52 capturing the image inside 98. After the processing in step ST12 is performed, the diagnostic process proceeds to step ST14.

[0176] In step ST14, the execution unit 82A sends the request information 110, including the diagnostic frame 40A acquired in step ST12, to the processing device 2 via the external I / F 80. After the processing in step ST14 is executed, the diagnostic processing proceeds to step ST16.

[0177] If the request information 110 is sent to the processing device 2 through the processing of step ST14, the processing device 2 performs identification processing 112 on the diagnostic frame 40A included in the request information 110, and performs a defect presence / absence determination processing 114 using the identification result 120. If the defect presence / absence determination processing 114 determines that there is no defect in the optical system 55, the defect absence information 122 is sent as a diagnostic result to the medical support device 24. Furthermore, if the defect presence / absence determination processing 114 determines that there is a defect in the optical system 55, the type determination processing 116 using the identification result 120 is performed. The type determination processing 116 generates defect presence information 128, which is sent as a diagnostic result to the medical support device 24.

[0178] Therefore, in step ST16, the execution unit 82A determines whether the external I / F 80 has received a diagnostic result sent from the processing device 2 (i.e., no adverse condition information 122 or adverse condition information 128). If, in step ST16, the external I / F 80 has not received a diagnostic result sent from the processing device 2, the determination is rejected, and step ST16 is performed again. If, in step ST16, the external I / F 80 has received a diagnostic result sent from the processing device 2, the determination is affirmative, and the diagnostic process proceeds to step ST18. If the external I / F 80 has received a diagnostic result sent from the processing device 2, the execution unit 82A acquires the diagnostic result received from the external I / F 80.

[0179] In step ST18, the control unit 82B visualizes the diagnostic results (i.e., no adverse condition information 122 or adverse condition information 128) obtained by the execution unit 82A and displays them on screen 35 (see reference). Figure 9 After the processing in step ST18 is performed, the diagnostic process moves to step ST20.

[0180] In step ST20, the control unit 82B determines whether the conditions for ending the diagnostic process are met. As an example of the conditions for ending the diagnostic process, a condition in which a command to end the diagnostic process is given to the diagnostic system 1 (for example, the condition in which the command to end the diagnostic process is received by the receiving device 64) can be given.

[0181] In step ST20, if the conditions for ending the diagnostic process are not met, the determination is negative, and the diagnostic process proceeds to step ST10. In step ST20, if the conditions for ending the diagnostic process are met, the determination is positive, and the diagnostic process ends.

[0182] As explained above, in diagnostic system 1, the diagnostic frame 40A obtained by camera 52 capturing the interior 98 of the endoscope 16 with the cover 94 attached to the front end 50 via accessory 96 is subjected to AI-based image analysis, thereby performing a diagnosis of the optical system 55. The diagnostic results obtained by performing the diagnosis are displayed on screen 35. The diagnostic results displayed on screen 35 are either information showing no adverse conditions 122 or information showing adverse conditions 128. Therefore, doctors 12 and other professionals can visually identify the information showing no adverse conditions 122 or the information showing adverse conditions 128 through screen 35, thereby enabling a highly accurate diagnosis of the optical system 55.

[0183] Furthermore, in the diagnostic system 1, the processing device 2 performs a defect determination process 114. This defect determination process 114 determines whether the optical system 55 has any defects based on the recognition result 120 generated by the recognition model 118. If the defect determination process 114 determines that the optical system 55 has a defect, the processing device 2 generates defect information 128 and sends it to the medical support device 24. If the defect determination process 114 determines that the optical system 55 does not have any defects, the processing device 2 generates no-defect information 122 and sends it to the medical support device 24. The no-defect information 122 or the defect information 128 is displayed as visualized information on the screen 35. This allows doctors 12 and other professionals to determine whether the optical system 55 has any defects.

[0184] Furthermore, in the diagnostic system 1, the processing device 2 performs a type determination process 116. The type determination process 116 includes determining the type of malfunction of the optical system 55 based on the recognition result 120 generated by the recognition model 118. The malfunctions of the optical system 55 are classified into types that do not require recovery and types that require recovery. Types that do not require recovery are those where the optical system 55 does not need to be recovered. Types that require recovery are those where the optical system 55 needs to be recovered. The type determination process 116 includes a differentiation process 116A, which, by performing the differentiation process 116A, differentiates the type of malfunction occurring in the optical system 55 into types that do not require recovery and types that require recovery, and the differentiation result is displayed on the screen 35. Therefore, when a malfunction occurs in the optical system 55, the doctor 12 and other professionals can determine whether a malfunction has occurred where the optical system 55 does not require recovery or where the optical system 55 requires recovery.

[0185] Here, dirt 90 can be cited as an example of a type that does not require recycling, while a malfunction can be cited as an example of a type that requires recycling. Furthermore, the screen 35 displays whether the malfunction occurring in the optical system 55 is dirt 90 or a malfunction. Therefore, when a malfunction occurs in the optical system 55, doctors 12 and other professionals can determine whether the malfunction is dirt or a malfunction.

[0186] Furthermore, in the diagnostic system 1, AI-based image analysis of the diagnostic frame 40A is achieved by inputting the diagnostic frame 40A into the recognition model 118 to generate a recognition result 120. Thus, compared to situations where the type of malfunction of the optical system 55 is determined solely by human intuition and experience, including whether the optical system 55 is functioning correctly (e.g., determining the type of malfunction of the optical system 55, including whether the optical system 55 is functioning correctly, while visually confirming the diagnostic frame 40A), the type of malfunction of the optical system 55 is determined quickly and with high accuracy, including whether the optical system 55 is functioning correctly.

[0187] Furthermore, in the diagnostic system 1, the diagnostic results of the optical system 55 are reviewed and evaluated by a professional through the screen 35. The evaluation results from the professional are received by the receiving device 64 and the like. The execution unit 82A causes the processing unit 2 to perform relearning of the recognition model 118 based on the evaluation results received by the receiving device 64 and the like. As a result, the recognition model 118 is strengthened. As a result, the diagnostic accuracy of the optical system 55 can be improved.

[0188] Furthermore, in the diagnostic system 1, the processing device 2 performs a type determination process 116, acquires location determination information 124, and sends it to the medical support device 24. The location determination information 124 is information that can determine the location of a malfunction in the optical system 55 on the diagnostic frame 40A, and is displayed as information in the first display area 35A (e.g., bounding box BB). Thus, doctors 12 and other professionals can determine the location of the malfunction in the optical system 55.

[0189] Furthermore, in the diagnostic system 1, the cover 94 is mounted to the front end 50 via the attachment 96. The inner surface 98 of the cover 94 has a first region 100 and a second region 102 formed as a test pattern. The second region 102 has a lower reflectivity than the first region 100, a smaller area than the first region 100, and a higher spatial frequency than the first region 100. In the diagnostic system 1, the optical system 55 is diagnosed using a diagnostic frame 40A generated by photographing the inner surface 98 thus configured by the camera 52. Therefore, it is possible to determine with high accuracy whether the optical system 55 has any defects and the type of defects in the optical system 55. In particular, dirt in defects generated on the surface of the lenses included in the optical system 55 tends to appear in a light gray gradient close to white on the screen 35 (here, the first display area 35A, for example). Therefore, when judging whether there is dirt only by the first region 100 (e.g., a white area) with a reflectivity higher than the first region 100, the accuracy of determining whether there is dirt is below a certain level. Therefore, in order to achieve a certain level of accuracy in determining whether something is dirty, in this embodiment, the second region 102, whose reflectivity is lower than that of the first region 100, is also included in the test pattern. This allows the accuracy of determining whether something is dirty to be improved to a certain level.

[0190] Furthermore, the inner surface 98 of the cover 94 is formed into a curved surface. Also, the inner surface 98 of the cover 94 is processed to allow light 30 to diffuse. Therefore, it is possible to suppress image quality issues caused by diffuse reflection of light within the cover 94 that render the image quality of the diagnostic frame 40A unsuitable for diagnosing optical system 55.

[0191] Furthermore, the cover 94 is formed as a double-layer structure. The outer surface of the cover 94 is black. Therefore, it can suppress the incidence of external light into the cover 94. Moreover, the double-layer structure of the cover 94 consists of an inner layer 94A and an outer layer 94B, with a hollow region 97 provided between the inner layer 94A and the outer layer 94B. This further enhances the effect of suppressing the incidence of external light into the cover 94.

[0192] Furthermore, the cover 94 is connected to the front end portion 50 of the endoscope observer 16 via the attachment 96, and the diameter of the portion connecting the cover 94 to the front end portion 50 can be adjusted via the attachment 96. Therefore, by adjusting the diameter of the portion connecting the cover 94 to the front end portion 50 via the attachment 96, the operation of installing the cover 94 onto the front end portion 50 can be simplified. Moreover, by adjusting the diameter of the portion connecting the cover 94 to the front end portion 50 via the attachment 96, the tightness of the seal between the front end portion 50 and the attachment 96 can be improved, thereby suppressing the incidence of external light into the cover 94 through the gap between the front end portion 50 and the attachment 96.

[0193] Furthermore, in the above embodiment, the example given is the input of the diagnostic frame 40A contained in the request information 110 sent from the medical support device 24 into the recognition model 118 by the processing device 2, but the present invention is not limited thereto. For example, such as Figure 11 As shown, the processing device 2 can perform noise removal processing 140 and edge extraction processing 142 on the diagnostic frame 40A in the stage before performing recognition processing 112. Figure 11 In the example shown, in addition to dirt 90, noise 138 is also reflected in diagnostic frame 40A. Noise removal processing 140 is the process of removing noise 138 from diagnostic frame 40A. Edge extraction processing 142 is performed to extract the edges of the image regions reflected in diagnostic frame 40A after noise 138 has been removed by noise removal processing 140. Processing device 2 inputs diagnostic frame 40A obtained by edge extraction processing 142 into recognition model 118. As a result, the accuracy of recognition processing 112 is improved, and consequently, the accuracy of diagnosing defects generated in optical system 55 is also improved.

[0194] Here, we give an example of performing noise removal processing 140 and edge extraction processing 142 in the pre-stage of recognition processing 112. However, this is only one example, and noise removal processing 140 or edge extraction processing 142 can also be performed in the pre-stage of recognition processing 112. In addition, it is more effective to perform noise removal processing 140 before edge extraction processing 142 than after edge extraction processing 142.

[0195] In the above embodiments, the use of recognition model 118 in recognition processing 112 has been described as an example, but the present invention is not limited thereto. For example, such as Figure 12 As shown, identification processing 112 can also replace identification model 118 by using area identification model 144, dirt identification model 146, and fault identification model 148. In this case, diagnostic frame 40A (in Figure 12 In the example shown, the diagnostic frame 40A, which has undergone noise removal processing 140 and edge extraction processing 142, is input into the region recognition model 144, the dirt recognition model 146, and the fault recognition model 148, respectively.

[0196] Region recognition model 144 is a learned model for object recognition using bounding boxes based on AI. Region recognition model 144 is optimized by machine learning on the neural network using second training data. The second training data is a dataset containing multiple data (i.e., multiple frames of data) obtained by establishing a correspondence between the second example data and the second correct answer data.

[0197] The second example data is the same image as the first example data described in the above embodiment. The second correct answer data is the correct answer data (i.e., annotation) for the second example data. Here, as an example of the second correct answer data, annotations that can determine the image region corresponding to the first region 100 in the image used as the second example data and annotations that can determine the image region corresponding to the second region 102 can be given.

[0198] Processing device 2 obtains diagnostic frame 40A from request information 110 and inputs the obtained diagnostic frame 40A into region recognition model 144. Thereby, region recognition model 144 identifies the first region 100 and the second region 102 reflected in the input diagnostic frame 40A.

[0199] The dirt recognition model 146 is a learned model for object recognition using an AI-based bounding box approach. The dirt recognition model 146 is optimized by machine learning on the neural network using a third training data. The third training data is a dataset containing multiple data sets (i.e., multiple frames of data) obtained by establishing a correspondence between the third example data and the third correct answer data.

[0200] The third example data is the same image as the first example data described in the above embodiment. The third correct answer data is the correct answer data (i.e., annotation) for the third example data. Here, as an example of the third correct answer data, an annotation that can determine the image area corresponding to the dirt 90 reflected in the image used as the third example data can be given.

[0201] The dirt recognition model 146 thus constructed determines the AI ​​of dirt 90 based on the diagnostic frame 40A (e.g., the AI ​​that determines the location of dirt 90 on the diagnostic frame 40A). The processing device 2 obtains the diagnostic frame 40A from the request information 110 and inputs the obtained diagnostic frame 40A into the dirt recognition model 146. Thus, the dirt recognition model 146 identifies the dirt 90 reflected in the input diagnostic frame 40A.

[0202] Fault recognition model 148 is a learned model for object recognition using an AI-based bounding box approach. Fault recognition model 148 is optimized by machine learning on a neural network using a fourth training data set. The fourth training data set is a dataset containing multiple data sets (i.e., multiple frames of data) obtained by establishing a correspondence between the fourth example data set and the fourth correct answer data set.

[0203] The fourth example data is the same image as the first example data described in the above embodiment. The fourth correct solution data is the correct solution data (i.e., annotation) for the fourth example data. Here, as an example of the fourth correct solution data, an annotation can be given that can identify the image area corresponding to various defects (e.g., fogging, stains, peeling, and lens scratches) reflected in the image used as the fourth example data.

[0204] The fault identification model 148 thus constructed is an AI for determining faults in the optical system 55 based on the diagnostic frame 40A (e.g., determining that a faulty part 92 exists on the diagnostic frame 40A (see reference)). Figure 4 The processing unit 2 obtains a diagnostic frame 40A from the request information 110 and inputs the obtained diagnostic frame 40A into the fault identification model 148. Thereby, the fault identification model 148 identifies the faults reflected in the input diagnostic frame 40A according to their types (e.g., fogging, stains, peeling, and lens scratches).

[0205] The identification process 112 organizes the identification results based on the region identification model 144, the dirt identification model 146, and the fault identification model 148 into the identification result 120 described in the above embodiment. The processing device 2 performs a comprehensive judgment process 150 using the identification result 120. The comprehensive judgment process 150 is a process that comprehensively judges the identification result 120. As an example of the comprehensive judgment process 150, it can be equivalent to the process of determining whether a defect exists 114 and the process of determining the type 116 described in the above embodiment. The comprehensive judgment process 150 generates information including information 122 indicating no defect and information 128 indicating a defect based on the identification result 120, namely, a judgment result 152. The processing device 2 sends the judgment result 152 to the medical support device 24. The medical support device 24 uses the judgment result 152 to perform display control on the display device 18 in the same manner as in the above embodiment.

[0206] exist Figure 12 In the examples shown, dirt identification model 146 is an example of the "first type determination AI" involved in the present invention, and fault identification model 148 is an example of the "second type determination AI" involved in the present invention.

[0207] Thus, dirt recognition model 146 is an AI specifically designed to identify dirt 90, and fault recognition model 148 is an AI specifically designed to identify faulty parts 92. Therefore, by using dirt recognition model 146 and fault recognition model 148 together, dirt 90 and faults in the optical system 55 can be determined with high accuracy.

[0208] Fault identification model 148 can be a patented identification model for each fault type. For example, such as Figure 13 As shown, in addition to the recognition model 118, the recognition process 112 may also use a fogging recognition model 153 optimized by performing machine learning specifically for fogging recognition, a peeling recognition model 154 optimized by performing machine learning specifically for peeling recognition, and a stain recognition model 156 optimized by performing machine learning specifically for stain recognition.

[0209] It is known that when fogging occurs in the optical system 55, the central portion of the diagnostic frame 40A displayed in the first display area 35A becomes unclear in the front view. Furthermore, it is known that when peeling occurs in the optical system 55, a large number of colored areas are projected onto the outer periphery of the diagnostic frame 40A displayed in the first display area 35A. Additionally, it is known that when smudges occur in the optical system 55, black spots are projected onto the left side of the diagnostic frame 40A displayed in the first display area 35A in the front view.

[0210] Therefore, the processing device 2 performs feature detection processing 158, which detects features projected onto the diagnostic frame 40A, in the pre-recognition processing stage of the recognition processing 112. In feature detection processing 158, it is determined whether the central portion of the front view of the diagnostic frame 40A is unclear. If it is determined that the central portion of the front view of the diagnostic frame 40A is unclear, the diagnostic frame 40A is processed using a fogging recognition model 153 in the recognition processing 112. In this case, the fogging recognition model 153 is input into the diagnostic frame 40A. Thus, the fogging recognition model 153 determines whether the optical system 55 is fogged, and if it is determined that the optical system 55 is fogged, it determines the position of the fogging projected onto the diagnostic frame 40A.

[0211] Furthermore, in the feature detection process 158, it is determined whether a large number of colored regions are projected onto the outer periphery of the diagnostic frame 40A. If it is determined that a large number of colored regions are projected onto the outer periphery of the diagnostic frame 40A, the diagnostic frame 40A is processed using the peeling recognition model 154 in the recognition process 112. In this case, the peeling recognition model 154 is input to the diagnostic frame 40A. Thus, the peeling recognition model 154 determines whether the optical system 55 has peeling, and if it determines that the optical system 55 has peeling, it determines the location of the peeling projected onto the diagnostic frame 40A.

[0212] Furthermore, in the feature detection process 158, it is determined whether a black spot is reflected on the left side of the front view of the diagnostic frame 40A. If it is determined that a large number of black spots are reflected on the left side of the front view of the diagnostic frame 40A, the diagnostic frame 40A is processed using the stain recognition model 156 in the recognition process 112. In this case, the stain recognition model 156 is input to the diagnostic frame 40A. Thus, the stain recognition model 156 determines whether there are stains on the optical system 55, and if it determines that there are stains on the optical system 55, it determines the location of the stain reflected on the diagnostic frame 40A.

[0213] In the identification process 112, following the same principle as in the above-described embodiment, the identification results based on the atomization identification model 153, the peeling identification model 154, and the stain identification model 156 are reflected in the identification result 120 obtained by the identification model 118. For example, the identification result 120 is adjusted by assigning the identification results based on the atomization identification model 153, the peeling identification model 154, and the stain identification model 156 as weights, respectively. The identification result 120 thus obtained is then used by the comprehensive judgment process 150. Then, the comprehensive judgment process 150 generates a judgment result 152 and sends it to the medical support device 24 (see reference). Figure 12 ).

[0214] The above examples illustrate the forms in which the processing device 2 performs identification processing 112, determination of the presence or absence of defects processing 114, type determination processing 116, noise removal processing 140, and edge extraction processing 142. However, this is only one example. For instance, such as... Figure 14 As shown, the processing device 2 can replace the identification processing 112, the defect determination processing 114, the type determination processing 116, the noise removal processing 140, and the edge extraction processing 142 to perform the generation processing 160.

[0215] In the generation process 160, a generation AI 162 is used. As an example of generation AI 162, ChatGPT, which uses GPT-4 (internet search <https: / / openai.com / gpt-4>), can be used. In generation AI 162, as a request message 110 sent from the medical support device 24, a prompt 110A (e.g., a prompt for generating a judgment result 152) and a diagnostic frame 40A as command data can be input, and a judgment result 152 is generated by generation AI 162. The judgment result 152 generated by generation AI 162 is sent to the medical support device 24 via the processing device 2.

[0216] Furthermore, the information generated by AI162 is not limited to the judgment result 152; for example, it could be displayed on... Figure 9The information shown on screen 35 (i.e., a no-adverse-condition message 134, an adverse-condition message 136, and / or bounding box BB, etc., of an example of the "diagnostic result" involved in this invention) may also be information obtained by converting the no-adverse-condition message 134 and the adverse-condition message 136 into audio. The information obtained by converting the no-adverse-condition message 134 and the adverse-condition message 136 into audio is an example of the "diagnostic result" involved in this invention.

[0217] In the above embodiments, examples are given of recognition processing 112, noise removal processing 140, edge extraction processing 142, comprehensive judgment processing 150, and generation processing 160 performed by the processing device 2 (i.e., the execution unit 82A indirectly performs recognition processing 112, noise removal processing 140, edge extraction processing 142, comprehensive judgment processing 150, and generation processing 160 using the processing device 2), but the present invention is not limited to this. For example, such as Figure 15 As shown, the execution unit 82A can directly perform the recognition processing 112, noise removal processing 140, edge extraction processing 142, comprehensive judgment processing 150 and / or generation processing 160.

[0218] In the above embodiments, examples have been given of the control unit 82B generating and displaying information visualizing the absence of adverse conditions 122 on the screen 35, and generating and displaying information visualizing the presence of adverse conditions 128 on the screen 35. However, the present invention is not limited thereto. For example, the processing device 2 may generate information visualizing the absence of adverse conditions 122 and / or the presence of adverse conditions 128 and send it to the medical support device 24, etc., and the medical support device 24, etc., may display the information visualizing the absence of adverse conditions 122 and / or the presence of adverse conditions 128 by the processing device 2 on the screen 35, etc.

[0219] In the above embodiment, a diagnostic frame 40A generated by the camera 52 capturing an image of the inner surface 98 on which a test pattern is formed is given as an example for diagnosing the optical system 55. However, this is only one example; a diagnostic frame 40A generated by the camera 52 capturing an image of the inner surface 98 on which no test pattern is formed can also be used for diagnosing the optical system 55. The inner surface 98 on which no test pattern is formed refers to a monochromatic surface. As an example of a monochromatic surface, a white surface can be given.

[0220] In the above embodiments, an example is given of displaying information that visualizes the absence of defects 122 and the presence of defects 128 on the screen 35 of the display device 18. However, this is only one example, and the absence of defects 122 and / or the presence of defects 128 may also be displayed on multiple display devices.

[0221] In the above embodiments, the bounding box method uses AI recognition processing 112 as an example, but this is only one example. For example, the segmentation method can also use AI recognition processing instead of the bounding box method.

[0222] In the above embodiments, a diagnostic process performed by computer 78 has been described as an example, but the present invention is not limited thereto. At least a portion of the processing included in the diagnostic process may also be performed by a device disposed outside computer 78.

[0223] In the above embodiments, an example of processing device 2 being implemented through cloud computing was given, but this is only one example. Processing device 2 can also be implemented through network computing such as fog computing, edge computing, or grid computing.

[0224] In the above embodiment, the example of storing the diagnostic program 108 in the storage device 86 has been described, but the present invention is not limited thereto. For example, the diagnostic program 108 may also be stored in a non-transitory storage medium readable by a portable computer, such as an SSD or USB memory. The diagnostic program 108 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope device 10. The processor 82 performs diagnostic processing according to the diagnostic program 108.

[0225] Furthermore, the diagnostic program 108 can also be stored in a storage device such as a computer or server connected to the endoscope device 10 via a network, and the diagnostic program 108 can be downloaded and installed in the computer 78 upon request from the endoscope device 10.

[0226] Alternatively, it is not necessary to store the entire diagnostic program 108 in a storage device such as another computer or server connected to the endoscope device 10, or to store the entire diagnostic program 108 in the storage device 86; a portion of the diagnostic program 108 may also be stored.

[0227] As hardware resources for performing diagnostic processing, various processors, as shown below, can be used. For example, a general-purpose processor, i.e., a CPU, can function as hardware resources that perform diagnostic processing by executing software, i.e., programs. Furthermore, as processors, special-purpose circuits, such as FPGAs, PLDs, or ASICs, can be used, which have circuit structures specifically designed for performing specific processes. All processors have built-in or connected memory, and all processors use this memory to perform various diagnostic processes.

[0228] The hardware resources for performing diagnostic processing can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resources for performing diagnostic processing can be a single processor.

[0229] As examples of a single processor, firstly, there is a combination of one or more CPUs and software to form a processor, which functions as a hardware resource for performing diagnostic processing. Secondly, there is a processor, such as a System-on-a-Chip (SoC), which uses a single IC chip to implement the functions of an overall system including multiple hardware resources for performing diagnostic processing. In this way, diagnostic processing is implemented using one or more of the aforementioned processors as hardware resources.

[0230] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits composed of combined semiconductor elements and other circuit components. Moreover, the diagnostic process described above is merely one example. Therefore, it is certainly possible to delete unnecessary steps, add new steps, or change the processing order without departing from the main point.

[0231] The descriptions and illustrations shown above are a detailed explanation of the parts involved in this invention, and are merely one example of this invention. For example, the descriptions related to the above structure, function, effect, and effect are examples of the structure, function, effect, and effect of the parts involved in this invention. Therefore, it is natural that unnecessary parts can be deleted, new elements can be added, or substitutions can be made to the descriptions and illustrations shown above without departing from the spirit of this invention. Furthermore, to avoid trouble and facilitate understanding of the parts involved in this invention, descriptions of technical common sense that do not require special explanation in implementing this invention have been omitted from the descriptions and illustrations shown above.

[0232] All documents, patent applications and technical standards described in this specification may be referenced in this specification to the same extent as the specific and individually described instances of reference to each document, patent application and technical standard.

Claims

1. A diagnostic device comprising a processor for an endoscopic observer, the endoscopic observer having a front end capable of illuminating light into the body and equipped with an optical system capable of capturing images of the body. The processor performs the following processing: AI-based image analysis is performed on an image of the interior obtained by a camera while light is illuminating the interior, with a cover installed at the front end. This performs diagnostics on the optical system, which is covered by the cover at the front end and formed within the interior that reflects the light. Output the diagnostic results obtained by performing the diagnostic procedure.

2. The diagnostic device according to claim 1, wherein, The diagnosis includes determining whether there are any defects in the optical system and whether those defects have been addressed.

3. The diagnostic device according to claim 1, wherein, The diagnosis includes a process for determining the type of malfunction in the optical system. The types of defects include a first type where the optical system does not need to be recycled and a second type where the optical system needs to be recycled.

4. The diagnostic device according to claim 3, wherein, The category determination process includes a differentiation process that distinguishes the types of adverse conditions into the first type and the second type.

5. The diagnostic device according to claim 3, wherein, The first type is dirt in the optical system. The second type is a malfunction of the optical system.

6. The diagnostic device according to claim 1, wherein, The image analysis is achieved by inputting the image into the learned model to generate the diagnostic result or information as the basis for the diagnostic result. The learned model generates the hypothetical diagnostic result or information as the basis for the diagnostic result by inputting the image into the learned model.

7. The diagnostic device according to claim 3, wherein, The AI ​​includes: a first category determination AI, which determines the first category based on the internal image; and a second category determination AI, which determines the second category based on the internal image.

8. The diagnostic device according to claim 1, wherein, The AI ​​is enhanced through relearning based on the diagnostic results.

9. The diagnostic device according to claim 1, wherein, The diagnosis includes a location determination process to determine the location of the malfunction in the optical system.

10. The diagnostic device according to claim 1, wherein, The image described is obtained by performing noise removal and / or edge extraction processing.

11. The diagnostic device according to claim 10, wherein, The image described is obtained by performing the edge extraction process after the noise removal process.

12. The diagnostic device according to claim 1, wherein, The structure includes a first region that reflects the light and a second region with a reflectivity lower than that of the first region.

13. The diagnostic device according to claim 12, wherein, The area of ​​the first region is larger than the area of ​​the second region, and the spatial frequency of the second region is higher than the spatial frequency of the first region.

14. The diagnostic device according to claim 12, wherein, The second region is the test map.

15. The diagnostic device according to claim 1, wherein, The interior is formed into a curved surface.

16. The diagnostic device according to claim 1, wherein, The inside is the surface that allows the light to diffuse.

17. The diagnostic device according to claim 1, wherein, The cover is formed by a double-layer structure.

18. The diagnostic device according to claim 17, wherein, The double-layer structure is formed by an inner layer and an outer layer. A hollow region is provided between the inner layer and the outer layer.

19. The diagnostic device according to claim 1, wherein, The outer surface of the cover is black.

20. The diagnostic device according to claim 1, wherein, The cover is attached to the front end via an accessory. The diameter of the portion of the accessory that connects the cover to the front end is adjustable.

21. A diagnostic system comprising: Terminal for an endoscopic observer, the endoscopic observer having a front end portion capable of illuminating light into the body and equipped with an optical system capable of capturing images of the body; and server, The terminal transmits an image of the interior, captured by a camera while light shines into it, to the server, with a cover installed at the front end. The cover, installed at the front end, covers the optical system and is formed on the interior, which is capable of reflecting light. The server performs the following processing: Diagnosis of the optical system is performed by using AI-based image analysis on the images contained therein; and The diagnostic results obtained by performing the diagnosis are sent to the terminal. The terminal receives the diagnostic results.

22. A diagnostic method for an endoscopic observer having a front end of an optical system capable of illuminating a body and equipped with a camera capable of capturing images of the body, the diagnostic method comprising the following steps: AI-based image analysis is performed on an image of the interior obtained by a camera while light is illuminating the interior, with a cover installed at the front end. This performs diagnostics on the optical system, which is covered by the cover at the front end and formed within the interior that reflects the light. Output the diagnostic results obtained by performing the diagnostic procedure.

23. A program for causing a computer to perform processing for an endoscopic observer having a front end capable of illuminating light into the body and provided with an optical system capable of capturing images of the body, the processing comprising the following steps: AI-based image analysis is performed on an image of the interior obtained by a camera while light is illuminating the interior, with a cover installed at the front end. This performs diagnostics on the optical system, which is covered by the cover at the front end and formed within the interior that reflects the light. Output the diagnostic results obtained by performing the diagnostic procedure.

Images