Medical assistance device, endoscope device, medical assistance system, medical assistance method, and computer program product

Abstract

Provided are a medical assistance device, an endoscope device, a medical assistance system, a medical assistance method, and a computer program product with which it is possible for a user to recognize the degree of time lag generated by transmission / reception between the medical assistance device and a processing device. A medical assistance device includes a communication device and a processor. The communication device transmits the first medical image to the processing device, and receives a processing result obtained by performing processing on the first medical image by the processing device. The processor performs a first control for outputting time information relating to the time required for the communication device to transmit the first medical image to the processing device and then to receive the processing result, and / or a second control for varying the output mode of the processing result in accordance with the time, the first control being configured to output time information relating to the time required for the communication device to transmit the first medical image to the processing device and the second control being configured to output time information relating to the time required for the communication device to receive the processing result.

Description

Technical Field

[0001] The present invention relates to a medical support device, an endoscope device, a medical support system, a medical support method and a computer program product. Background Art

[0002] Patent document 1 discloses an electronic endoscope system, which includes: an electronic endoscope that photographs an observation object to generate an image signal; a processor that is detachably connected to the electronic endoscope; and an image processing device that is detachably connected to the processor. In the electronic endoscope system described in Patent document 1, the image processing device includes: a first image processing unit that performs a first image processing on an image signal to generate a first video signal; and a processing time output unit that outputs information related to the processing time in the first image processing to the processor. In the electronic endoscope system described in Patent document 1, the processor includes: an image signal output unit that outputs an image signal to the image processing device; a second image processing unit that performs a second image processing on the image signal to generate a second video signal; and a first timing control unit that controls the timing of outputting the second video signal based on the information related to the processing time.

[0003] Patent document 2 discloses an inspection support device that is used in connection with an endoscope system, wherein the endoscope system includes a camera unit inserted into the body of a subject. The inspection support device described in patent document 2 includes an acquisition unit, a detection unit, a diagnosis unit, and a display control unit. In the inspection support device described in patent document 2, the acquisition unit sequentially acquires image signals provided by the endoscope system to an external device. The detection unit detects whether the frame images sequentially generated from the image signals are still images or dynamic images. When the detection unit detects that the frame image is a still image, the diagnosis unit performs an in-vivo diagnosis by inputting a diagnostic image that cuts at least a portion of the image area of ​​the frame image into a learned model. The display control unit displays the diagnosis result output by the diagnosis unit on the display unit.

[0004] In the inspection support device described in patent document 2, the display control unit does not display the diagnosis result until a preset retention time has passed after the detection unit detects that the frame image is a still image. Even if the retention time has passed, the diagnosis result is not displayed if the detection unit detects that the frame image generated based on the image signal acquired by the acquisition unit at the moment when the retention time has passed is a dynamic image.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2011-036488

[0006] Patent Document 2: Japanese Patent Application Publication No. 2022-142990 Summary of the invention

[0007] One embodiment of the present invention provides a medical support device, an endoscope device, a medical support system, a medical support method, and a computer program product that enable a user to recognize the degree of time lag generated by transmission and reception between the medical support device and a processing device.

[0008] The first method involved in the present invention is a medical support device, which includes: a communication device; and a processor, the communication device sends a first medical image to a processing device, and receives a processing result obtained by the processing device performing processing on the first medical image, the processor performs a first control and / or a second control, the first control outputs time information related to the time required for the communication device to send the first medical image to the processing device to receive the processing result, and the second control makes the output method of the processing result different according to the time.

[0009] A second aspect of the present invention is the medical support device according to the first aspect, wherein the processor outputting the time information includes the processor displaying the time information on the first screen.

[0010] The third method involved in the present invention is the medical support device involved in the second method, wherein the first control includes the following control: displaying the first medical image and / or the second medical image obtained after the first medical image in time on the second screen in a state that can be compared with time information.

[0011] A fourth aspect according to the present invention is the medical support device according to any one of the first to third aspects, wherein the time information includes first specific information capable of identifying the time itself.

[0012] A fifth aspect according to the present invention is the medical support device according to any one of the first to fourth aspects, wherein the time information includes second specific information capable of specifying a difference between the time and the first reference time.

[0013] A sixth aspect according to the present invention is the medical support apparatus according to the fifth aspect, wherein the second specific information includes length information capable of specifying the length of time relative to the first reference time.

[0014] A seventh aspect according to the present invention is the medical support device according to any one of the first to sixth aspects, wherein the time information is visualized information.

[0015] An eighth aspect according to the present invention is the medical support device according to the seventh aspect, wherein the time information is information visualized by text and / or a mark.

[0016] A ninth aspect according to the present invention is the medical support device according to the seventh aspect or the eighth aspect, wherein the processor changes the visualization format according to time, and the visualization format is a format for visualizing time information.

[0017] A tenth aspect of the present invention is the medical support device according to the ninth aspect, wherein the processor makes the visual recognition of the time information when the time is equal to or greater than the second reference time higher than the visual recognition of the time information when the time is less than the second reference time.

[0018] An eleventh aspect according to the present invention is the medical support device according to any one of the first to tenth aspects, wherein the output aspect is a display aspect in which the processor displays the processing result on the third screen.

[0019] The 12th mode involved in the present invention is the medical support device involved in the 11th mode, wherein the display mode includes a first display mode and a second display mode, the first display mode is used when the time is greater than a third reference time, and can determine that the time is greater than the third reference time, and the second display mode is used when the time is less than the third reference time, and can determine that the time is less than the third reference time.

[0020] A thirteenth aspect according to the present invention is the medical support device according to the twelfth aspect, wherein the second display aspect has higher visibility than the first display aspect.

[0021] A fourteenth aspect according to the present invention is the medical support apparatus according to the thirteenth aspect, wherein the processing result is not displayed in the first display aspect, and the processing result is displayed in the second display aspect.

[0022] The 15th method involved in the present invention is a medical support device involved in any one of the 11th to 14th methods, wherein the second control includes the following control: displaying the first medical image and / or the third medical image obtained after the first medical image in time on the fourth screen in a state that can be compared with the processing result.

[0023] The 16th method involved in the present invention is a medical support device involved in any one of the 11th to 14th methods, wherein the second control includes the following control: when the time is less than the 4th reference time, the first medical image is displayed on the 4th screen in a state that can be compared with the processing result.

[0024] A seventeenth aspect of the present invention is the medical support apparatus according to the sixteenth aspect, wherein the fourth reference time is a maximum time during which the degree of affinity between the first medical image and the processing result falls within an allowable range.

[0025] An eighteenth aspect according to the present invention is the medical support apparatus according to any one of the first to seventeenth aspects, wherein the processing is AI processing of the first medical image.

[0026] A nineteenth aspect according to the present invention is the medical support apparatus according to any one of the first to eighteenth aspects, wherein the processing includes recognition processing of recognizing a characteristic region appearing in the first medical image.

[0027] A 20th aspect according to the present invention is the medical support device according to the 19th aspect, wherein the characteristic region is a lesion.

[0028] A 21st aspect according to the present invention is the medical support apparatus according to any one of the 1st to 20th aspects, wherein the first medical image is an endoscopic image generated by imaging a body with a camera mounted on an endoscope.

[0029] A 22nd aspect of the present invention is an endoscope device comprising: the medical support device according to any one of the first to 21st aspects; and a camera that is inserted into a body and captures an image of the body to generate a first medical image.

[0030] The 23rd method involved in the present invention is a medical support system, which comprises: a medical support device; and a processing device, the medical support device sends a first medical image to the processing device, the processing device performs processing on the first medical image, and sends the processing result obtained by performing the processing to the medical support device, the medical support device receives the processing result sent from the processing device, and performs a first control and / or a second control, the first control outputs time information related to the time required for the medical support device to receive the processing result after the first medical image is sent to the processing device, and the second control makes the output method of the processing result different according to the time.

[0031] The 24th method involved in the present invention is a medical support method, which includes the following steps: sending a first medical image to a processing device; receiving a processing result obtained by processing the first medical image by the processing device; and performing a first control and / or a second control, wherein the first control outputs time information related to the time required from sending the first medical image to receiving the processing result to the processing device, and the second control outputs the processing result in a different manner according to the time.

[0032] The 25th method involved in the present invention is a program for causing a computer to perform medical support processing, the medical support processing comprising the following steps: sending a first medical image to a processing device; receiving a processing result obtained by processing the first medical image by the processing device; and performing a first control and / or a second control, the first control outputting time information related to the time required from sending the first medical image to receiving the processing result to the processing device, and the second control making the output method of the processing result different according to the time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a conceptual diagram showing an example of how a doctor uses a medical support system.

[0034] Figure 2 This is a conceptual diagram showing an example of the overall configuration of a medical support system.

[0035] Figure 3 This is a block diagram showing an example of the hardware configuration of the electrical system of the medical support system.

[0036] Figure 4 This is a block diagram showing an example of the main functions of a processor included in the medical support device and an example of information stored in a storage device.

[0037] Figure 5 This is a conceptual diagram showing an example of the processing contents of the medical support device and the processing device.

[0038] Figure 6 This is a conceptual diagram showing an example of the processing content of the control unit and an example of the content displayed on the screen of the display device.

[0039] Figure 7 This is a flowchart showing an example of the flow of medical support processing.

[0040] Figure 8 This is a conceptual diagram showing a modified example of the content displayed in the second display area when the transmission and reception time is longer than the reference time.

[0041] Fig. 9 This is a conceptual diagram showing a modified example of the content displayed on the screen of the display device.

[0042] Fig.10 This is a conceptual diagram showing a display example in which a frame obtained after a frame displayed in the first display area when the transmission and reception time is less than the reference time is used as a frame displayed in the first display area when the transmission and reception time is greater than the reference time. DETAILED DESCRIPTION

[0043] Hereinafter, an example of an embodiment of a medical support device, an endoscope device, a medical support system, a medical support method, and a program according to the present invention will be described with reference to the drawings.

[0044] First, the words and phrases used in the following description are explained.

[0045] CPU is short for "Central Processing Unit". GPU is short for "Graphics Processing Unit". GPGPU is short for "General-Purpose computing on Graphics Processing Units". APU is short for "Accelerated Processing Unit". TPU is short for "Tensor Processing Unit". RAM is short for "Random Access Memory". NVM is short for "Non-volatile memory". EEPROM is short for "Electrically Erasable Programmable Read-Only Memory". ASIC is short for "Application Specific Integrated Circuit". PLD is short for "Programmable Logic Device". FPGA is short for "Field-Programmable Gate Array". SoC is the abbreviation of "System-on-a-chip". SSD is the abbreviation of "Solid State Drive". USB is the abbreviation of "Universal Serial Bus". HDD is the abbreviation of "Hard Disk Drive". EL is the abbreviation of "Electro-Luminesce nce". CMOS is the abbreviation of "Complementary Metal Oxide Semiconductor". CCD is the abbreviation of "Charge Coupled Device". AI is the abbreviation of "Artificial Intelligence". BLI is the abbreviation of "Blue Light Imaging". LCI is the abbreviation of "Linked Color Imaging". I / F is the abbreviation of "Interface". SSL is the abbreviation of "Sessile Serrated Lesion".LAN is the abbreviation of "Local Area Network". WAN is the abbreviation of "Wide Area Network". 5G is the abbreviation of "5th Generation Mobile Communication System". RTC is the abbreviation of "Real Time Clock". IC is the abbreviation of "Integrated Circuit". MRI is the abbreviation of "Magnetic Res onance Imaging". CT is the abbreviation of "Computed Tomography".

[0046] In the following description, a processor with a symbol (hereinafter referred to as a "processor") can be a physical or virtual computing device, or a combination of multiple physical or virtual computing devices. In addition, a processor can be a computing device, or a combination of multiple computing devices. As an example of a computing device, a CPU, a GPU, a GPGPU, an APU, or a TPU can be cited.

[0047] In the following description, a memory with a symbol is a memory such as a RAM that temporarily stores information and is used as a working memory by a processor.

[0048] In the following description, the storage device with a symbol is one or more non-volatile storage devices that store various programs and various parameters, etc. As an example of a non-volatile storage device, a flash memory, a magnetic disk, or a magnetic tape, etc. can be cited. In addition, as another example of a storage device, a cloud storage can be cited.

[0049] In the following embodiments, the external I / F with a symbol is responsible for the exchange of various information between multiple devices connected to each other. As an example of an external I / F, a USB interface can be cited. A communication I / F including a communication processor and an antenna can be applied to the external I / F. The communication I / F is responsible for communication between multiple computers. As an example of a communication standard applied to the communication I / F, wireless communication standards including 5G, Wi-Fi (registered trademark) or Bluetooth (registered trademark) can be cited.

[0050] In the following embodiments, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be only A, only B, or a combination of A and B. In addition, in this specification, when three or more items are connected and expressed with "and / or", the same thinking as "A and / or B" is also applied.

[0051] Figure 11 is a conceptual diagram showing an example of a method of using the medical support system 1. Figure 1 As shown, the medical support system 1 includes a processing device 2 and an endoscope device 10, and the processing device 2 and the endoscope device 10 are communicatively connected via a network 3. As an example of the network 3, the Internet can be cited. However, the Internet is only an example, and as other examples of the network 3, a WAN and / or a LAN can be cited. In this embodiment, the medical support system 1 is an example of the "medical support system" involved in the present invention, and the processing device 2 is an example of the "processing device" involved in the present invention.

[0052] The endoscope device 10 is used by a doctor 12 in endoscopic examination or the like. The endoscopic examination is assisted by staff such as a nurse 14.

[0053] The information obtained by the endoscope device 10 is sent to the processing device 2 via the network 3. As an example of the processing device 2, a cloud server can be cited. However, the cloud server is only an example, and the processing device 2 can be a local server or a personal computer. The processing device 2 receives the information sent from the endoscope device 10, performs processing using the received information, and sends the processing result obtained by performing the processing to the endoscope device 10.

[0054] The endoscope device 10 includes an endoscope observation device 16, a display device 18, a light source device 20, a control device 22, and a medical support device 24. In the present embodiment, the endoscope observation device 16 is an example of the "endoscope observation device" involved in the present invention, the endoscope device 10 is an example of the "endoscope device" involved in the present invention, and the medical support device 24 is an example of the "medical support device" involved in the present invention.

[0055] The endoscope apparatus 10 is a medical imaging device for diagnosing a large intestine 28 included in a subject 26 (eg, a patient) using an endoscope 16. In the present embodiment, the large intestine 28 is an object to be observed by the doctor 12.

[0056] The endoscope 16 is used by the doctor 12 and inserted into the body of the subject 26. In the present embodiment, the endoscope 16 is inserted into the large intestine 28 which is a lumen organ of the subject 26.

[0057] The endoscope apparatus 10 causes the endoscope 16 inserted into the large intestine 28 of the subject 26 to capture images of the inside of the large intestine 28 of the subject 26 and to perform various medical procedures on the large intestine 28 as necessary.

[0058] The endoscope device 10 acquires and outputs an image showing the shape of the large intestine 28 by photographing the inside of the large intestine 28 of the subject 26. In the present embodiment, the endoscope device 10 is an endoscope having an optical imaging function for photographing reflected light obtained by irradiating light 30 in the large intestine 28 and reflecting it from the intestinal wall 32 of the large intestine 28.

[0059] In addition, although the endoscopic examination of the large intestine 28 is exemplified here, this is only an example, and the present invention is also applicable to the endoscopic examination of a tubular organ such as the esophagus, stomach, duodenum, or trachea.

[0060] The light source device 20, the control device 22 and the medical support device 24 are installed on the carriage 34. A plurality of carriages are installed in the vertical direction, and the medical support device 24, the control device 22 and the light source device 20 are installed from the lower stage to the upper stage. In addition, the display device 18 is installed on the uppermost stage of the carriage 34.

[0061] The control device 22 controls the entire endoscope device 10. The medical support device 24 performs various image processing on the image obtained by the endoscope 16 photographing the intestinal wall 32 under the control of the control device 22. The medical support device 24 is communicatively connected to the processing device 2 via the network 3, and receives the requested service from the processing device 2 by requesting the processing device 2 to provide the service.

[0062] The display device 18 displays various information including images. An example of the display device 18 is a liquid crystal display or an EL display. In addition, a tablet terminal with a display may be used instead of the display device 18 or together with the display device 18.

[0063] The screen 35 is displayed on the display device 18. The screen 35 includes a plurality of display areas. The plurality of display areas are arranged in parallel in the screen 35. Figure 1 In the example shown, as an example of a plurality of display areas, the first display area 35A, the second display area 35B, and the third display area 35C are shown. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as a main display area, and the second display area 35B is used as a sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited thereto, as long as it is a size relationship that falls within the screen 35. The third display area 35C is a display area other than the first display area 35A and the second display area 35B in the screen 35, and can be used as a sub-display area. In the present embodiment, the screen 35 is an example of the "first screen", "second screen", "third screen", and "fourth screen" involved in the present invention.

[0064] The first display area 35A displays an endoscope dynamic image 39. The endoscope dynamic image 39 is a dynamic image acquired by imaging the intestinal wall 32 in the large intestine 28 of the subject 26 using the endoscope 16. Figure 1 In the illustrated example, a dynamic image showing the intestinal wall 32 is shown as an example of the endoscopic dynamic image 39 .

[0065] In the intestinal wall 32 shown in the endoscope dynamic image 39, as the region of interest (i.e., the observation target region) watched by the doctor 12, a lesion 42 (e.g., Figure 1 The example shown is a lesion 42, and the doctor 12 can visually recognize the shape of the intestinal wall 32 including the lesion 42 through the endoscopic dynamic image 39. In this embodiment, the lesion 42 is an example of the "characteristic region" and "lesion" involved in the present invention.

[0066] There are various types of lesions 42. For example, tumor polyps and non-tumor polyps can be cited as types of lesions 42. For example, adenomatous polyps (for example, SSL) can be cited as types of tumor polyps. For example, hamartomatous polyps, hyperplastic polyps, and inflammatory polyps can be cited as types of non-tumor polyps. In addition, the types exemplified here are types that are pre-assumed as types of lesions 42 when endoscopic examination is performed on the large intestine 28. If the organ to be endoscopically examined is different, the type of lesion 42 is also different.

[0067] In the present embodiment, for the sake of convenience, an example is given in which one lesion 42 is shown in the endoscopic dynamic image 39 . However, the present invention is not limited thereto, and the present invention is also applicable even when a plurality of lesions 42 are shown in the endoscopic dynamic image 39 .

[0068] In the present embodiment, the lesion 42 is illustrated, but this is only an example, and the area of ​​interest (i.e., the observation object area) that the doctor 12 focuses on may be a characteristic area having certain inherent characteristics, such as an organ (e.g., the duodenal papilla), a mark, an artificial treatment instrument (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after removal of a polyp, etc.).

[0069] The image displayed in the first display area 35A is one frame 40 included in a moving image composed of a plurality of time-sequential frames 40. That is, in the first display area 35A, a plurality of time-sequential frames 40 are displayed at a predetermined frame rate (eg, several tens of frames / second).

[0070] As an example of a dynamic image displayed in the first display area 35A, a dynamic image in an instant preview mode can be cited. The instant preview mode is only an example, and a dynamic image that is displayed after being temporarily stored in a memory or the like, such as a dynamic image in a post-browse mode, can also be a dynamic image. In addition, each frame included in a recording dynamic image stored in a memory or the like can also be played and displayed on the screen 35 (for example, the first display area 35A) as an endoscope dynamic image 39.

[0071] In the screen 35, the second display area 35B is adjacent to the first display area 35A and is displayed in the lower right of the screen 35. The display position of the second display area 35B can be any position as long as it is in the screen 35 of the display device 18, but it is preferably displayed in a position that can be compared with the endoscope dynamic image 39.

[0072] The second display area 35B displays auxiliary information 44 that assists the doctor 12 in making medical judgments during endoscopy. The auxiliary information 44 is information that is referred to by the doctor 12. Examples of the auxiliary information 44 include various information related to the subject 26 inserted into the body by the endoscope 16 and / or various information obtained by performing medical support processing described later.

[0073] The third display area 35C displays visual information obtained by processing the frame 40 using AI. The details will be described later.

[0074] Figure 2 1 is a conceptual diagram showing an example of the overall structure of the endoscope device 10. Figure 2 As shown, the endoscope 16 includes an operation portion 46 and an insertion portion 48. The insertion portion 48 is partially bent by being operated by the operation portion 46. The insertion portion 48 is bent according to the doctor 12 (refer to Figure 1 ) to operate the operating unit 46, while according to the large intestine 28 (reference Figure 1 ) is bent into the shape and inserted into the large intestine 28.

[0075] A camera 52, a lighting device 54, and a treatment instrument opening 56 are provided at the distal end portion 50 of the insertion portion 48. The camera 52 and the lighting device 54 are provided at the distal end surface 50A of the distal end portion 50. In addition, here, an example in which the camera 52 and the lighting device 54 are provided at the distal end surface 50A of the distal end portion 50 is given, but this is only an example, and the camera 52 and the lighting device 54 may be provided at the side of the distal end portion 50 so that the endoscope 16 is configured as a side mirror.

[0076] The camera 52 is mounted on the endoscope 16, inserted into the body cavity of the subject 26 and photographs the observation object area, thereby generating a frame 40 as an endoscopic image. In the present embodiment, the camera 52 generates an endoscopic dynamic image 39 including a plurality of frames 40 in a time series by photographing the inside of the subject 26 (for example, the inside of the large intestine 28). As an example of the camera 52, a CMOS camera can be cited. However, this is only an example, and other types of cameras such as a CCD camera can also be used. In the present embodiment, the camera 52 is an example of the "camera" involved in the present invention, and the frame 40 is an example of the "first medical image", "second medical image", "third medical image" and "endoscopic image" involved in the present invention.

[0077] The lighting device 54 has lighting windows 54A and 54B. The lighting device 54 irradiates the light 30 (see Figure 1 ). Examples of the type of light 30 irradiated from the lighting device 54 include visible light (e.g., white light) and non-visible light (e.g., near infrared light). Furthermore, the lighting device 54 irradiates special light via the lighting windows 54A and 54B. Examples of special light include light for BLI and / or light for LCI. The camera 52 takes an optical image of the inside of the large intestine 28 while the lighting device 54 irradiates the light 30 inside the large intestine 28.

[0078] The treatment tool opening 56 is an opening for allowing the treatment tool 58 to protrude from the distal end portion 50. The treatment tool opening 56 can also be used as a suction port for suctioning blood and body wastes, and a delivery port for delivering fluids.

[0079] The operation portion 46 is formed with a treatment instrument insertion port 60, and the treatment instrument 58 is inserted into the insertion portion 48 through the treatment instrument insertion port 60. The treatment instrument 58 passes through the insertion portion 48 and protrudes from the treatment instrument opening 56 to the outside. Figure 2 In the example shown, a puncture needle is shown as the treatment instrument 58 protruding from the treatment instrument opening 56. Here, the puncture needle is shown as the treatment instrument 58, but this is only an example, and the treatment instrument 58 may also be a grasping forceps, a papillotomy knife, a snare, a catheter, a guide wire, a cannula, and / or a puncture needle with a guide sheath.

[0080] The endoscope 16 is connected to the light source device 20 and the control device 22 via a universal cord 62. The control device 22 is connected to a medical support device 24 and a receiving device 64. Furthermore, the medical support device 24 is connected to the display device 18 in addition to the processing device 2. That is, the control device 22 is connected to the processing device 2 and the display device 18 via the medical support device 24.

[0081] Here, the medical support device 24 is exemplified as an external device for expanding the functions performed by the control device 22, and thus the control device 22 and the display device 18 are indirectly connected via the medical support device 24. However, this is only an example. For example, the display device 18 may be directly connected to the control device 22. In this case, for example, the functions of the medical support device 24 may be installed on the control device 22.

[0082] The receiving device 64 receives the command from the doctor 12, and outputs the received command as an electric signal to the control device 22. Examples of the receiving device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0083] The control device 22 controls the light source device 20 , or exchanges various signals with the camera 52 , or exchanges various signals with the medical support device 24 .

[0084] The light source device 20 emits light under the control of the control device 22 and supplies the light to the lighting device 54. The lighting device 54 has a built-in light guide, and the light supplied from the light source device 20 is irradiated from the lighting windows 54A and 54B through the light guide. The light irradiated from the lighting windows 54A and 54B is the light 30 (refer to Figure 1 The control device 22 causes the camera 52 to take pictures and obtains the endoscope dynamic image 39 from the camera 52 (refer to Figure 1 ) and outputs it to a specified output destination (e.g., the medical support device 24).

[0085] The medical support device 24 supports medical treatment (for example, endoscopy in this example) by performing various image processing on the endoscopic moving image 39 input from the control device 22. The medical support device 24 outputs the endoscopic moving image 39 subjected to various image processing to a predetermined output destination (for example, the display device 18).

[0086] In addition, here, the example in which the endoscope dynamic image 39 output from the control device 22 is output to the display device 18 via the medical support device 24 is given for explanation, but this is only an example. For example, the control device 22 and the display device 18 may be connected, and the endoscope dynamic image 39 subjected to image processing by the medical support device 24 may be displayed on the display device 18 via the control device 22.

[0087] Figure 3 1 is a block diagram showing an example of the hardware configuration of the electrical system of the endoscope device 10. Figure 3As shown, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage device 76. The processor 72, the memory 74, the storage device 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the storage device 76 are used by the processor 72.

[0088] The external I / F 70 is responsible for the exchange of various information between one or more devices (hereinafter also referred to as “first external devices”) existing outside the control device 22 and the processor 72 .

[0089] The camera 52 is connected to the external I / F 70 as one of the first external devices. The external I / F 70 is responsible for the exchange of various information between the camera 52 and the processor 72. The processor 72 controls the camera 52 via the external I / F 70. In addition, the processor 72 obtains the large intestine 28 (reference) obtained by the camera 52 via the external I / F 70. Figure 1 ) and obtain an endoscopic dynamic image 39 (reference Figure 1 ).

[0090] The external I / F 70 is connected to the light source device 20 as one of the first external devices, and the external I / F 70 is responsible for the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light to the lighting device 54 under the control of the processor 72. The lighting device 54 irradiates the light supplied from the light source device 20.

[0091] The receiving device 64 is connected to the external I / F 70 as one of the first external devices. The processor 72 acquires a command received by the receiving device 64 via the external I / F 70 and executes a process corresponding to the acquired command.

[0092] The medical support device 24 includes a computer 78, an RTC 79, and an external I / F 80. The computer 78 includes a processor 82, a memory 84, and a storage device 86. The processor 82, the memory 84, the storage device 86, the RTC 79, and the external I / F 80 are connected to a bus 88. In the present embodiment, the computer 78 is an example of a "computer" involved in the present invention, and the processor 82 is an example of a "processor" involved in the present invention.

[0093] In addition, the hardware structure of the computer 78 (ie, the processor 82, the memory 84 and the storage device 86) is substantially the same as the hardware structure of the computer 66, and thus the description of the hardware structure of the computer 78 is omitted here.

[0094] RTC79 is an IC that generates and outputs data such as the current time. Processor 82 obtains data such as the current time from RTC79. In addition, here, an example of the form in which processor 82 obtains data of the current time from RTC79 is cited, but this is only an example, and processor 82 can also obtain data of the current time from a server on network 3.

[0095] The external I / F 80 is responsible for the exchange of various information between one or more devices (hereinafter also referred to as “second external devices”) existing outside the medical support device 24 and the processor 82 .

[0096] The control device 22 is connected to the external I / F 80 as one of the second external devices. Figure 3 In the example shown, the external I / F 80 is connected to the external I / F 70 of the control device 22. The external I / F 80 is responsible for the exchange of various information between the processor 82 of the medical support device 24 and the processor 72 of the control device 22. For example, the processor 82 obtains the endoscope dynamic image 39 (see FIG. 1 ) from the processor 72 of the control device 22 via the external I / Fs 70 and 80. Figure 1 ) and perform various image processing on the acquired endoscopic dynamic image 39.

[0097] The display device 18 is connected as one of the second external devices to the external I / F 80. The processor 82 controls the display device 18 via the external I / F 80, thereby causing the display device 18 to display various information (for example, the endoscope dynamic image 39 subjected to various image processing, etc.).

[0098] The processing device 2 is connected to the external I / F 80 as one of the second external devices via the network 3. The processor 82 exchanges various information with the processing device 2 via the external I / F 80. For example, the external I / F 80 sends the endoscopic dynamic image 39 acquired by the processor 82 from the camera 52 to the processing device 2. The processing device 2 receives the endoscopic dynamic image 39 sent from the external I / F 80, performs processing using at least one frame 40 included in the received endoscopic dynamic image 39, and sends the processing result to the medical support device 24. The external I / F 80 receives the processing result sent from the processing device 2. The processor 82 acquires the processing result received by the external I / F 80. In the present embodiment, the external I / F 80 is an example of a "communication device" involved in the present invention.

[0099] In recent years, the following technology has been developed: a learned model optimized by machine learning of a model (e.g., a neural network) is used to perform object recognition processing on each frame 40 contained in an endoscopic dynamic image 39 to identify a lesion 42, and the recognition result based on the object recognition processing or information based on the recognition result based on the object recognition processing is displayed on the screen 35.

[0100] However, the processing load of the object recognition processing performed on each frame 40 included in the endoscope dynamic image 39 is very large. Therefore, the following technology is currently being studied: the endoscope device 10 causes the processing device 2 that is communicably connected to the endoscope device 10 via the network 3 to perform the object recognition processing, and sends the processing results based on the object recognition processing to the endoscope device 10, thereby reducing the processing load imposed on the endoscope device 10.

[0101] When this technology is used, a time lag (hereinafter also referred to as "time lag") is generated by the transmission and reception between the endoscope device 10 and the processing device 2. If the time lag occurs, the content of the frame 40 displayed in the first display area 35A of the endoscope device 10 may be inconsistent with the content of the processing result based on the processing device 2 (for example, the recognition result based on the object recognition processing or the information based on the recognition result based on the object recognition processing). In addition, if the time lag is very short, the content of the frame 40 displayed in the first display area 35A of the endoscope device 10 is roughly consistent with the content of the processing result based on the processing device 2, so it will not have a great impact on the diagnosis of the doctor 12.

[0102] If the processing device 2 communicatively connected to the endoscope device 10 via the network 3 executes the object recognition process, the doctor 12 can recognize the extent of the time lag, which is very important for the doctor 12 to make a correct diagnosis during the endoscope examination. In addition, this is not limited to the endoscope examination, and it can also be said that it is applicable to other types of medical examinations (for example, radiographic examinations, MRI examinations, or CT examinations, etc.).

[0103] Therefore, in view of this situation, in this embodiment, as an example, Figure 4 As shown, the medical support process is performed by the processor 82 of the medical support device 24. Figure 4 This is a block diagram showing an example of the main functions of the processor 82 included in the medical support device 24 and an example of information stored in the storage device 86 .

[0104] The storage device 86 stores a medical support program 90. The medical support program 90 is an example of the "program" involved in the present invention. The processor 82 reads the medical support program 90 from the storage device 86 and performs medical support processing by executing the read medical support program 90 on the memory 84. The medical support processing is realized by the medical support program 90 executed by the processor 82 on the memory 84 as the communication unit 82A and the control unit 82B.

[0105] Figure 5 2 is a conceptual diagram showing an example of the processing content performed by the medical support device 24 and the processing device 2. Figure 5 As shown, the communication unit 82A and the control unit 82B acquire each of the multiple frames 40 in time series contained in the endoscopic dynamic image 39 generated by the camera 52 shooting at a shooting frame rate (for example, tens of frames / second) from the camera 52 in time series in units of one frame. The communication unit 82A and the control unit 82B acquire the timing synchronization of the frames 40.

[0106] The control unit 82B outputs the endoscope dynamic image 39 to the display device 18. For example, the control unit 82B displays the endoscope dynamic image 39 as a live preview image in the first display area 35A. That is, the control unit 82B displays the acquired frames 40 in the first display area 35A in sequence at a display frame rate (for example, several tens of frames per second) each time the frame 40 is acquired from the camera 52. In addition, the control unit 82B displays the auxiliary information 44 in the second display area 35B. In addition, the control unit 82B updates the display content (for example, the auxiliary information 44) of the second display area 35B along with the display content of the first display area 35A.

[0107] The communication unit 82A communicates with the external I / F 80 (see Figure 3 ) communicates with the processing device 2 (i.e., sends and receives various information). The communication unit 82A transmits the endoscope dynamic image 39 acquired from the camera 52 via the external I / F 80 (refer to Figure 3 ) is sent to the processing device 2. Here, the communication unit 82A obtains the sending time 92, which is the time when the endoscope dynamic image 39 is sent to the processing device 2. The sending time 92 is, for example, the time obtained by the communication unit 82A from the RTC 79 at the timing of sending the endoscope dynamic image 39 from the external I / F 80 to the processing device 2.

[0108] The processing device 2 receives the endoscope dynamic image 39 transmitted from the communication unit 82A. The processing device 2 has a recognition model 94, and performs a recognition process 95 using the recognition model 94 on the endoscope dynamic image 39. The processing device 2 may be a device that performs only the recognition process 95, or may be a device that performs a plurality of processes including the recognition process 95. For example, the plurality of processes may include a process for processing at least one frame 40 included in the endoscope dynamic image 39. In the present embodiment, the recognition process 95 is an example of the above-mentioned object recognition process, and is an example of the "AI process" and the "recognition process" involved in the present invention.

[0109] The processing device 2 identifies the lesion 42 in the endoscopic dynamic image 39 based on the endoscopic dynamic image 39. That is, the processing device 2 identifies the lesion 42 reflected in the frame 40 by sequentially performing the identification process 95 on each of the multiple frames 40 in time series included in the endoscopic dynamic image 39 acquired from the camera 52.

[0110] For example, the processing device 2 identifies the geometric characteristics of the lesion 42 (for example, position, size, shape, etc.), the type of the lesion 42, and the type of the lesion 42 (for example, pedunculated, sub-pedunculated, sessile, surface raised, surface flat, and surface depressed, etc.). In addition, the processing device 2 identifies the part (hereinafter referred to as "part") in the endoscopic dynamic image 39 based on the endoscopic dynamic image 39 obtained from the camera 52. Here, the part refers to the part of the large intestine 28. As the part of the large intestine 28, for example, the anal canal, the lower rectum, the upper rectum, the sigmoid colon, the descending colon, the transverse colon, the ascending colon, and the ileocecal part can be cited.

[0111] The recognition process 95 is performed by the processing device 2 on the acquired frame 40 each time the frame 40 is acquired. The recognition process 95 is a process for recognizing the lesion 42 and the part using AI. Here, the recognition process 95 uses the recognition model 94.

[0112] The recognition model 94 is a learned model for object recognition based on the bounding box method of AI. The recognition model 94 is optimized by performing machine learning on the neural network using training data. The training data is a data set containing multiple data (i.e., multiple frames of data) obtained by establishing a correspondence between example data and correct answer data.

[0113] The example data is an image assuming frame 40. As a first example of an image assuming frame 40, an image obtained by actually photographing the inside of the large intestine with a camera can be cited. As a second example of an image assuming frame 40, a virtually created image (for example, an image generated by generating AI) can be cited. The correct answer data is the correct answer data (i.e., annotations) for the example data. Here, as an example of the correct answer data, annotations are used to determine the geometric characteristics of the lesions reflected in the image used as the example data, the type of lesions, the type and location of the lesions.

[0114] The processing device 2 acquires the frame 40 from the camera 52, and inputs the acquired frame 40 into the recognition model 94. Thus, the recognition model 94 recognizes the lesion 42 and the part reflected in the input frame 40 each time the frame 40 is input, and generates and outputs the recognition result 96. In the present embodiment, the recognition result 96 is an example of the "processing result" involved in the present invention.

[0115] The processing device 2 transmits the recognition result 96 itself to the medical support device 24, or transmits information generated based on the recognition result 96 to the medical support device 24. In the medical support device 24, the external I / F 80 receives the recognition result 96 transmitted from the processing device 2. The communication unit 82A acquires the recognition result 96 received by the external I / F 80.

[0116] Here, the communication unit 82A obtains reception time 98 which is the time when the endoscope dynamic image 39 is received by the external I / F 80. The reception time 98 is the time obtained by the communication unit 82A from the RTC 79 when the external I / F 80 receives the recognition result 96 transmitted from the processing device 2, for example.

[0117] The control unit 82B obtains the recognition result 96, the transmission time 92, and the reception time 98 from the communication unit 82A. Based on the recognition result 96, the transmission time 92, and the reception time 98, the control unit 82B displays the recognition result 96 itself on the screen 35 (for example, the first display area 35A, the second display area 35B, and / or the third display area 35C) of the display device 18, or displays information generated based on the recognition result 96 on the screen 35 (for example, the first display area 35A, the second display area 35B, and / or the third display area 35C).

[0118] The recognition result 96 includes lesion presence information 100. The lesion presence information 100 is information indicating whether the lesion 42 is reflected in the frame 40 input to the recognition model 94. The recognition result 96 also includes geometric characteristic information 102, a lesion position map 106, and lesion feature information 108.

[0119] The geometric characteristic information 102 is information (e.g., coordinates) that can identify the shape, size, and position of the lesion 42 in the frame 40. The geometric characteristic information 102 or information based on the geometric characteristic information 102 can be displayed in the first display area 35A, can be displayed in the second display area 35B as one of the auxiliary information 44, and can also be displayed in the third display area 35C.

[0120] The lesion position map 106 is a map that can identify the position of the lesion 42 in the frame 40. The lesion characteristic information 108 is information that can identify the type of the lesion 42 reflected in the frame 40 input to the recognition model 94 and the type of the lesion 42.

[0121] The geometric characteristics of the lesion location map 106 (for example, the shape and size of the outer contour) correspond to the geometric characteristics of the frame 40 (for example, the shape and size of the outer contour). The lesion location map 106 is displayed on the screen 35 by the control unit 82B. In the present embodiment, the lesion location map 106 is displayed on the second display area 35B by the control unit 82B as one of the auxiliary information 44. For example, the display and non-display of the lesion location map 106 are switched according to the command and / or various conditions received by the receiving device 64. The lesion location map 106 displayed on the screen 35 is updated according to the display frame rate applied to the first display area 35A. By configuring in this way, the doctor 12 can grasp the approximate position of the lesion 42 in the endoscopic dynamic image 39 displayed in the first display area 35A by observing the endoscopic dynamic image 39 displayed in the first display area 35A while referring to the lesion location map 106 displayed in the second display area 35B.

[0122] The control unit 82B displays the lesion characteristic information 108 on the screen 35. For example, the lesion characteristic information 108 may be displayed as one of the auxiliary information 44 in the second display area 35B, or in the first display area 35A and / or the third display area 35C.

[0123] The recognition result 96 also includes part information 110. The part information 110 is information that can identify the part shown in the frame 40. The part information 110 can be displayed in the first display area 35A, in the second display area 35B as one of the auxiliary information 44, or in the third display area 35C.

[0124] Figure 6 82A and 82B and an example of the processing content displayed on the screen 35. Figure 6As shown, the control unit 82B calculates the transmission and reception time 112 based on the transmission time 92 and the reception time 98 acquired from the communication unit 82A. Here, the transmission and reception time 112 refers to the time required from when the frame 40 included in the endoscopic dynamic image 39 is transmitted to the processing device 2 through the external I / F 80 to when the external I / F 80 receives the recognition result 96 obtained by performing the recognition process 95 on the frame 40. The transmission and reception time 112 can also be said to be an index indicating the degree of time lag generated by the transmission and reception between the medical support device 24 and the processing device 2. In the present embodiment, the transmission and reception time 112 is an example of the "time" involved in the present invention.

[0125] The control unit 82B determines the difference between the transmission / reception time 112 and the reference time 114 by comparing the transmission / reception time 112 with the reference time 114. In other words, the difference between the transmission / reception time 112 and the reference time 114 refers to the length of the transmission / reception time 112 relative to the reference time 114. In the present embodiment, the reference time 114 is an example of the "first reference time", "second reference time", "third reference time" and "fourth reference time" involved in the present invention.

[0126] As an example of the reference time 114, the longest time during which the degree of affinity between the frame 40 displayed in the first display area 35A and the recognition result 96 falls within the allowable range can be cited. For example, regarding the longest time during which the degree of affinity between the frame 40 displayed in the first display area 35A and the recognition result 96 falls within the allowable range, the time previously obtained through experiments based on actual machines and / or computer simulations can be cited as the longest time during which the doctor 12 or the like recognizes that the content of the frame 40 displayed in the first display area 35A and the content of the recognition result 96 are consistent (i.e., matching).

[0127] Here, as an example of the reference time 114, the longest time when the affinity between the frame 40 displayed in the first display area 35A and the recognition result 96 falls within the allowable range is illustrated, but this is only an example, and it may also be a statistical time (for example, the most frequent value, average value, or median value, etc.) obtained in advance through experiments based on actual machines and / or computer simulations as the time when the doctor 12 or the like recognizes that the content of the frame 40 displayed in the first display area 35A is consistent with the content of the recognition result 96 (i.e., matching). In addition, the reference time 114 may be a fixed value or a variable value that changes according to commands received by the receiving device 64 or the like and / or various conditions.

[0128] The control unit 82B determines whether to display the recognition result 96 based on the transmission and reception time 112, and performs or does not display the recognition result 96 on the display device 18 according to the determination result.

[0129] The control unit 82B obtains the recognition result 96 from the processing device 2, generates and outputs the processing result information 116 which is information related to the obtained recognition result 96. Figure 6 In the example shown, the control unit 82B outputting the processing result information 116 means that the control unit 82B displays the processing result information 116 on the screen 35. The processing result information 116 includes, for example, information that can identify the lesion 42, information indicating the characteristics of the lesion 42, and / or information that can identify the location where the lesion 42 exists.

[0130] The longer the transmission and reception time 112 is, the greater the degree of time lag caused by the transmission and reception between the medical support device 24 and the processing device 2. This means that the greater the degree of time lag, the higher the possibility that the content of the frame 40 displayed on the screen 35 does not match the content of the processing result information 116. On the contrary, the smaller the degree of time lag is, the higher the possibility that the content of the frame 40 displayed on the screen 35 matches the content of the processing result information 116.

[0131] In the present embodiment, in order for the control unit 82B to determine whether the transmission / reception time 112, which can also be regarded as an index indicating the degree of time lag, is a time at which the content of the frame 40 displayed on the screen 35 matches the content of the processing result information 116, the control unit 82B compares the transmission / reception time 112 with the reference time 114 to determine whether the transmission / reception time 112 is greater than the reference time 114. That is, the control unit 82B determines that the content of the frame 40 displayed on the screen 35 does not match the content of the processing result information 116 if the transmission / reception time 112 is greater than the reference time 114, and determines that the content of the frame 40 displayed on the screen 35 matches the content of the processing result information 116 if the transmission / reception time 112 is less than the reference time 114. The determination result based on the control unit 82B is expressed in the form of output of the processing result information 116, or expressed by outputting information related to the transmission / reception time 112.

[0132] For example, the control unit 82B performs a first control for outputting information related to the transmission / reception time 112 , and a second control for making the output form of the processing result information 116 different according to the transmission / reception time 112 .

[0133] Here, first, an example of the second control is described. As an example of an output method of the processing result information 116, the display method in which the control unit 82B displays the processing result information 116 on the screen 35 can be cited. The display method includes a first display method and a second display method. The first display method is used when the transmission and reception time 112 is greater than the reference time 114, and it can be determined that the transmission and reception time 112 is greater than the reference time 114. The second display method is used when the transmission and reception time 112 is less than the reference time 114, and it can be determined that the transmission and reception time 112 is less than the reference time 114. Figure 6 In the example shown, when the transmission / reception time 112 is greater than the reference time 114, the control unit 82B displays the processing result information 116 on the screen 35 in the first display mode, and when the transmission / reception time 112 is less than the reference time 114, the control unit 82B displays the processing result information 116 on the screen 35 in the second display mode. The second display mode has higher visual recognition than the first display mode. For example, in the first display mode, the processing result information 116 is not displayed, and in the second display mode, the processing result information 116 is displayed.

[0134] Furthermore, the second control performed by the control unit 82B includes displaying the frame 40 on the screen 35 (in a state where it can be compared with the processing result information 116). Figure 6 The example shown is the control on the first display area 35A). Figure 6 In the example shown, when the sending and receiving time 112 is less than the reference time 114, the processing result information 116 is displayed in the second display area 35B and the third display area 35C, and in order to obtain the recognition result 96 used to generate the processing result information 116, the frame 40 identical to the frame 40 input into the recognition model 94 is displayed in the first display area 35A.

[0135] Here, an example is given in which a frame 40 identical to the frame 40 input into the recognition model 94 is displayed in the first display area 35A in order to obtain the recognition result 96 for generating the processing result information 116. However, this is only an example, and a frame 40 obtained after the frame 40 input into the recognition model 94 in order to obtain the recognition result 96 for generating the processing result information 116 (for example, a frame 40 obtained after several to dozens of frames) may be displayed in the first display area 35A.

[0136] exist Figure 6 In the example shown, when the transmission and reception time 112 is less than the reference time 114, the control unit 82B generates part name information 118, lesion identification information 120, identification information 122, and size information 124 as processing result information 116 and displays them in the third display area 35C (in Figure 6 In the example shown, the area below the first display area 35A).

[0137] The part name information 118 is information indicating the name of the part identified by performing the identification process 95, and is obtained based on the part information 110 (refer to Figure 5 ) is generated. Here, the site information 110 is illustrated, but any information other than the site information 110 may be used as long as it is information that can identify the site where the lesion 42 exists.

[0138] The lesion identification information 120 is information that can identify the lesion 42 identified by performing the identification process 95. The identification information 122 is information indicating the result of identifying the lesion 42 (for example, the degree of malignancy), and is obtained based on the lesion feature information 108 (refer to Figure 5 The size information 124 is information indicating the size of the lesion 42 identified by the identification process 95, and is generated based on the geometric characteristic information 102 (reference Figure 5 Here, the identification information 122 and the size information 124 are exemplified, but any information other than the identification information 122 and the size information 124 may be used as long as the information represents the characteristics of the lesion 42.

[0139] At the lesion location Figure 106 (ref. Figure 6 ) includes a bounding box BB, and when the transmission and reception time 112 is less than the reference time 114, the control unit 82B displays the bounding box BB in the first display area 35A as the processing result information 116. The bounding box BB is a rectangular box that can determine the position recognized by the recognition model 94 as the position of the lesion 42 projected in the frame 40 (for example, a rectangular box that is circumscribed with the image area representing the lesion 42). Figure 6 In the example shown, a bounding box BB is superimposed and displayed at a position where a lesion 42 appears in a frame 40 displayed in the first display area 35A. The display of the bounding box BB is updated in synchronization with the display timing of the endoscopic dynamic image 39 displayed in the first display area 35A.

[0140] In addition, here, the bounding box BB is cited, but this is only an example, and the control unit 82B may display an identifier (for example, marks defined by the four corners of a rectangular frame) instead of the bounding box BB on the screen 35 (in Figure 6 In the example shown, in the first display area 35A) within the screen 35.

[0141] exist Figure 6 In the example shown, the control unit 82B generates past information 126 as one of the auxiliary information 44 and displays it in the second display area 35B. The past information 126 is information generated based on the recognition result 96 obtained in the past. For example, the past information 126 includes various information displayed in the third display area 35C in the past and images or texts that can identify the parts that were not observed or treated in the current endoscopic examination.

[0142] Furthermore, when the transmission / reception time 112 is shorter than the reference time 114 , the control unit 82B displays the lesion position map 106 as one of the processing result information 116 and as one of the auxiliary information 44 in the second display area 35B.

[0143] Here, an example of the first control is described. The control unit 82B performs control to output time information 130 related to the transmission / reception time 112 as the first control, regardless of whether the transmission / reception time 112 is greater than or equal to the reference time 114. The time information 130 is an example of "time information" involved in the present invention.

[0144] exist Figure 6 In the illustrated example, as an example of the method in which the control unit 82B outputs the time information 130 , the method in which the control unit 82B displays the time information 130 in the second display area 35B is illustrated.

[0145] The first control performed by the control unit 82B includes displaying the frame 40 on the screen 35 (in a state where it can be compared with the time information 130). Figure 6 The example shown is the control on the first display area 35A). Figure 6 In the example shown, time information 130 is displayed in the second display area 35B, and in order to obtain a recognition result 96 received by the external I / F 80 at the receiving time 98 for calculating the sending and receiving time 112 corresponding to the time information 130, a frame 40 identical to the frame 40 input into the recognition model 94 is displayed in the first display area 35A.

[0146] Here, an example is given of displaying a frame 40 identical to the frame 40 input into the recognition model 94 in the first display area 35A in order to obtain a recognition result 96 received by the external I / F 80 at the receiving time 98 for calculating the receiving and sending time 112 corresponding to the time information 130. However, this is only an example, and a frame 40 obtained after the frame 40 input into the recognition model 94 in order to obtain a recognition result 96 received by the external I / F 80 at the receiving time 98 for calculating the receiving and sending time 112 corresponding to the time information 130 (for example, a frame 40 obtained after several to several dozen frames) may be displayed in the first display area 35A.

[0147] The time information 130 includes first specific information 130A that can identify the transmission / reception time 112 itself and second specific information 130B that can identify the difference between the transmission / reception time 112 and the reference time 114. The first specific information 130A and the second specific information 130B are information visualized by text. The first specific information 130A is an example of the “first specific information” involved in the present invention, and the second specific information 130B is an example of the “second specific information” involved in the present invention.

[0148] exist Figure 6In the example shown, the transmission and reception time 112 itself is displayed in the second display area 35B as the first specific information 130A in text form. The first specific information 130A is displayed in the second display area 35B as information that can be identified as indicating a time lag (ie, communication delay time).

[0149] exist Figure 6 In the example shown, as the second specific information 130B, length information that can determine the length of the transmission / reception time 112 relative to the reference time 114 is displayed in the second display area 35B in text form. An example of the length information is information that can determine whether the transmission / reception time 112 is longer than the reference time 114. Figure 6 In the example shown, if the transmission / reception time 112 is greater than the reference time 114, a text "high" is displayed as the long / short information, and if the transmission / reception time 112 is less than the reference time 114, a text "low" is displayed as the long / short information. In addition, the second specific information 130B is displayed in the second display area 35B with content that can be determined as information indicating the degree of time lag. In addition, here, the text "high" and the text "low" are examples of the "long / short information" involved in the present invention.

[0150] Furthermore, here, as the length information, a text such as "high" and a text such as "low" are illustrated, but this is only an example. When the transmission / reception time 112 is greater than the reference time 114, some information that can determine that the transmission / reception time 112 is greater than the reference time 114 (for example, an indicator that can determine that the transmission / reception time 112 is greater than the reference time 114) can be output, and when the transmission / reception time 112 is less than the reference time 114, some information that can determine that the transmission / reception time 112 is less than the reference time 114 (for example, an indicator that can determine that the transmission / reception time 112 is less than the reference time 114) can be output.

[0151] In addition, when the transmission / reception time 112 is less than the reference time 114, some information that can be determined to be less than the reference time 114 may not be output. This is because, when the transmission / reception time 112 is less than the reference time 114, the content of the frame 40 displayed in the first display area 35A matches the processing result information 116, and even if the time information 130 is not output, there is no particular obstacle for the doctor 12 to perform endoscopic examination.

[0152] The control unit 82B changes the form of visualizing the time information 130 according to the transmission / reception time 112. For example, the control unit 82B makes the visual recognition of the time information 130 higher when the transmission / reception time 112 is greater than the reference time 114 than when the transmission / reception time 112 is less than the reference time 114. Figure 6In the example shown, the control unit 82B displays the time information 130 at different visual recognition levels when the transmission / reception time 112 is less than the reference time 114 and when the transmission / reception time 112 is greater than the reference time 114. For example, the control unit 82B displays the time information 130 at the first visual recognition level when the transmission / reception time 112 is less than the reference time 114, and displays the time information 130 at the second visual recognition level when the transmission / reception time 112 is greater than the reference time 114.

[0153] The second visual recognition level is a higher visual recognition level than the first visual recognition level. For example, the height of the visual recognition level is determined by the size of the text of the time information 130. Figure 6 In the example shown, the size of the text of the time information 130 at the second visual recognizability level is larger than the size of the text of the time information 130 at the first visual recognizability level. That is, the control unit 82B makes the size of the text of the time information 130 when the transmission / reception time 112 is greater than the reference time 114 greater than the size of the text of the time information 130 when the transmission / reception time 112 is less than the reference time 114, thereby making the visual recognizability of the time information 130 when the transmission / reception time 112 is greater than the reference time 114 higher than the visual recognizability of the time information 130 when the transmission / reception time 112 is less than the reference time 114.

[0154] In addition, here, an example of changing the size of the text of the time information 130 to make the visual recognition level different is given, but this is only an example. For example, the transparency of the text of the time information 130, the color depth of the text of the time information 130, and / or the brightness of the text of the time information 130 can also be changed to make the visual recognition level different. In addition, when the transmission and reception time 112 is greater than the reference time 114, the time information 130 is displayed, and when the transmission and reception time 112 is less than the reference time 114, the time information 130 is not displayed, thereby making it possible to make the visual recognition of the time information 130 different.

[0155] Next, refer to Figure 7 The operation of the portion of the medical support system 1 according to the present invention will be described. Figure 7 The illustrated flow of the medical support process is an example of the “medical support method” according to the present invention.

[0156] exist Figure 7In the medical support process shown, first, in step ST10, the control unit 82B determines whether the camera 52 has captured one frame of images in the large intestine 28. In step ST10, if the camera 52 has not captured one frame of images in the large intestine 28, the determination is negative, and the medical support process moves to step ST28. In step ST10, if the camera 52 has captured one frame of images in the large intestine 28, the determination is positive, and the medical support process moves to step ST12.

[0157] In step ST12, the communication unit 82A and the control unit 82B acquire the frame 40 obtained by photographing the inside of the large intestine 28 (for example, the intestinal wall 32) by the camera 52. Then, the control unit 82B displays the frame 40 in the first display area 35A. In addition, when the frame 40 is displayed in the first display area 35A, the control unit 82B updates the frame 40 displayed in the first display area 35A. That is, by repeatedly executing the process of step ST12, the frame 40 is displayed in the first display area 35A in a real-time preview mode. After executing the process of step ST12, the medical support process is transferred to step ST14.

[0158] In step ST14, the communication unit 82A causes the external I / F 80 to transmit the frame 40 acquired in step ST12 to the processing device 2. Then, the communication unit 82A acquires the transmission time 92. After the process of step ST14 is executed, the medical support process proceeds to step ST16.

[0159] When the frame 40 is transmitted to the processing device 2 by executing the process of step ST14 , the processing device 2 performs the recognition process 95 on the frame 40 and transmits the recognition result 96 based on the recognition process 95 to the medical support device 24 .

[0160] In step ST16, the communication unit 82A determines whether the recognition result 96 is received by the external I / F 80. In step ST16, if the recognition result 96 is not received by the external I / F 80, the determination is negative, and the determination of step ST16 is performed again. In step ST16, if the recognition result 96 is received by the external I / F 80, the determination is positive, and the medical support process shifts to step ST18. Furthermore, in step ST16, if the recognition result 96 is received by the external I / F 80, the communication unit 82A acquires the reception time 98.

[0161] In step ST18, the control unit 82B acquires the transmission time 92 and the reception time 98 from the communication unit 82A, and calculates the transmission and reception time 112 based on the transmission time 92 and the reception time 98. After the process of step ST18 is executed, the medical support process proceeds to step ST20.

[0162] In step ST20, the control unit 82B determines whether the transmission / reception time 112 calculated in step ST18 is less than the reference time 114. In step ST20, if the transmission / reception time 112 calculated in step ST18 is not less than the reference time 114, the determination is negative, and the medical support process moves to step ST26. In step ST20, if the transmission / reception time 112 calculated in step ST18 is less than the reference time 114, the determination is positive, and the medical support process moves to step ST22.

[0163] In step ST22, the control unit 82B displays the processing result information 116 on the screen 35. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0164] In step ST24, the control unit 82B displays the time information 130 in the second display area 35B at the first visibility level. After the process of step ST24 is executed, the medical support process proceeds to step ST28.

[0165] In step ST26, the control unit 82B displays the time information 130 in the second display area 35B at the second visibility level. After the process of step ST26 is executed, the medical support process proceeds to step ST28.

[0166] In step ST28, the control unit 82B determines whether the condition for ending the medical support process is satisfied. An example of the condition for ending the medical support process is a condition in which a command to end the medical support process is given to the medical support system 1 (for example, a condition in which a command to end the medical support process is received by the receiving device 64).

[0167] In step ST28, if the condition for ending the medical support process is not satisfied, the determination is negative, and the medical support process moves to step ST10. In step ST28, if the condition for ending the medical support process is satisfied, the determination is positive, and the medical support process ends.

[0168] As described above, in the medical support system 1, the time information 130 is displayed on the screen 35. The time information 130 is information about the transmission and reception time 112 required from the transmission of the frame 40 from the external I / F 80 of the medical support device 24 to the reception of the recognition result 96 obtained by the processing device 2 performing the recognition process 95 on the frame 40 by the external I / F 80 of the medical support device 24. The transmission and reception time 112 can also be said to be an index indicating the degree of time lag generated by the transmission and reception between the medical support device 24 and the processing device 2. Therefore, in the medical support system 1, the display mode of the processing result information 116 is made different according to the transmission and reception time 112. In this way, the doctor 12 can visually recognize the degree of time lag generated by the transmission and reception between the medical support device 24 and the processing device 2 according to the display mode of the time information 130 and / or the processing result information 116 displayed on the screen 35.

[0169] In the medical support system 1 , the time information 130 and the frame 40 are displayed on the screen 35 in a comparable state. Therefore, the doctor 12 can determine whether the content of the frame 40 matches the processing result information 116 while referring to the time information 130 .

[0170] Furthermore, in the medical support system 1, the first specific information 130A is displayed in the second display area 35B. The first specific information 130A is information that can identify the transmission and reception time 112 itself. Therefore, the doctor 12 can visually recognize the transmission and reception time 112 itself that indicates the degree of time lag generated by transmission and reception between the medical support device 24 and the processing device 2.

[0171] Furthermore, in the medical support system 1, the second specific information 130B is displayed in the second display area 35B. The second specific information 130B is information that can identify the difference between the transmission and reception time 112 and the reference time 114 (i.e., the length of the transmission and reception time 112 relative to the reference time 114). Therefore, the doctor 12 can visually recognize the difference between the transmission and reception time 112 and the reference time 114 (i.e., the length of the transmission and reception time 112 relative to the reference time 114) indicating the degree of time lag generated by transmission and reception between the medical support device 24 and the processing device 2.

[0172] In addition, in the medical support system 1, the time information 130 is displayed as visualized information in the second display area 35B. For example, the time information 130 is visualized as text. Therefore, the degree of time lag generated by the transmission and reception between the medical support device 24 and the processing device 2 can be visualized and perceived by the doctor 12.

[0173] Furthermore, in the medical support system 1, the visualization format is changed according to the transmission / reception time 112. For example, the size of the text of the time information 130 is changed according to the transmission / reception time 112. In this case, for example, the control unit 82B makes the size of the text of the time information 130 displayed in the second display area 35B larger when the transmission / reception time 112 is greater than the reference time 114 than when the transmission / reception time 112 is less than the reference time 114. Thus, the doctor 12 can visually recognize whether the transmission / reception time 112 is greater than the reference time 114.

[0174] In the medical support system 1 , the display mode of the processing result information 116 is changed according to whether the transmission / reception time 112 is longer than the reference time 114 . Therefore, the doctor 12 can determine whether the transmission / reception time 112 is longer than the reference time 114 according to the display mode of the processing result information 116 .

[0175] Furthermore, in the medical support system 1, when the transmission / reception time 112 is less than the reference time 114, the processing result information 116 is displayed on the screen 35, and when the transmission / reception time 112 is greater than the reference time 114, the processing result information 116 is not displayed on the screen 35. Thus, the doctor 12 can visually distinguish between the case where the transmission / reception time 112 is less than the reference time 114 and the case where the transmission / reception time 112 is greater than the reference time 114. Furthermore, since the processing result information 116 is not displayed on the screen 35, the doctor 12 can recognize that the content of the frame 40 does not match the processing result information 116.

[0176] Then, in the medical support system 1, the frame 40 and the processing result information 116 are displayed on the screen 35 in a state where they can be compared. Therefore, the doctor 12 can determine whether the frame 40 and the processing result information 116 match each other.

[0177] Furthermore, in the medical support system 1, when the transmission / reception time 112 is less than the reference time 114, the frame 40 and the processing result information 116 are displayed on the screen 35 in a comparable state. Therefore, the doctor 12 can visually grasp that the transmission / reception time 112 is less than the reference time 114, and can determine whether the frame 40 and the processing result information 116 match.

[0178] Furthermore, in the medical support system 1, as the reference time 114 to be compared by the control unit 82B with the transmission and reception time 112, the longest time during which the degree of affinity between the frame 40 displayed in the first display area 35A and the recognition result 96 falls within the allowable range is used. Therefore, compared with the case where the time during which the degree of affinity between the frame 40 displayed in the first display area 35A and the recognition result 96 does not fall within the allowable range is used as the reference time 114, it is possible to suppress a situation in which the doctor 12 mistakenly recognizes that the frame 40 displayed in the first display area 35A at the current time matches the processing result information 116 displayed on the screen 35 even though the recognition result 96 that does not match the frame 40 displayed in the first display area 35A at the current time is obtained due to an excessive time lag.

[0179] In addition, in the above embodiment, when the transmission and reception time 112 is less than the reference time 114, the lesion location map 106 is displayed in the second display area 35B as one of the processing result information 116 and as one of the auxiliary information 44, but the present invention is not limited to this. Figure 8 As shown, regardless of whether the transmission / reception time 112 is smaller than the reference time 114, the lesion location map 106 can be displayed in the second display area 35B.

[0180] The matching degree between the recognition result 96 and the frame 40 displayed in the first display area 35A when the transmission / reception time 112 is greater than the reference time 114 is lower than the matching degree between the recognition result 96 and the frame 40 displayed in the first display area 35A when the transmission / reception time 112 is less than the reference time 114. Therefore, in order to allow the doctor 12 to visually recognize this fact, the control unit 82B makes the visual recognition degree of the lesion position map 106 displayed in the second display area 35B lower when the transmission / reception time 112 is greater than the reference time 114 than the visual recognition degree of the lesion position map 106 displayed in the second display area 35B when the transmission / reception time 112 is less than the reference time 114. Thus, the doctor 12 can visually recognize whether the degree of time lag is greater than the reference time 114 based on the visual recognition degree of the lesion position map 106. Furthermore, the doctor 12 can visually grasp the degree of matching degree between the recognition result 96 and the frame 40 displayed in the first display area 35A based on the visual recognition degree of the lesion position map 106.

[0181] In the above embodiment, when the transmission and reception time 112 is longer than the reference time 114, the processing result information 116 is not displayed on the screen 35. However, this is only an example. Fig. 9As shown, even when the transmission / reception time 112 is greater than the reference time 114, the processing result information 116 can be displayed on the screen 35. In this case, the visual recognition of the processing result information 116 is reduced. That is, when the transmission / reception time 112 is greater than the reference time 114, the visual recognition of the processing result information 116 displayed on the screen 35 is reduced compared to the case where the transmission / reception time 112 is less than the reference time 114. Fig. 9 In the example shown, when the transmission / reception time 112 is greater than the reference time 114, the part name information 118, the lesion identification information 120, the discrimination information 122, and the size information 124 are displayed in a grayed-out state in the third display area 35C, and the line of the boundary box BB is displayed thinner than the line of the boundary box BB displayed when the transmission / reception time 112 is less than the reference time 114. There are many methods for reducing visual recognition. For example, there are methods for reducing visual recognition by increasing the transparency of display, methods for reducing visual recognition by covering the display part with a certain mask, and methods for reducing visual recognition by flashing display.

[0182] And, as an example, Fig. 9 As shown, when the transmission / reception time 112 is less than the reference time 114, the time information 130 may not be displayed on the screen 35. By not displaying the time information 130 on the screen 35, the doctor 12 can perform medical treatment without paying attention to the time information 130.

[0183] Moreover, as an example, Fig. 9 As shown, when the transmission and reception time 112 is greater than the reference time 114, a notification mark 132 may be displayed on the screen 35 (in Fig. 9 In the example shown, the second display area 35B). The notification mark 132 is a mark for notifying that the transmission / reception time 112 is greater than the reference time 114 (in other words, the processing result information 116 displayed on the screen 35 does not match the content of the frame 40 displayed in the first display area 35A). In this way, by displaying the notification mark 132 on the screen 35, the doctor 12 can visually grasp that the transmission / reception time 112 is greater than the reference time 114 and that the processing result information 116 displayed on the screen 35 does not match the content of the frame 40 displayed in the first display area 35A based on the notification mark 132. In addition, the notification mark 132 is an example of the "time information" and "mark" involved in the present invention.

[0184] exist Fig. 9In the example shown, when the transmission / reception time 112 is greater than the reference time 114, the visual recognition of the bounding box BB displayed on the screen 35 is reduced compared to when the transmission / reception time 112 is less than the reference time 114. However, this is only an example, and the display method (for example, color, line type, or shape, etc.) of the bounding box BB may be different when the transmission / reception time 112 is greater than the reference time 114 and when the transmission / reception time 112 is less than the reference time 114. Furthermore, the display method (for example, color, line type, or shape, etc.) of the bounding box BB may be different depending on the type and / or category of the lesion 42.

[0185] In the above embodiment, the time information 130 and the processing result information 116 are displayed on the screen 35, but this is only an example, and the time information 130 and / or the processing result information 116 may be outputted from a speaker (not shown) in the form of sound. In this case, for example, the control unit 82B may adjust the sound quality and / or volume of the sound outputted from the speaker so that the sound when the transmission / reception time 112 is greater than the reference time 114 is easier to hear than the sound when the transmission / reception time 112 is less than the reference time 114.

[0186] In the above embodiment, the frame 40 displayed in the first display area 35A when the transmission / reception time 112 is less than the reference time 114 is the same as the frame 40 displayed in the first display area 35A when the transmission / reception time 112 is greater than the reference time 114. However, the present invention is not limited to this. Fig.10 As shown, the control unit 82B may use a frame 40 obtained after the frame 40 displayed in the first display area 35A when the transmission / reception time 112 is less than the reference time 114, as the frame 40 displayed in the first display area 35A when the transmission / reception time 112 is greater than the reference time 114. Here, the frame 40 displayed in the first display area 35A when the transmission / reception time 112 is greater than the reference time 114 is an example of the “second medical image” and the “third medical image” involved in the present invention.

[0187] In this way, the frame 40 obtained after the frame 40 displayed in the first display area 35A when the sending and receiving time 112 is less than the reference time 114 is used as the frame 40 displayed in the first display area 35A when the sending and receiving time 112 is greater than the reference time 114. Therefore, even when the sending and receiving time 112 is greater than the reference time 114, the content of the frame 40 displayed in the first display area 35A can be matched with the processing result information 116 displayed on the screen 35.

[0188] Furthermore, when the transmission and reception time 112 is longer than the reference time 114, guidance information recommending that the camera 52 inserted into the large intestine 28 be stopped or the speed of inserting or removing the camera 52 be slowed down may be displayed on the screen 35. In this way, the content of the frame 40 displayed in the first display area 35A can be matched with the processing result information 116 displayed on the screen 35.

[0189] In the above embodiment, the time information 130 is displayed in the second display area 35B, but this is only an example, and the time information 130 may be displayed in an area other than the second display area 35B. The same applies to the processing result information 116.

[0190] Furthermore, the processing result information 116 and the time information 130 do not need to be displayed only on the screen 35 of the display device 18. For example, the processing result information 116 may be displayed in a distributed manner on a plurality of display devices, and the time information 130 may also be displayed in a distributed manner on a plurality of display devices.

[0191] In the above-mentioned embodiment, the lesion 42 is illustrated as an example of the "characteristic area" involved in the present invention, but the present invention is not limited to this. Even if a resection area, a bleeding area, a marking area, an organ or a treatment instrument (for example, a hemostatic clip placed in the body) is used instead of the lesion 42, the technology of the present invention is also applicable.

[0192] In the above embodiment, the camera 52 mounted on the endoscope 16 is exemplified, but the present invention is not limited thereto. For example, the technology of the present invention can also be applied to a fundus camera or a camera mounted on a medical microscope.

[0193] In the above embodiment, the frame 40 is obtained by photographing the inside of the large intestine 28 by the camera 52 mounted on the endoscope 16, but this is only an example, and the frame 40 can also be a medical image obtained by various medical imaging equipment such as a radiation imaging device, MRI, CT or an ultrasonic probe.

[0194] In the above embodiment, the recognition model 94 is illustrated, but as the recognition model 94, a so-called generative AI may be used. For example, command data (so-called prompt) including the frame 40 may be input to the generative AI, and the recognition result 96 or the processing result information 116 may be output from the generative AI. Furthermore, the generative AI may output auxiliary information 44 in addition to the recognition result 96 or the processing result information 116 based on the input prompt. As an example of generative AI, ChatGPT using GPT-4 (Internet search <https: / / openai.com / gpt-4>) and the like may be cited.

[0195] The command data including at least a part of the processing result information 116 can be used as input information for the generating AI. As an example of the information output from the generating AI, there can be cited information related to the operation content of the camera 52, information indicating the content of the medical treatment recommended for the endoscope examination, and / or information indicating the content of the medical treatment recommended after the endoscope examination. The information output from the generating AI can be stored in various storage areas (for example, the storage device 76 and / or 86), or displayed on the display device 18 as auxiliary information 44, or printed on a medium by a printer, or output from a speaker in the form of sound.

[0196] In the above embodiment, the recognition process 95 using AI in the bounding box method is exemplified, but this is only an example, and for example, the recognition process using AI in the segmentation method may be performed instead of the recognition process 95 using AI in the bounding box method. Furthermore, the recognition process using a non-AI method (for example, a template matching method) may be performed instead of the recognition process using the AI ​​method, or the recognition process using a combination of the non-AI method and the AI ​​method may be performed.

[0197] In the above embodiment, the medical support process is performed by the computer 78 , but the present invention is not limited thereto, and at least a part of the process included in the medical support process may be performed by a device provided outside the computer 78 .

[0198] In the above embodiment, the processing device 2 is implemented by cloud computing, but this is only an example. The processing device 2 may also be implemented by network computing such as fog computing, edge computing, or grid computing.

[0199] In the above embodiment, the example in which the medical support program 90 is stored in the storage device 86 is described, but the present invention is not limited to this. For example, the medical support program 90 may be stored in a non-transitory storage medium such as an SSD or a USB memory that can be read by a portable computer. The medical support program 90 stored in the non-transitory storage medium is installed in the computer 78 of the endoscope device 10. The processor 82 executes the medical support process according to the medical support program 90.

[0200] Furthermore, the medical support program 90 may be stored in a storage device such as another computer or server connected to the endoscope apparatus 10 via a network, and the medical support program 90 may be downloaded in response to a request from the endoscope apparatus 10 and installed on the computer 78 .

[0201] Furthermore, the entire medical support program 90 does not need to be stored in a storage device such as another computer or server device connected to the endoscope device 10 , or the entire medical support program 90 is not necessarily stored in the storage device 86 , and a portion of the medical support program 90 may be stored.

[0202] As hardware resources for executing medical support processing, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, which functions as a hardware resource for executing medical support processing by executing software, i.e., a program, can be cited. Also, as a processor, for example, a processor, i.e., a dedicated circuit, such as an FPGA, a PLD, or an ASIC, which has a circuit structure specially designed for executing specific processing can be cited. Any processor has a memory built in or connected, and any processor executes medical support processing by using the memory.

[0203] The hardware resource for executing the medical support processing may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resource for executing the medical support processing may be a single processor.

[0204] As an example of a system composed of one processor, there is a first method in which one processor is composed of a combination of one or more CPUs and software, and the processor functions as a hardware resource for executing medical support processing. There is a second method in which a processor is used, such as a SoC, to realize the functions of an entire system including multiple hardware resources for executing medical support processing by using one IC chip. In this way, medical support processing is realized by using one or more of the above-mentioned various processors as hardware resources.

[0205] In addition, as the hardware structure of these various processors, more specifically, a circuit composed of a combination of circuit elements such as semiconductor elements can be used. In addition, the above-mentioned medical support processing is only an example. Therefore, of course, unnecessary steps can be deleted, new steps can be added, or the processing order can be replaced within the scope of the subject matter.

[0206] The above-described contents and illustrated contents are detailed descriptions of the parts involved in the present invention, which are only examples of the present invention. For example, the descriptions related to the above-described structures, functions, actions and effects are descriptions related to an example of the structures, functions, actions and effects of the parts involved in the present invention. Therefore, it is self-evident that unnecessary parts can be deleted from the above-described contents and illustrated contents, or new elements can be added, or replacement can be performed without departing from the gist of the present invention. Furthermore, in order to avoid complexity and facilitate understanding of the parts involved in the present invention, in the above-described contents and illustrated contents, on the basis of being able to implement the present invention, descriptions related to technical common sense that does not need to be specifically described are omitted.

[0207] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, and technical standard was specifically and individually stated.

[0208] Regarding the above embodiment, the following additional notes are also disclosed.

[0209] (Note 1)

[0210] A medical support device comprising:

[0211] communications device; and

[0212] processor,

[0213] The communication device sends the medical image to the processing device.

[0214] and receiving a processing result obtained by the processing device performing processing on the medical image,

[0215] The processor performs the first control and / or the second control, wherein the first control outputs time information related to the time required for the communication device to receive the processing result after sending the medical image to the processing device, and the second control outputs the processing result differently according to the time.

[0216] (Note 2)

[0217] The medical support device according to Supplementary Note 1, wherein:

[0218] The processor outputting the time information includes the processor displaying the time information on a screen.

[0219] (Note 3)

[0220] The medical support device according to Supplementary Note 2, wherein:

[0221] The first control includes a control of displaying the medical image on a screen in a state where the medical image can be compared with the time information when the time is smaller than a reference time.

[0222] This allows a user (for example, a doctor) to easily understand the relationship between the medical image processed by the processing device and the time lag generated in transmission and reception.

[0223] (Note 4)

[0224] The medical support device according to Supplementary Note 2, wherein:

[0225] The first control includes a control for displaying the medical image on a second screen in a state where the medical image can be compared with the time information when the time is equal to or longer than a reference time.

[0226] This allows a user (for example, a doctor) to easily understand the relationship between the medical image processed by the processing device and the time lag generated in transmission and reception.

[0227] Explanation of symbols

[0228] 1- medical support system, 2- processing device, 3- network, 10- endoscope device, 12- doctor, 14- nurse, 16- endoscope observation device, 18- display device, 20- light source device, 22- control device, 24- medical support device, 26- subject, 28- large intestine, 30- light, 32- intestinal wall, 34- carriage, 35- screen, 35A- first display area, 35B- second display area, 35C- third Display area, 39-endoscope dynamic image, 40-frame, 42-lesion, 44-auxiliary information, 46-operating unit, 48-insertion unit, 50-front end, 50A-front end surface, 52-camera, 54-illumination device, 54A, 54B-illumination window, 56-opening for treatment instrument, 58-treatment instrument, 60-treatment instrument insertion port, 62-universal cord, 64-receiving device, 66, 78-computer, 68, 88-bus, 70, 80-external I / F, 72, 82-processor, 74, 84-memory, 76, 86-storage device, 79-RTC, 82A-communication unit, 82B-control unit, 90-medical support program, 92-transmission time, 94-recognition model, 95-recognition processing, 96-recognition result, 98-reception time, 100-lesion presence information, 102-geometric characteristic information, 106-lesion location map, 108-lesion feature information, 110-site information, 112-transmission and reception time, 114-reference time, 116-processing result information, 118-site name information, 120-lesion identification information, 122-identification information, 124-size information, 126-past information, 130-time information, 130A-first specific information, 130B-second specific information, 132-notification mark, BB-bounding box.

Claims

1. A medical support device comprising: communications device; and processor, The communication device transmits the first medical image to the processing device and receives a processing result obtained by the processing device performing processing on the first medical image. The processor performs a first control and / or a second control, wherein the first control outputs time information related to the time required for the communication device to receive the processing result after sending the first medical image to the processing device, and the second control outputs the processing result differently according to the time.

2. The medical support device according to claim 1, wherein: The process of the processor outputting the time information includes the processor displaying the time information on a first screen.

3. The medical support device according to claim 2, wherein: The first control includes control for displaying the first medical image and / or a second medical image temporally acquired after the first medical image on a second screen in a state where the image can be compared with the time information.

4. The medical support device according to claim 1, wherein: The time information includes first specific information capable of identifying the time itself.

5. The medical support device according to claim 1, wherein: The time information includes second specific information capable of specifying a difference between the time and a first reference time.

6. The medical support device according to claim 5, wherein: The second specific information includes length information capable of specifying the length of the time relative to the first reference time.

7. The medical support device according to claim 1, wherein: The time information is visualized information.

8. The medical support device according to claim 7, wherein: The time information is information visualized through text and / or marks.

9. The medical support device according to claim 7, wherein: The processor changes a visualization form according to the time, the visualization form being a form in which the time information is visualized.

10. The medical support device according to claim 9, wherein: The processor makes the visual recognition of the time information when the time is equal to or greater than a second reference time higher than the visual recognition of the time information when the time is less than the second reference time.

11. The medical support device according to claim 1, wherein: The output method is a display method in which the processor displays the processing result on a third screen.

12. The medical support device according to claim 11, wherein: The display mode includes a first display mode and a second display mode, The first display mode is used when the time is equal to or longer than a third reference time, and it can be determined that the time is equal to or longer than the third reference time. The second display mode is used when the time is smaller than the third reference time, and can determine that the time is smaller than the third reference time.

13. The medical support device according to claim 12, wherein: The second display mode has higher visibility than the first display mode.

14. The medical support device according to claim 13, wherein: In the first display mode, the processing result is not displayed. In the second display mode, the processing result is displayed.

15. The medical support device according to claim 11, wherein: The second control includes control for displaying the first medical image and / or a third medical image temporally acquired after the first medical image on a fourth screen in a state where the first medical image can be compared with the processing result.

16. The medical support device according to claim 11, wherein: The second control includes a control of displaying the first medical image on a fourth screen in a state where the first medical image can be compared with the processing result when the time is less than a fourth reference time.

17. The medical support device according to claim 16, wherein: The fourth reference time is a maximum time during which the degree of affinity between the first medical image and the processing result falls within an allowable range.

18. The medical support device according to claim 1, wherein: The processing is AI processing of the first medical image.

19. The medical support device according to claim 1, wherein: The processing includes recognition processing for recognizing a characteristic region reflected in the first medical image.

20. The medical support device according to claim 19, wherein: The characteristic area is a lesion.

21. The medical support device according to claim 1, wherein: The first medical image is an endoscopic image generated by imaging the inside of the body with a camera mounted on an endoscope.

22. An endoscope device comprising: The medical support device according to any one of claims 1 to 21; and The camera is inserted into a body and captures an image of the body to generate the first medical image.

23. A medical support system comprising: Medical support devices; and processing device, The medical support device sends the first medical image to the processing device, The processing device processes the first medical image and transmits a processing result obtained by performing the processing to the medical support device. The medical support device receives the processing result sent from the processing device and performs a first control and / or a second control, wherein the first control outputs time information related to the time required for the medical support device to receive the processing result after the first medical image is sent to the processing device, and the second control outputs the processing result in a different manner according to the time.

24. A medical support method comprising the following steps: sending the first medical image to a processing device; receiving a processing result obtained by processing the first medical image by the processing device; and A first control and / or a second control is performed, wherein the first control outputs time information related to the time required from sending the first medical image to receiving the processing result, and the second control makes the output method of the processing result different according to the time.

25. A computer program product for causing a computer to execute a medical assistance process, the medical assistance process comprising the following steps: sending the first medical image to a processing device; receiving a processing result obtained by processing the first medical image by the processing device; and A first control and / or a second control is performed, wherein the first control outputs time information related to the time required from sending the first medical image to receiving the processing result, and the second control makes the output method of the processing result different according to the time.

Images