Medical support device, medical support system, operation method of medical support device, and program

Abstract

To provide a medical support device, a medical support system, an operation method of the medical support device, and a program capable of allowing a user or the like to whom an execution result obtained by executing a first process by the medical support device is provided from the medical support device to grasp switching contents when the processing contents of the first process are switched.SOLUTION: The medical support device includes a processor, and communicates with an external apparatus that transmits medical information including a medical image and / or modality-related information. A medical image is obtained by imaging a subject by a modality. The modality-related information is information related to the modality at the timing at which the medical image is obtained. The processor receives medical information transmitted from an external device, executes first process on the medical information, transmits an execution result, switches processing content of the first process in accordance with the medical information, and transmits switching content information indicating switching content.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a medical support device, a medical support system, an operation method for a medical support device, and a program. [Background technology]

[0002] Patent Document 1 discloses an information processing method including receiving first information associated with an operation of a medical testing device and outputting at least a part of the first information, the first information being associated with data collection performed by the medical testing device during operation. In the information processing method described in Patent Document 1, receiving the first information includes receiving the first information from a cloud storage device, the first information being transmitted from a data analysis device of the medical testing device to the cloud storage device, and the first information being determined based on input data of the medical testing device.

[0003] Patent Document 2 discloses an information processing device including an estimation means for estimating a risk of a subject developing a disease using a trained model that has learned the relationship between features acquired from a fundus image, biological information acquired by an examination device, and the risk of developing the disease, and a display control means for displaying the estimated risk of developing the disease on a display unit. In the information processing device described in Patent Document 2, an instruction from an examiner regarding the estimation of the risk of developing the disease is information obtained using at least one trained model among a trained model for character recognition, a trained model for voice recognition, and a trained model for gesture recognition.

[0004] Patent Document 3 discloses a method for supporting diagnosis of a disease using an endoscopic image of a digestive organ using a convolutional neural network. The method for supporting diagnosis described in Patent Document 3 trains a convolutional neural network using a first endoscopic image of the digestive organ and at least one definitive diagnosis result of a positive or negative diagnosis of a disease of the digestive organ, a past disease, a level of severity, a depth of invasion of the disease, or information corresponding to an imaged part, which corresponds to the first endoscopic image. The trained convolutional neural network outputs at least one of the probability of positive and / or negative of a disease of the digestive organ, the probability of a past disease, the level of severity of the disease, the depth of invasion of the disease, or the probability corresponding to an imaged part, based on a second endoscopic image of the digestive organ. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2023-526412 [Patent Document 2] Patent Publication No. 2021-039748 [Patent Document 3] JP 2020-078539 A Summary of the Invention

[0006] One embodiment of the present disclosure provides a medical support device, a medical support system, an operating method of a medical support device, and a program that enable a user, etc. to whom the medical support device provides an execution result obtained by executing a first process by the medical support device to understand the switching content when the processing content of the first process is switched. [Means for solving the problem]

[0007] A first aspect of the present disclosure is a medical support device that includes a processor and communicates with an external device that transmits medical information including medical images and / or modality-related information, wherein the medical images are obtained by imaging a subject using a modality, and the modality-related information is information regarding the modality at the time the medical image is obtained, and the processor receives the medical information transmitted from the external device, performs a first process on the medical information, transmits an execution result obtained by executing the first process, switches the processing content of the first process in accordance with the medical information, and transmits switching content information indicating the switching content when the processing content is switched.

[0008] A second aspect of the present disclosure is a medical support device according to the first aspect, wherein the first processing includes an AI processing for generating associated information by inputting medical information to a trained model that generates second information corresponding to associated information related to the medical information by inputting first information corresponding to the medical information.

[0009] A third aspect of the present disclosure is a medical support device according to the second aspect, in which switching content information when the processing content of the AI ​​processing is switched includes AI processing switching content information indicating the content to which the processing content of the AI ​​processing has been switched.

[0010] A fourth aspect of the present disclosure is a medical support device according to the third aspect, in which there are multiple types of trained models, and in AI processing, the trained model as the input destination of medical information is switched depending on the medical information, and when the trained model is switched, the AI ​​processing switching content information includes model switching content information indicating the content to which the trained model has been switched.

[0011] A fifth aspect of the present disclosure is a medical support device according to the fourth aspect, in which the model switching content information includes trained model identification information capable of identifying the trained model used to obtain the execution result transmitted by the processor.

[0012] A sixth aspect of the present disclosure is a medical support device according to the fourth or fifth aspect, in which the medical information includes modality-related information, and in the AI ​​processing, a trained model as an input destination for the medical information is switched depending on the modality-related information.

[0013] A seventh aspect of the present disclosure is a medical support device according to the sixth aspect, in which the medical information includes medical images, and in the AI ​​processing, a trained model as an input destination for the medical images is switched depending on the modality-related information.

[0014] An eighth aspect of the present disclosure is a medical support device relating to any one of the fourth to seventh aspects, in which the medical information includes a medical image and modality-related information, the modality-related information includes light source identification information capable of identifying a light source used in imaging to obtain the medical image, and in the AI ​​processing, a trained model as an input destination for the medical image is switched according to the light source identification information.

[0015] A ninth aspect of the present disclosure is a medical support device relating to any one of the fourth to eighth aspects, in which the medical information includes a medical image and modality-related information, the modality-related information includes magnification specification information capable of specifying a magnification to be applied to the medical image, and in the AI ​​processing, a trained model as an input destination for the medical image is switched according to the magnification specification information.

[0016] A tenth aspect of the present disclosure is a medical support device relating to any one of the fourth to ninth aspects, in which the medical information includes modality-related information, the modality-related information includes modality-specific information capable of identifying the modality, and in the AI ​​processing, a trained model as an input destination for the medical information is switched depending on the modality-specific information.

[0017] An eleventh aspect of the present disclosure is a medical support device according to the tenth aspect, in which the medical information includes medical images, and in the AI ​​processing, a trained model as an input destination for the medical images is switched depending on modality-specific information.

[0018] A twelfth aspect of the present disclosure is a medical support device relating to any one of the second to eleventh aspects, in which the medical information includes medical images and modality-related information, the modality-related information includes image quality information that is information regarding the image quality of the medical images, and the processing content of the AI ​​processing is switched depending on the image quality information.

[0019] A thirteenth aspect of the present disclosure is a medical support device according to the twelfth aspect, wherein the image quality information includes parameters defining image quality and / or image mode specific information identifying an image mode applied to a modality to obtain a medical image.

[0020] A fourteenth aspect of the present disclosure is a medical support device relating to any one of the second to thirteenth aspects, in which the medical information includes a medical image, the first information is a first image corresponding to the medical image, the second information is information capable of identifying a first area corresponding to a characteristic area appearing in the medical image, and the trained model generates, as related information, characteristic area identification information capable of identifying the characteristic area when the medical image is input.

[0021] A fifteenth aspect of the present disclosure is a medical support device relating to any one of the first to fourteenth aspects, wherein the modality is an endoscopic device and the medical image is an endoscopic image obtained by imaging the inside of a subject's body using the endoscopic device.

[0022] A sixteenth aspect of the present disclosure is a medical support system including a medical support device according to any one of the first to fifteenth aspects and an external device.

[0023] A seventeenth aspect of the present disclosure is a medical support system according to the sixteenth aspect, in which the external device transmits the medical information to the medical support device on condition that a change has occurred in the medical information.

[0024] An eighteenth aspect of the present disclosure is a medical support system according to the sixteenth aspect, in which the external device transmits a part of the medical information to the medical support device on condition that a change occurs in the part included in the medical information.

[0025] A 19th aspect of the present disclosure is an operation method of a medical support device that communicates with an external device that transmits medical information including medical images and / or modality-related information, wherein the medical images are obtained by imaging a subject using a modality, and the modality-related information is information regarding the modality at the time the medical images are obtained, the operation method of the medical support device including: receiving the medical information transmitted from the external device, performing a first processing on the medical information, transmitting an execution result obtained by performing the first processing, switching the processing content of the first processing in accordance with the medical information, and transmitting switching content information indicating the switching content when the processing content is switched.

[0026] A twentieth aspect of the present disclosure is a program for causing a computer to execute medical support processing, which is applied to a medical support device that communicates with an external device that transmits medical information including medical images and / or modality-related information, wherein the medical images are obtained by imaging a subject using a modality, the modality-related information is information regarding the modality at the time the medical image is obtained, and the medical support processing includes executing a first processing on the medical information, transmitting an execution result obtained by executing the first processing, switching the processing content of the first processing in accordance with the medical information, and transmitting switching content information indicating the switching content when the processing content is switched. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a conceptual diagram showing an example of how the medical support system is used by a doctor. [Diagram 2] 1 is a conceptual diagram showing an example of an overall configuration of a medical support system. [Diagram 3] 2 is a block diagram showing an example of a hardware configuration of an electrical system of the medical support system. FIG. [Figure 4]2 is a block diagram showing an example of main functions of a processor included in the server and an example of information stored in a storage. FIG. [Diagram 5] 4 is a conceptual diagram showing an example of processing contents performed by the server and the endoscope device. FIG. [Figure 6] 4 is a conceptual diagram showing an example of processing contents performed by the server and the endoscope device. FIG. [Figure 7] 13 is a flowchart showing an example of the flow of a medical support process. [Figure 8] 6 is a first modified example of the configuration shown in FIG. [Figure 9] 7 is a first modified example of the configuration shown in FIG. [Figure 10] 6 is a second modified example of the configuration shown in FIG. [Figure 11] 7 is a second modified example of the configuration shown in FIG. [Figure 12] 7 is a third modified example of the configuration shown in FIG. [Figure 13] 7 is a third modified example of the configuration shown in FIG. [Figure 14] 7 is a fourth modified example of the configuration shown in FIG. [Figure 15] 7 is a fourth modified example of the configuration shown in FIG. [Figure 16] 5. This is a fifth modified example of the configuration shown in FIG. [Figure 17] 7 is a fifth modified example of the configuration shown in FIG. [Figure 18] 6 is a sixth modified example of the configuration shown in FIG. [Figure 19] 7 is a sixth modified example of the configuration shown in FIG. [Figure 20] 10 is a conceptual diagram showing an example of a manner in which medical information is transmitted from an endoscope device to a server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, examples of embodiments of a medical support device, a medical support system, an operating method of a medical support device, and a program according to the present disclosure will be described with reference to the attached drawings.

[0029] First, the terms used in the following description will be explained.

[0030] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network".5G is an abbreviation for "5th Generation Mobile Communication System." IC is an abbreviation for "Integrated Circuit." MRI is an abbreviation for "Magnetic Resonance Imaging." CT is an abbreviation for "Computed Tomography." THI is an abbreviation for "Tissue Harmonic Imaging." CH is an abbreviation for "Compound Harmonic." CHI is an abbreviation for "Contrast Harmonic Imaging."

[0031] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be one physical or virtual arithmetic device, or a combination of multiple physical or virtual arithmetic devices. Also, a processor may be one type of arithmetic device, or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0032] In the following description, a signed memory is a memory, such as a RAM, in which information is temporarily stored and which is used as a working memory by a processor.

[0033] In the following description, a storage with a symbol is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0034] In the following embodiments, the external I / F with a symbol controls the transmission and reception of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor and an antenna may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0035] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0036] FIG. 1 is a conceptual diagram showing an example of an aspect in which a medical support system 1 is used. As shown in FIG. 1, the medical support system 1 includes a server 2 and an endoscope device 10, and the server 2 and the endoscope device 10 are communicatively connected via a network 3. An example of the network 3 is the Internet. However, the Internet is merely an example, and other examples of the network 3 include a WAN and / or a LAN. In this embodiment, the medical support system 1 is an example of a "medical support system" according to the present disclosure, the server 2 is an example of a "medical support device" according to the present disclosure, and the endoscope device 10 is an example of a "modality," "external device," and "endoscope device" according to the present disclosure.

[0037] The endoscope device 10 is used by a doctor 12 in an endoscopic examination or the like. The endoscopic examination is assisted by staff such as nurses. In this embodiment, the doctor 12 is a user of the endoscope device 10 and also a user of the medical support system 1.

[0038] The endoscope device 10 includes an endoscope body 16, a display device 18, a light source device 20, a control device 22, and a function expansion device 24. The endoscope device 10 is a modality for performing medical treatment inside the body of a subject 26 (e.g., a patient) using the endoscope body 16. The medical treatment includes observation, treatment, and the like. In this embodiment, the upper digestive tract 28 (e.g., esophagus, stomach, duodenum, etc.), which is a tubular organ of the subject 26, is the target on which the doctor 12 performs medical treatment. In this embodiment, the subject 26 is an example of a "subject" according to the present disclosure.

[0039] In the endoscope device 10, imaging of the inside of the body of a subject 26 is performed by the endoscope body 16. In this embodiment, imaging refers to a process of detecting physical energy (e.g., reflected light or reflected ultrasonic waves, etc.) and converting the detection results into an image.

[0040] In this embodiment, a hybrid type endoscopic device of an optical upper endoscope device and an ultrasonic upper endoscope device (i.e., an endoscopic device combining an optical upper endoscope device and an ultrasonic upper endoscope device) is used as an example of the endoscopic device 10. The optical upper endoscope device refers to, for example, an endoscopic device that captures an image of reflected light obtained by irradiating light 30 in the upper digestive tract 28 with an endoscope main body 16 inserted into the upper digestive tract 28 and reflecting it on an inner wall 32 of the upper digestive tract 28. The ultrasonic upper endoscope device refers to, for example, an endoscopic device that generates an image based on a reflected wave obtained by emitting ultrasonic waves in the upper digestive tract 28 with an endoscope main body 16 inserted into the upper digestive tract 28.

[0041] Here, for convenience of explanation, an upper endoscope device used for upper endoscopy is illustrated, but this is merely one example, and the present disclosure can be applied to a lower endoscope device used for lower endoscopy. The present disclosure can also be applied to an endoscope device used for endoscopic examination of organs other than the digestive system (for example, a bronchoscope device, an otorhinolaryngological endoscope device, an intracerebral endoscope device, a laparoscope, etc.).

[0042] The information obtained by the endoscope device 10 is transmitted to the server 2 via the network 3. An example of the server 2 is a cloud server. The cloud server is merely one example, and the server 2 may be an on-premise server. The server 2 receives the information transmitted from the endoscope device 10, executes processing using the received information, and transmits the processing results obtained by executing the processing to the endoscope device 10.

[0043] The endoscope body 16 is inserted into the upper digestive tract 28 by being operated by the doctor 12. The endoscope device 10 causes the endoscope body 16 inserted into the upper digestive tract 28 of the subject 26 to capture images of the inside of the subject 26's upper digestive tract 28, and generates a medical image 29 showing the state of the inside of the subject 26. In this embodiment, the medical image 29 is an example of a "medical image" and an "endoscopic image" according to the present disclosure.

[0044] In this embodiment, the concept of medical image 29 includes optical image 29A and ultrasound image 29B. The endoscope device 10 generates an optical image 29A by capturing an image of reflected light obtained by irradiating light 30 in the upper digestive tract 28 with the endoscope body 16 inserted in the upper digestive tract 28 and reflecting the light on an inner wall 32 of the upper digestive tract 28. The endoscope device 10 also generates an ultrasound image 29B based on a reflected wave obtained by emitting ultrasound in the upper digestive tract 28 with the endoscope body 16 inserted in the upper digestive tract 28 and reflecting the ultrasound within the body of the subject 26. The doctor 12 performs a medical procedure on the upper digestive tract 28 based on the medical image 29 captured by the endoscope body 16.

[0045] In this embodiment, an endoscopic examination of the upper digestive tract 28 is illustrated, but this is merely one example, and the present disclosure also applies to an endoscopic examination of a hollow organ such as the large intestine or trachea.

[0046] The light source device 20, the control device 22, and the function expansion device 24 are installed on a wagon 34. The wagon 34 has a plurality of platforms arranged vertically, and the function expansion device 24, the control device 22, and the light source device 20 are installed from the lower platform to the upper platform. In addition, the display device 18 is installed on the top platform of the wagon 34.

[0047] The control device 22 controls the entire endoscope device 10. The function expansion device 24 performs various processes on information obtained from the control device 22 and information obtained from the server 2 (for example, an image of the inner wall 32 captured by the endoscope body 16, and a processing result by the server 2, etc.). The function expansion device 24 is communicably connected to the server 2 via the network 3, and by requesting the server 2 to provide a service, the function expansion device 24 receives the requested service from the server 2.

[0048] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Also, instead of the display device 18 or together with the display device 18, a tablet terminal with a display may be used.

[0049] A screen 35 is displayed on the display device 18. The screen 35 includes a plurality of display areas. In the example shown in FIG. 1, a first display area 35A and a second display area 35B are shown as an example of a plurality of display areas. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as a main display area, and the second display area 35B is used as a sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited thereto, and may be any size relationship that fits within the screen 35.

[0050] A medical image 29 is displayed in the first display area 35A. For example, an optical image 29A and an ultrasound image 29B are selectively displayed in the first display area 35A. In the example shown in FIG. 1, the optical image 29A is displayed in the first display area 35A. The medical image 29 displayed in the first display area 35A is a moving image. In the example shown in FIG. 1, a moving image showing the inner wall 32 is shown as an example of the optical image 29A displayed in the first display area 35A.

[0051] The inner wall 32 shown in the optical image 29A displayed in the first display area 35A includes a lesion 36 (e.g., one lesion 36 in the example shown in FIG. 1) as a region of interest (i.e., a region to be observed) gazed upon by the doctor 12, and the doctor 12 can visually recognize the state of the inner wall 32 including the lesion 36 through the optical image 29A. In this embodiment, the lesion 36 is an example of a "characteristic region" according to the present disclosure.

[0052] There are various types of lesions 36, and examples of the types of lesions 36 include gastric ulcers and gastric cancer. The types exemplified here are types that are assumed in advance as types of lesions 36 when an endoscopic examination is performed on the upper gastrointestinal tract 28, and the types of lesions 36 differ depending on the organ that is subjected to the endoscopic examination. If the organ that is subjected to the endoscopic examination is the large intestine, examples of the types of lesions 36 include neoplastic polyps and non-neoplastic polyps. Examples of the types of neoplastic polyps include adenomatous polyps (e.g., SSL). Examples of the types of non-neoplastic polyps include hamartomatous polyps, hyperplastic polyps, and inflammatory polyps.

[0053] In this embodiment, for ease of explanation, an example is given in which one lesion 36 is captured in optical image 29A, but the present disclosure is not limited to this, and the present disclosure also applies when multiple lesions 36 are captured in optical image 29A.

[0054] In this embodiment, a lesion 36 is illustrated, however, this is merely one example, and the area of ​​interest (i.e., the area to be observed) gazed upon by the doctor 12 may be a characteristic area having some unique feature, such as an organ (e.g., the duodenal papilla), a mark, an artificial treatment device (e.g., an artificial clip), or a treated area (e.g., an area where traces remain after the removal of a polyp, etc.), etc.

[0055] The medical image 29 (optical image 29A in the example shown in FIG. 1) displayed in the first display area 35A is one frame included in a moving image including a plurality of frames in chronological order. In other words, the first display area 35A displays a plurality of frames in chronological order at a default frame rate (e.g., several tens of frames per second).

[0056] An example of a moving image displayed in the first display area 35A is a moving image in a live view format. The live view format is merely one example, and a moving image that is temporarily stored in a memory or the like and then displayed, such as a moving image in a post view format, may also be used. Furthermore, each frame included in a moving image for recording stored in a memory or the like may be reproduced and displayed as a moving image on the screen 35 (for example, the first display area 35A).

[0057] The second display area 35B is displayed in the lower right corner as viewed from the front within the screen 35. The display position of the second display area 35B may be anywhere within the screen 35 of the display device 18, but it is preferable that the second display area 35B be displayed in a position that can be compared with the medical image 29 displayed in the first display area 35A.

[0058] In the second display area 35B, auxiliary information 38 is displayed. The auxiliary information 38 is information that assists the doctor 12 in making medical decisions during an endoscopic examination, and is referred to by the doctor 12. Examples of the auxiliary information 38 include various types of information related to the subject 26 into whose body the endoscope main body 16 is inserted, and / or various types of information obtained by performing medical support processing, which will be described later. In the example shown in FIG. 1, information including a medical image 29 obtained in the past (for example, an ultrasound image 29B obtained in the past) is shown as an example of the auxiliary information 38.

[0059] FIG. 2 is a conceptual diagram showing an example of the overall configuration of the endoscope device 10. As shown in FIG. 2, the endoscope body 16 includes an operation section 40 and an insertion section 42. The insertion section 42 is formed in a tubular shape. The insertion section 42 has a tip section 44, a bending section 46, and a soft section 48. The tip section 44, the bending section 46, and the soft section 48 are arranged in this order from the tip side to the base end side of the insertion section 42. The soft section 48 is formed of a long, flexible material, and connects the operation section 40 and the bending section 46. The bending section 46 is partially bent or rotates around the axis of the insertion section 42 when the operation section 40 is operated. In this way, the insertion section 42 is sent to the back side of the upper digestive tract 28 while bending according to the shape of the tubular organ (for example, the shape of the duct of the upper digestive tract 28) or rotating around the axis of the insertion section 42.

[0060] The tip portion 44 is provided with an ultrasonic probe 50 and a treatment tool opening 52. The ultrasonic probe 50 is provided on the tip side of the tip portion 44. The ultrasonic probe 50 is a convex ultrasonic probe that emits ultrasonic waves and receives reflected waves obtained when the emitted ultrasonic waves are reflected by a target region (for example, an organ such as the pancreas). Here, a convex ultrasonic probe is given as an example of the ultrasonic probe 50, but this is merely an example, and for example, a radial ultrasonic probe may also be used.

[0061] The treatment tool opening 52 is formed closer to the base end of the tip portion 44 than the ultrasonic probe 50. The treatment tool opening 52 is an opening for allowing the treatment tool 54 to protrude from the tip portion 44. A treatment tool insertion port 56 is formed in the operation section 40, and the treatment tool 54 is inserted into the insertion section 42 from the treatment tool insertion port 56. The treatment tool 54 passes through the insertion section 42 and protrudes from the treatment tool opening 52 to the outside of the endoscope body 16. The treatment tool opening 52 is also used as a suction port for sucking blood, internal waste, etc., and as a delivery port for delivering fluids.

[0062] 2, a puncture needle is shown as the treatment tool 54. Note that this is merely an example, and the treatment tool 54 may be a grasping forceps, a papillotomy knife, a snare, a catheter, a guide wire, a cannula, or a puncture needle with a guide sheath, etc.

[0063] In the example shown in Fig. 2, an illumination device 58 and a camera 60 are provided at the tip portion 44. The illumination device 58 irradiates light 30 (see Fig. 1). Types of light 30 irradiated from the illumination device 58 include, for example, white light and special light. Examples of the special light include light for BLI and / or light for LCI.

[0064] The camera 60 is mounted on the endoscope body 16. The camera 60 is inserted into the upper digestive tract 28 of the subject 26, and generates an optical image 29A by capturing an image of an observation target area in the upper digestive tract 28 while being irradiated with light 30 by the illumination device 58. An example of the camera 60 is a CMOS camera. The CMOS camera is merely an example, and other types of cameras such as a CCD camera may be used. Note that the image captured by the camera 60 may be displayed on the display device 18, displayed on a display device other than the display device 18 (for example, a display of a tablet terminal), stored in a storage medium (for example, a flash memory, a HDD, and / or a magnetic tape, etc.), or transmitted to the server 2.

[0065] The endoscope body 16 is connected to the light source device 20 and the control device 22 via a universal cord 62. The control device 22 is connected to a function expansion device 24 and a reception device 64. The function expansion device 24 is also connected to the display device 18 in addition to the server 2. That is, the control device 22 is connected to the server 2 and the display device 18 via the function expansion device 24.

[0066] Here, the function expansion device 24 is exemplified as an external device for expanding the functions performed by the control device 22, and therefore an example is given in which the control device 22 and the display device 18 are indirectly connected via the function expansion device 24, but this is merely one example. For example, the display device 18 may be directly connected to the control device 22. In this case, it is sufficient that the functions of the function expansion device 24 are installed in the control device 22, for example.

[0067] The reception device 64 receives instructions from the doctor 12 and outputs the received instructions as an electrical signal to the control device 22. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0068] The control device 22 controls the light source device 20, exchanges various signals with the endoscope body 16, and exchanges various signals with the function expansion device 24 in accordance with instructions received by the reception device 64.

[0069] The light source device 20 emits light under the control of the control device 22, and supplies the light 30 to the illumination device 58. The illumination device 58 has a built-in light guide, and the light 30 supplied from the light source device 20 is irradiated from the tip 44 via the light guide. The control device 22 causes the camera 60 to capture an image according to an instruction received by the reception device 64, acquires an optical image 29A (see FIG. 1) from the camera 60, and outputs it to a predetermined output destination (for example, the function expansion device 24). The control device 22 also causes the ultrasonic probe 50 to emit ultrasonic waves according to an instruction received by the reception device 64, generates an ultrasonic image 29B (see FIG. 1) based on the reflected wave received by the ultrasonic probe 50, and outputs it to a predetermined output destination.

[0070] The function expansion device 24 is a device that expands the functions of the endoscope device 10. The function expansion device 24 supports medical procedures (here, as an example, endoscopic examination) by executing processing using medical images 29 input from the control device 22. The function expansion device 24 transmits various information including the medical images 29 to the server 2 via the network 3, or outputs the information to the display device 18. The function expansion device 24 also receives information transmitted from the server 2, and outputs the received information to the display device 18 or stores the information in a storage medium.

[0071] Although the embodiment in which the medical image 29 output from the control device 22 is output to the display device 18 via the function expansion device 24 has been described above, this is merely one example. For example, the control device 22 and the display device 18 may be connected, and the medical image 29 acquired by the function expansion device 24 may be displayed on the display device 18 via the control device 22.

[0072] Fig. 3 is a block diagram showing an example of a hardware configuration of an electrical system of the endoscope device 10. As shown in Fig. 3, the control device 22 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire control device 22. The memory 74 and the storage 76 are used by the processor 72.

[0073] The external I / F 70 is responsible for transmitting and receiving various types of information between the processor 72 and one or more devices (hereinafter, also referred to as “first external devices”) that exist outside the control device 22.

[0074] A camera 60 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 is responsible for transmitting and receiving various types of information between the camera 60 and the processor 72. The processor 72 controls the camera 60 via the external I / F 70. The processor 72 also acquires, via the external I / F 70, an optical image 29A (see FIG. 1) obtained by the camera 60 capturing an image of the inside of the upper gastrointestinal tract 28 (see FIG. 1).

[0075] The light source device 20 is connected to the external I / F 70 as one of the first external devices, and the external I / F 70 controls the exchange of various information between the light source device 20 and the processor 72. The light source device 20 supplies light to the illumination device 58 under the control of the processor 72. The illumination device 58 irradiates the light supplied from the light source device 20.

[0076] A reception device 64 is connected to the external I / F 70 as one of the first external devices, and the processor 72 acquires instructions accepted by the reception device 64 via the external I / F 70 and executes processing according to the acquired instructions.

[0077] A transmission / reception circuit 78 is connected to the external I / F 70. The transmission / reception circuit 78 generates an ultrasonic emission signal 80 of a pulse waveform according to an instruction from the processor 72 and outputs it to the ultrasonic probe 50. The ultrasonic probe 50 converts the ultrasonic emission signal 80 input from the transmission / reception circuit 78 into ultrasonic waves and emits the ultrasonic waves to an emission target region 81 (for example, an organ such as the prostate). The ultrasonic probe 50 receives a reflected wave obtained by reflecting the ultrasonic waves emitted from the ultrasonic probe 50 at the emission target region 81, converts the reflected wave into a reflected wave signal 82 which is an electric signal, and outputs it to the transmission / reception circuit 78. The transmission / reception circuit 78 digitizes the reflected wave signal 82 input from the ultrasonic probe 50, and outputs the digitized reflected wave signal 82 to the processor 72 via the external I / F 70. The processor 72 generates an ultrasonic image 29B (see FIG. 1) showing the state of the emission target region 81 based on the reflected wave signal 82 input from the transmission / reception circuit 78 via the external I / F 70.

[0078] The function expansion device 24 includes a computer 84 and an external I / F 86. The computer 84 includes a processor 88, a memory 90, and a storage 92. The processor 88, the memory 90, the storage 92, and the external I / F 86 are connected to a bus 94. Note that the hardware configuration of the computer 84 (i.e., the processor 88, the memory 90, and the storage 92) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 84 will be omitted here.

[0079] The external I / F 86 controls the exchange of various information between the processor 88 and one or more devices (hereinafter, also referred to as “second external devices”) existing outside the function expansion device 24.

[0080] The control device 22 is connected to the external I / F 86 as one of the second external devices. In the example shown in Fig. 3, the external I / F 70 of the control device 22 is connected to the external I / F 86. The external I / F 86 controls the exchange of various information between the processor 88 of the function extension device 24 and the processor 72 of the control device 22. For example, the processor 88 acquires a medical image 29 (see Fig. 1) from the processor 72 of the control device 22 via the external I / Fs 70 and 86. Then, the processor 88 executes processing using the medical image 29.

[0081] The display device 18 is connected to the external I / F 86 as one of the second external devices. The processor 88 controls the display device 18 via the external I / F 86 to cause the display device 18 to display various information (e.g., medical image 29, etc.). For example, the processor 88 displays the medical image 29 on the screen 35 of the display device 18 (e.g., first display area 35A).

[0082] The external I / F 86 is connected to the server 2 via the network 3 as one of the second external devices. The processor 88 exchanges various information with the server 2 via the external I / F 86. For example, the external I / F 86 transmits a medical image 29 acquired by the processor 88 to the server 2. The server 2 receives the medical image 29 transmitted from the external I / F 86, executes processing using the received medical image 29, and transmits the processing result to the function extension device 24. The external I / F 86 receives the processing result transmitted from the server 2.

[0083] FIG. 4 is a block diagram showing an example of the hardware configuration of the electrical system of the server 2. As shown in FIG. 4, the server 2 includes a computer 96 and an external I / F 98. The computer 96 includes a processor 100, a memory 102, and a storage 104. The processor 100, the memory 102, the storage 104, and the external I / F 98 are connected to a bus 106. In this embodiment, the computer 96 is an example of a "computer" according to the present disclosure, and the processor 100 is an example of a "processor" according to the present disclosure. Note that the hardware configuration of the computer 96 (i.e., the processor 100, the memory 102, and the storage 104) is basically the same as the hardware configuration of the computer 66, so a description of the hardware configuration of the computer 96 will be omitted here.

[0084] The external I / F 98 is responsible for transmitting various types of information between the processor 100 and one or more devices (hereinafter also referred to as "third external devices") existing outside the server 2. The function expansion device 24 is connected to the external I / F 98 via the network 3 as one of the third external devices. The processor 100 transmits and receives various types of information to and from the function expansion device 24 via the external I / F 98. For example, the external I / F 98 receives various types of information such as medical images 29 transmitted from the function expansion device 24. The processor 100 executes processing using the information received by the external I / F 98, and transmits the processing results to the function expansion device 24.

[0085] In recent years, there has been progress in the development of a technology in which a trained model optimized by performing machine learning on a model (e.g., a neural network) recognizes a lesion 36 by performing object recognition processing on a medical image 29, and the recognition result of the object recognition processing or information based on the recognition result of the object recognition processing is displayed on a screen 35.

[0086] For example, the trained model is installed in the server 2. The endoscope device 10 requests the server 2 to execute object recognition processing, and the server 2 transmits the execution result of the object recognition processing to the endoscope device 10. Then, the endoscope device 10 displays the execution result on the screen 35.

[0087] Here, consider a case where the processing contents of the object recognition processing are switched in the server 2. The switched contents when the processing contents of the object recognition processing are switched in the server 2 are information that can be useful for the doctor 12 in performing accurate medical treatment. However, for the doctor 12 who is performing an endoscopic examination using the endoscope device 10, it is difficult to understand the switched contents when the processing contents of the object recognition processing are switched in the server 2 only from the execution result displayed on the screen 35. This is not limited to endoscopic examinations, but also applies to other types of medical examinations (for example, radiographic examinations, MRI examinations, CT examinations, etc.).

[0088] In view of such circumstances, in this embodiment, as an example shown in FIG. 4, a medical support process is performed by the processor 100 of the server 2. A medical support program 108, a first recognition model 110A, and a second recognition model 110B are stored in the storage 104. The medical support program 108 is an example of a "program" according to the present disclosure. The processor 100 reads the medical support program 108 from the storage 104 and executes the read medical support program 108 on the memory 102 to perform the medical support process. The medical support process is realized by the processor 100 operating as a control unit 100A and a recognition unit 100B according to the medical support program 108 executed on the memory 102. The first recognition model 110A and the second recognition model 110B are used by the recognition unit 100B, as will be described in detail later.

[0089] Fig. 5 is a conceptual diagram showing an example of the processing contents performed by the server 2 and the endoscope device 10. As shown in Fig. 5, the server 2 communicates with the endoscope device 10. The endoscope device 10 transmits medical information 112 to the server 2. In the server 2, the control unit 100A receives the medical information 112 via the external I / F 98 (see Fig. 4). The medical information 112 includes a medical image 29 and modality-related information 114.

[0090] The medical image 29 is an image obtained by imaging the inside of the body of the subject 26 by the endoscope device 10. In the example shown in Fig. 5, an optical image 29A or an ultrasound image 29B is shown as an example of the medical image 29 transmitted from the endoscope device 10 to the server 2.

[0091] The modality-related information 114 is information about the endoscope device 10 at the timing when the medical image 29 is obtained. A first example of the timing when the medical image 29 is obtained is a period from the start of imaging a predetermined number of frames to the end of imaging. A second example of the timing when the medical image 29 is obtained is the start of imaging a predetermined number of frames. A third example of the timing when the medical image 29 is obtained is the end of imaging a predetermined number of frames (for example, the time when the optical image 29A is generated or the time when the ultrasound image 29B is generated).

[0092] The modality-related information 114 includes modality-specific information 114A. The modality-specific information 114A is information capable of identifying the type of upper endoscope device currently used in the endoscope device 10. In other words, the modality-specific information 114A can also be said to be information (e.g., an identifier) ​​capable of identifying whether an optical upper endoscope device or an ultrasonic upper endoscope device is currently used. The endoscope body 16 is equipped with a computer (not shown), and the modality-specific information 114A is held by the computer of the endoscope body 16. The content of the modality-specific information 114A is changed in accordance with a change in the type of the upper endoscope device used in the endoscope device 10. The processor 72 of the control device 22 acquires the modality-specific information 114A from the endoscope body 16 at the timing when the content of the modality-specific information 114A is changed. The processor 72 transmits the modality-specific information 114A acquired from the endoscope body 16 to the server 2 via the function extension device 24.

[0093] In the server 2, the control unit 100A outputs the medical image 29 to the recognition unit 100B. The recognition unit 100B recognizes characteristic areas (e.g., lesions, organs, etc.) in the medical image 29 based on the medical image 29 input from the control unit 100A. To achieve this, the recognition unit 100B executes a first recognition process 116 and a second recognition process 118. The first recognition process 116 and the second recognition process 118 are examples of the "first process" and "AI process" according to the present disclosure.

[0094] In the first recognition process 116, based on the optical image 29A, the geometric characteristics (e.g., position, size, shape, etc.) of the lesion 36 captured in the optical image 29A, the type of the lesion 36, and the type of the lesion 36 (e.g., mass type, localized ulcer type, ulcer infiltration type, diffuse infiltration type, etc.) are recognized. In addition, in the first recognition process 116, based on the optical image 29A, a portion captured in the optical image 29A (hereinafter simply referred to as a "first portion") is recognized. Here, the first portion refers to a portion of the upper digestive tract 28. Examples of the first portion include the esophagus, cardia, fundus, upper body, middle body, lower body, angle of the stomach, vestibule, pylorus, duodenal bulb, and duodenum.

[0095] The first recognition process 116 is performed by the recognition unit 100B on the acquired optical image 29A every time the optical image 29A is acquired. The first recognition process 116 is a process for recognizing the lesion 36 and the first region by a method using AI. Here, the first recognition process 116 is a process using the first recognition model 110A.

[0096] The first recognition model 110A is a trained model for object recognition by AI using a bounding box method. The first recognition model 110A is optimized by performing machine learning on a neural network using first teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which the first example data and the first correct answer data are associated with each other. In other words, the first recognition model 110A is a trained model optimized so that the first correct answer data is generated by inputting the first example data.

[0097] The first example data is an image corresponding to optical image 29A (in other words, a sample image simulating optical image 29A). The image corresponding to optical image 29A is an example of the "first information" and "first image" according to the present disclosure. An example of an image corresponding to optical image 29A is an image obtained by actually capturing an image of the upper digestive tract with a camera. A second example of an image corresponding to optical image 29A is a virtually created image (for example, an image generated by a generation AI).

[0098] The first correct answer data is information corresponding to information related to optical image 29A (e.g., information capable of identifying lesion 36 and the first site shown in optical image 29A). The information corresponding to the information related to optical image 29A is an example of the "second information" and the "second image" according to the present disclosure.

[0099] In this embodiment, the first correct answer data refers to correct answer data (i.e., annotations) for the first example data. Here, annotations that identify the geometric characteristics, the type, the form, and the first site of a lesion in an image used as the first example data are used as an example of the first correct answer data.

[0100] Recognition unit 100B acquires optical image 29A from control unit 100A and inputs acquired optical image 29A to first recognition model 110A. As a result, first recognition model 110A recognizes lesion 36 and first region shown in input optical image 29A every time optical image 29A is input, and generates and outputs first recognition result 122 which is the recognition result. In this embodiment, first recognition result 122 is an example of the "execution result," "related information," and "characteristic region identifying information" according to the present disclosure.

[0101] Meanwhile, in the second recognition process 118, the geometric characteristics (e.g., position, size, shape, etc.) of the lesion 120 shown in the ultrasound image 29B, the type of the lesion 120, the model of the lesion 120, etc. are recognized based on the ultrasound image 29B. Also, in the second recognition process 118, a part shown in the ultrasound image 29B (hereinafter simply referred to as a "second part") is recognized based on the ultrasound image 29B. Here, the second part refers to a part recognized by the ultrasound type upper endoscope device. Examples of the second part include organs such as the pancreas, bile duct, gallbladder, and liver.

[0102] The second recognition process 118 is performed by the recognition unit 100B on the acquired ultrasound image 29B every time the ultrasound image 29B is acquired. The second recognition process 118 is a process for recognizing the lesion 120 and the second part by a method using AI. Here, as the second recognition process 118, a process using the second recognition model 110B is performed.

[0103] The second recognition model 110B is a trained model for object recognition by AI using a bounding box method. The second recognition model 110B is optimized by performing machine learning on the neural network using second teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which the second example data and the second correct answer data are associated with each other. In other words, the second recognition model 110B is a trained model optimized so that the second correct answer data is generated by inputting the second example data.

[0104] The second example data is an image corresponding to ultrasound image 29B (in other words, a sample image assuming ultrasound image 29B). The image corresponding to ultrasound image 29B is an example of the "first information" and "first image" according to the present disclosure. An example of an image corresponding to ultrasound image 29B is an image generated based on reflected waves obtained by actually emitting ultrasound in the upper digestive tract by an ultrasound-type upper endoscope device. A second example of an image corresponding to ultrasound image 29B is a virtually created image (for example, an image generated by a generation AI).

[0105] The second answer data is information corresponding to information related to ultrasound image 29B (e.g., information capable of identifying lesion 120 and the second part shown in ultrasound image 29B). The information corresponding to the information related to ultrasound image 29B is an example of the "second information" and "second image" according to the present disclosure.

[0106] In this embodiment, the second supervised answer data refers to supervised answer data (i.e., annotations) for the second example data. Here, annotations that identify the geometric characteristics, the type of the lesion, the lesion model, and the second site of the lesion appearing in the image used as the second example data are used as an example of the second supervised answer data.

[0107] The recognition unit 100B acquires an ultrasound image 29B from the control unit 100A, and inputs the acquired ultrasound image 29B to the second recognition model 110B. As a result, the second recognition model 110B recognizes the lesion 120 and the second part shown in the input ultrasound image 29B every time the ultrasound image 29B is input, and generates and outputs a second recognition result 124 that is the recognition result. In this embodiment, the second recognition result 124 is an example of the "execution result," "related information," and "characteristic region identifying information" according to the present disclosure.

[0108] The control unit 100A switches the processing contents in the recognition unit 100B in response to the modality related information 114. In the example shown in Fig. 5, the control unit 100A switches the processing contents in the recognition unit 100B in response to the modality specific information 114A. That is, the control unit 100A selectively executes the first recognition process 116 and the second recognition process 118 in response to the modality specific information 114A.

[0109] When the control unit 100A refers to the modality identification information 114A and identifies that the type of upper endoscope device currently being used in the endoscope device 10 is an optical upper endoscope device, the control unit 100A outputs the optical image 29A to the recognition unit 100B and causes the recognition unit 100B to execute a first recognition process 116. That is, the recognition unit 100B inputs the optical image 29A to a first recognition model 110A.

[0110] When the control unit 100A refers to the modality identification information 114A and identifies that the type of upper endoscope device currently being used in the endoscope device 10 is an ultrasonic upper endoscope device, the control unit 100A outputs the ultrasound image 29B to the recognition unit 100B and causes the recognition unit 100B to execute the second recognition process 118. That is, the recognition unit 100B inputs the ultrasound image 29B to the second recognition model 110B.

[0111] In this way, the server 2 has the first recognition model 110A and the second recognition model 110B, and in the processing of the recognition unit 100B, the learned model as the input destination of the medical image 29 is switched according to the modality identification information 114A. That is, the control unit 100A causes the recognition unit 100B to selectively use the first recognition model 110A and the second recognition model 110B according to the modality identification information 114A.

[0112] Fig. 6 is a conceptual diagram showing an example of the processing contents of the server 2 and the endoscope device 10. As shown in Fig. 6, when the first recognition process 116 is executed by the recognition unit 100B, the recognition unit 100B outputs a first recognition result 122 to the control unit 100A. When the second recognition process 118 is executed by the recognition unit 100B, the recognition unit 100B outputs a second recognition result 124 to the control unit 100A. When the first recognition process 116 is switched to the second recognition process 118 by the recognition unit 100B, or when the second recognition process 118 is switched to the first recognition process 116 by the recognition unit 100B, the recognition unit 100B generates switching content information 126 and outputs the generated switching content information 126 to the control unit 100A.

[0113] The switching content information 126 is information indicating switching content when the processing content in the recognition unit 100B is switched (i.e., the content to which the processing content in the recognition unit 100B has been switched). An example of the switching content information 126 when the processing executed by the recognition unit 100B is switched from the second recognition processing 118 to the first recognition processing 116 is information capable of identifying an event that the second recognition processing 118 is switched to the first recognition processing 116 by the recognition unit 100B. Also, an example of the switching content information 126 when the processing executed by the recognition unit 100B is switched from the first recognition processing 116 to the second recognition processing 118 is information capable of identifying an event that the processing executed by the recognition unit 100B is switched from the first recognition processing 116 to the second recognition processing 118.

[0114] The switching content information 126 includes model switching content information 126A. The model switching content information 126A is information indicating the switching content of the learned model when the learned model as the input destination of the medical image 29 is switched. An example of the model switching content information 126A when the learned model used by the recognition unit 100B is switched from the second recognition model 110B to the first recognition model 110A is information capable of identifying an event that the learned model to which the medical image 29 is input is switched from the second recognition model 110B (i.e., a learned model for an ultrasonic upper endoscope device) to the first recognition model 110A (i.e., a learned model for an optical upper endoscope device). In addition, an example of model switching content information 126A when the trained model used by the recognition unit 100B is switched from the first recognition model 110A to the second recognition model 110B is information capable of identifying an event in which the trained model into which the medical image 29 is input has been switched from the first recognition model 110A (i.e., a trained model for an optical upper endoscopy device) to the second recognition model 110B (i.e., a trained model for an ultrasonic upper endoscopy device).

[0115] In this embodiment, the switching content information 126 is an example of the “switching content information” according to the present disclosure, and the model switching content information 126A is an example of the “AI processing switching content information” and “model switching content information” according to the present disclosure.

[0116] The control unit 100A transmits the first recognition result 122, the second recognition result 124, and the switching content information 126 input from the recognition unit 100B to the endoscope device 10 via the external I / F 98 (see FIG. 4). In the endoscope device 10, the processor 88 receives the first recognition result 122, the second recognition result 124, and the switching content information 126 via the external I / F 86.

[0117] When processor 88 receives first recognition result 122 via external I / F 86, processor 88 generates first region name information 128, first lesion identification information 130, first discrimination information 132, and first size information 134 as information based on first recognition result 122, and displays them in first display area 35A. First region name information 128, first lesion identification information 130, first discrimination information 132, and first size information 134 are displayed at positions different from the position at which optical image 29A is displayed.

[0118] The first part name information 128 is information indicating the name of the first part recognized by performing the first recognition process 116. The first lesion identification information 130 is information capable of identifying the lesion 36 recognized by performing the first recognition process 116. The first discrimination information 132 is information indicating the result of discrimination of the lesion 36 (e.g., the degree of malignancy). The first size information 134 is information indicating the size of the lesion 36 recognized by performing the first recognition process 116. Here, the first discrimination information 132 and the first size information 134 are exemplified, but information other than the first discrimination information 132 and the first size information 134 (e.g., a feature map, etc.) may be displayed on the screen 35 as long as it indicates the characteristics of the lesion 36.

[0119] When the optical image 29A is displayed in the first display area 35A (i.e., when the optical image 29A used in the first recognition process 116 is displayed in the first display area 35A), the processor 88 superimposes and displays a bounding box BB1 generated according to the geometric characteristics of the lesion 36 recognized by performing the first recognition process 116 on the optical image 29A (e.g., the optical image 29A used in the first recognition process 116) displayed in the first display area 35A. In the example shown in FIG. 6, the bounding box BB1 is superimposed and displayed at the position where the lesion 36 is captured in the optical image 29A displayed in the first display area 35A. The display of the bounding box BB1 is updated in synchronization with the display timing of the optical image 29A newly displayed in the first display area 35A (i.e., the timing when the optical image 29A displayed in the first display area 35A is updated).

[0120] Note that although a bounding box BB1 is given here, this is merely one example, and the processor 88 may display an identifier in place of the bounding box BB1 (for example, a mark defined by the four corners of a rectangular frame) on the screen 35 (in the example shown in FIG. 6, the first display area 35A within the screen 35).

[0121] When the trained model used by the recognition unit 100B is switched from the second recognition model 110B to the first recognition model 110A, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 136 is displayed as one piece of auxiliary information 38. The message 136 is a message indicating an event that the trained model for an ultrasonic upper endoscope device has been switched to the trained model for an optical upper endoscope device.

[0122] When processor 88 receives second recognition result 124 via external I / F 86, processor 88 generates second region name information 138, second lesion identification information 140, second differentiation information 142, and second size information 144 as information based on second recognition result 124, and displays them in first display region 35A. Second region name information 138, second lesion identification information 140, second differentiation information 142, and second size information 144 are displayed at positions different from the position at which ultrasound image 29B is displayed.

[0123] The second part name information 138 is information indicating the name of the second part recognized by performing the second recognition process 118. The second lesion identification information 140 is information capable of identifying the lesion 120 recognized by performing the second recognition process 118. The second discrimination information 142 is information indicating the result of discriminating the lesion 120 (e.g., the degree of malignancy). The second size information 144 is information indicating the size of the lesion 120 recognized by performing the second recognition process 118. Here, the second discrimination information 142 and the second size information 144 are exemplified, but information other than the second discrimination information 142 and the second size information 144 (e.g., a feature map, etc.) may be displayed on the screen 35 as long as it indicates the characteristics of the lesion 120.

[0124] When the ultrasound image 29B is displayed in the first display region 35A (i.e., when the ultrasound image 29B used in the second recognition process 118 is displayed in the first display region 35A), the processor 88 superimposes and displays a bounding box BB2 generated according to the geometric characteristics of the lesion 120 recognized by performing the second recognition process 118 on the ultrasound image 29B (e.g., the ultrasound image 29B used in the second recognition process 118) displayed in the first display region 35A. In the example shown in FIG. 6, the bounding box BB2 is superimposed and displayed at the position where the lesion 120 is captured in the ultrasound image 29B displayed in the first display region 35A. The display of the bounding box BB2 is updated in synchronization with the display timing of the ultrasound image 29B newly displayed in the first display region 35A (i.e., the timing at which the ultrasound image 29B displayed in the first display region 35A is updated).

[0125] Note that although a bounding box BB2 is given here, this is merely one example, and the processor 88 may display an identifier in place of the bounding box BB2 (for example, a mark defined by the four corners of a rectangular frame) on the screen 35 (in the example shown in FIG. 6, the first display area 35A within the screen 35).

[0126] When the trained model used by the recognition unit 100B is switched from the first recognition model 110A to the second recognition model 110B, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 146 is displayed as one of the auxiliary information 38. The message 146 is a message indicating an event that the trained model for an optical upper endoscope device has been switched to the trained model for an ultrasonic upper endoscope device.

[0127] 6 shows an example in which messages 136 and 146 are displayed in the second display area 35B, but this is merely one example. Any form of information may be used as long as the event of switching from a trained model for an ultrasonic upper endoscope device to a trained model for an optical upper endoscope device, or the event of switching from a trained model for an optical upper endoscope device to a trained model for an ultrasonic upper endoscope device, is perceptible by the doctor 12. A first example of information perceptible by the doctor 12 is visible information (e.g., a mark and / or a code) indicating the event of switching from a trained model for an ultrasonic upper endoscope device to a trained model for an optical upper endoscope device, or the event of switching from a trained model for an optical upper endoscope device to a trained model for an ultrasonic upper endoscope device. A second example of information perceivable by physician 12 includes an event of switching from a trained model for an ultrasound-based upper endoscopy device to a trained model for an optical upper endoscopy device, or audible information (e.g., voice) indicating an event of switching from a trained model for an optical upper endoscopy device to a trained model for an ultrasound-based upper endoscopy device.

[0128] 6 shows an example in which the message 136 or 146 is displayed on the screen 35, but the message 136 or 146, or information replacing the message 136 or 146, may be stored in the storage medium. The message 136 or 146, or information replacing the message 136 or 146, and at least a part of the medical information 112 (see FIG. 5) (for example, the medical image 29 and the modality-related information 114, which correspond to each other) may be stored in the storage medium in a corresponding state.

[0129] Next, the operation of the portion of the medical support system 1 according to the present disclosure will be described with reference to Fig. 7. The flow of the medical support process shown in Fig. 7 is an example of the "operation method of the medical support device" according to the present disclosure.

[0130] 7, first, in step ST10, the control unit 100A determines whether or not the medical image 29 and the modality related information 114 have been received by the external I / F 98. If the medical image 29 and the modality related information 114 have not been received by the external I / F 98 in step ST10, the determination is negative, and the determination in step ST10 is performed again. If the medical image 29 and the modality related information 114 have been received by the external I / F 98 in step ST10, the determination is positive, and the medical support process proceeds to step ST12.

[0131] In step ST12, the control unit 100A acquires the modality specific information 114A from the modality related information 114 received in step ST10 by the external I / F 98. After the process of step ST12 is executed, the medical support process proceeds to step ST14.

[0132] In step ST14, the control unit 100A causes the recognition unit 100B to use the first recognition model 110A or the second recognition model 110B according to the modality identification information 114A acquired in step ST12, thereby executing the first recognition process 116 or the second recognition process 118 for the medical image 29 received by the external I / F 98 in step ST10. That is, when the control unit 100A determines that an optical upper endoscope device is currently being used by referring to the modality identification information 114A, the control unit 100A causes the recognition unit 100B to input the optical image 29A to the first recognition model 110A, thereby outputting the first recognition result 122 from the first recognition model 110A. Furthermore, when the control unit 100A determines that an ultrasonic upper endoscope device is currently being used by referring to the modality identification information 114A, it causes the recognition unit 100B to input the ultrasonic image 29B to the second recognition model 110B, thereby causing the second recognition model 110B to output the second recognition result 124. After the process of step ST14 is executed, the medical support process proceeds to step ST16.

[0133] In step ST16, when the first recognition process 116 is executed by the recognition unit 100B, the control unit 100A acquires a first recognition result 122 from the recognition unit 100B. In addition, when the second recognition process 118 is executed by the recognition unit 100B, the control unit 100A acquires a second recognition result 124 from the recognition unit 100B. After the process of step ST16 is executed, the medical support process proceeds to step ST18.

[0134] In step ST18, the control unit 100A judges whether or not switching has been performed from one of the optical upper endoscope device and the ultrasonic upper endoscope device to the other by referring to the modality identification information 114A acquired in step ST12. If switching has not been performed from one of the optical upper endoscope device and the ultrasonic upper endoscope device to the other in step ST18, the judgment is negative, and the medical support process proceeds to step ST20.

[0135] In step ST20, the control unit 100A transmits the first recognition result 122 or the second recognition result 124 acquired in step ST16 to the endoscope device 10 via the external I / F 98. In the endoscope device 10, the processor 88 displays the medical image 29 used in the first recognition process 116 in the first display area 35A, and also displays information based on the first recognition result 122 or the second recognition result 124 transmitted from the control unit 100A via the external I / F 98 by executing the process of step ST20 in the first display area 35A. After the process of step ST20 is executed, the medical support process proceeds to step ST30.

[0136] In step ST18, if switching has been performed from one of the optical upper endoscope device and the ultrasonic upper endoscope device to the other, the determination is positive and the medical support process proceeds to step ST22.

[0137] When switching is performed from one of the optical upper endoscope device and the ultrasonic upper endoscope device to the other, switching content information 126 is generated by recognition unit 100B.

[0138] Therefore, in step ST22, the control unit 100A acquires the switching content information 126 generated by the recognition unit 100B from the recognition unit 100B. After the process of step ST22 is executed, the medical support process proceeds to step ST24.

[0139] In step ST24, the control unit 100A judges whether the switching from one of the optical upper endoscopic device and the ultrasonic upper endoscopic device to the other is a switching from an ultrasonic upper endoscopic device to an optical upper endoscopic device. In step ST24, if the switching from one of the optical upper endoscopic device and the ultrasonic upper endoscopic device to the other is a switching from an ultrasonic upper endoscopic device to an optical upper endoscopic device, the judgment is affirmative, and the medical support processing proceeds to step ST26. In step ST24, if the switching from one of the optical upper endoscopic device and the ultrasonic upper endoscopic device to the other is not a switching from an ultrasonic upper endoscopic device to an optical upper endoscopic device, the judgment is negative, and the medical support processing proceeds to step ST28.

[0140] In step ST26, the control unit 100A transmits the first recognition result 122 and the switching content information 126 to the endoscope device 10. The switching content information 126 transmitted to the endoscope device 10 by executing the process of this step ST26 is information capable of identifying an event that the recognition unit 100B has switched from the second recognition process 118 to the first recognition process 116. The switching content information 126 transmitted to the endoscope device 10 by executing the process of this step ST26 also includes model switching content information 126A. The model switching content information 126A included in the switching content information 126 in this step ST26 is information capable of identifying an event that the trained model to which the medical image 29 is input has been switched from the second recognition model 110B (i.e., a trained model for an ultrasonic upper endoscope device) to the first recognition model 110A (i.e., a trained model for an optical upper endoscope device). In the endoscope device 10, the processor 88 displays the optical image 29A used in the first recognition process 116 in the first display area 35A, and also displays information based on the first recognition result 122 transmitted from the control unit 100A via the external I / F 98 as a result of the execution of the process of step ST26 in the first display area 35A. In addition, the processor 88 displays a message 136 in which the switching content information 126 transmitted from the control unit 100A via the external I / F 98 as a result of the execution of the process of step ST26 is converted into text in the second display area 35B. After the process of step ST26 is executed, the medical support process proceeds to step ST30.

[0141] In step ST28, the control unit 100A transmits the second recognition result 124 and the switching content information 126 to the endoscope device 10. The switching content information 126 transmitted to the endoscope device 10 by executing the process of this step ST28 is information capable of identifying an event that the recognition unit 100B has switched from the first recognition process 116 to the second recognition process 118. The switching content information 126 transmitted to the endoscope device 10 by executing the process of this step ST28 also includes model switching content information 126A. The model switching content information 126A included in the switching content information 126 in this step ST28 is information capable of identifying an event that the trained model to which the medical image 29 is input has been switched from the first recognition model 110A (i.e., a trained model for an optical upper endoscope device) to the second recognition model 110B (i.e., a trained model for an ultrasonic upper endoscope device). In the endoscope device 10, the processor 88 displays the ultrasound image 29B used in the second recognition process 118 in the first display area 35A, and also displays information based on the second recognition result 124 transmitted from the control unit 100A via the external I / F 98 as a result of the execution of the process of step ST28 in the first display area 35A. The processor 88 also displays a message 146 in the second display area 35B, which is a text version of the switching content information 126 transmitted from the control unit 100A via the external I / F 98 as a result of the execution of the process of step ST28. After the process of step ST28 is executed, the medical support process proceeds to step ST30.

[0142] As described above, in the medical support system 1, the medical information 112 transmitted from the endoscope device 10 is received by the server 2. Then, in the server 2, the first recognition process 116 and the second recognition process 118 included in the medical information 112 are selectively performed. The first recognition process 116 is performed on the optical image 29A, and the second recognition process 118 is performed on the ultrasound image 29B. The first recognition result 122 obtained by performing the first recognition process 116 on the optical image 29A, and the second recognition result 124 obtained by performing the second recognition process 118 on the ultrasound image 29B are selectively transmitted from the server 2 to the endoscope device 10. The first recognition process 116 and the second recognition process 118 executed by the server 2 are switched according to the modality related information 114. Then, when one of the first recognition process 116 and the second recognition process 118 is switched to the other, the switching content information 126 is transmitted from the server 2 to the endoscope device 10.

[0143] In the endoscope device 10, information based on the first recognition result 122 transmitted from the server 2 or information based on the second recognition result 124 transmitted from the server 2 is displayed on a screen 35. In addition, in the endoscope device 10, a message 136 or 146 obtained by converting switching content information 126 transmitted from the server 2 into text is displayed on the screen 35. The switching content information 126 is information indicating switching content when the processing content in the recognition unit 100B is switched.

[0144] Therefore, the doctor 12 can grasp information based on the first recognition result 122 or information based on the second recognition result 124 through the screen 35. Furthermore, the doctor 12 can grasp the switching content when the processing content in the recognition unit 100B is switched through the screen 35. That is, the doctor 12 can grasp the switching content from one of the first recognition process 116 and the second recognition process 118 to the other through the screen 35. Furthermore, the doctor 12 can grasp the event that the trained model for an ultrasonic upper endoscope device has been switched to the trained model for an optical upper endoscope device, or the event that the trained model for an optical upper endoscope device has been switched to the trained model for an ultrasonic upper endoscope device, from the message 136 or 146 in which the model switching content information 126A included in the switching content information 126 is transcribed. Furthermore, the doctor 12 can understand from the message 136 or 146 displayed on the screen 35 whether the trained model currently being used by the recognition unit 100B is the first recognition model 110A or the second recognition model 110B.

[0145] Furthermore, in the medical support system 1, multiple types of trained models (i.e., the first recognition model 110A and the second recognition model 110B) as input destinations of the medical image 29 are switched according to the modality related information 114 (here, as an example, the modality specific information 114A). This makes it possible to input the medical image 29 to an appropriate trained model, compared to a case in which multiple types of trained models as input destinations of the medical image 29 are switched regardless of the modality related information 114 (here, as an example, the modality specific information 114A).

[0146] In the above embodiment, the modality related information 114 includes the modality specific information 114A, but this is merely an example. For example, as shown in FIG. 8, the modality related information 114 may include the light source specific information 114B.

[0147] The light source identification information 114B is information capable of identifying a light source used in imaging to obtain the optical image 29A. The light sources used in imaging to obtain the optical image 29A are classified into a first light source that emits white light and a second light source that emits special light. For convenience of explanation, a light source that emits light for LCI is used as the second light source in the description, but this is merely an example, and the present disclosure is valid even if a light source that emits light for BLI is used as the second light source.

[0148] The light source identification information 114B is held by a computer (not shown) of the endoscope body 16. The content of the light source identification information 114B is changed by changing one of the first light source and the second light source to the other. The processor 72 of the control device 22 acquires the light source identification information 114B from the endoscope body 16 at the timing when the content of the light source identification information 114B is changed. The processor 72 transmits the light source identification information 114B acquired from the endoscope body 16 to the server 2 via the function expansion device 24.

[0149] In the server 2, the control unit 100A outputs the optical image 29A to the recognition unit 100B. The recognition unit 100B recognizes characteristic regions (e.g., lesions, organs, etc.) in the optical image 29A based on the optical image 29A input from the control unit 100A. To achieve this, the recognition unit 100B executes a third recognition process 148 and a fourth recognition process 150. The third recognition process 148 and the fourth recognition process 150 are examples of the "first process" and "AI process" according to the present disclosure.

[0150] In the third recognition process 148, based on an image (hereinafter referred to as a "white light optical image") obtained as optical image 29A by capturing an image of the inside of upper gastrointestinal tract 28 by camera 60 while white light is irradiated inside upper gastrointestinal tract 28 by lighting device 58, the geometric characteristics of lesion 36, the type of lesion 36, the form of lesion 36, etc., shown in the white light optical image are recognized. Also, in the third recognition process 148, the first site is recognized based on the white light optical image.

[0151] The third recognition process 148 is performed by the recognition unit 100B on the acquired white light optical image every time the white light optical image is acquired. The third recognition process 148 is a process for recognizing the lesion 36 and the first region by a method using AI. Here, the third recognition process 148 is a process using the third recognition model 110C.

[0152] The third recognition model 110C is a trained model for object recognition by AI using a bounding box method. The third recognition model 110C is optimized by performing machine learning on the neural network using third teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which the third example data and the third correct answer data are associated with each other. In other words, the third recognition model 110C is a trained model optimized so that the third correct answer data is generated by inputting the third example data.

[0153] The third example data is an image corresponding to a white light optical image (in other words, an image that is assumed to be a white light optical image). The image corresponding to a white light optical image is an example of the "first information" and "first image" according to the present disclosure.

[0154] The third correct answer data is information corresponding to information related to the white light optical image (e.g., information capable of identifying the lesion 36 and the first part shown in the white light optical image). The information corresponding to the information related to the white light optical image is an example of the "second information" and the "second image" according to the present disclosure.

[0155] The third correct answer data refers to correct answer data (i.e., annotations) for the third example data. Here, annotations that identify the geometric characteristics, the type of the lesion, the lesion model, and the first site of the lesion in the image used as the third example data are used as an example of the third correct answer data.

[0156] The recognition unit 100B acquires a white light optical image from the control unit 100A, and inputs the white light optical image acquired from the control unit 100A to the third recognition model 110C. As a result, every time a white light optical image is input, the third recognition model 110C recognizes the lesion 36 and the first region shown in the input white light optical image, and generates and outputs the recognition result as a third recognition result 152. The third recognition result 152 is an example of the "execution result," "related information," and "characteristic region identifying information" according to the present disclosure.

[0157] Meanwhile, in the fourth recognition process 150, the geometric characteristics, type, and model of the lesion 36 shown in the special light optical image are recognized based on an image (hereinafter referred to as the "special light optical image") obtained as optical image 29A by capturing an image of the inside of the upper digestive tract 28 by the camera 60 while special light is irradiated by the lighting device 58 inside the upper digestive tract 28. Also, in the fourth recognition process 150, the first site is recognized based on the special light optical image.

[0158] The fourth recognition process 150 is performed on the acquired special light optical image by the recognition unit 100B every time the special light optical image is acquired. The fourth recognition process 150 is a process for recognizing the lesion 36 and the first region by a method using AI. Here, as the fourth recognition process 150, a process using the fourth recognition model 110D is performed.

[0159] The fourth recognition model 110D is a trained model for object recognition by AI using a bounding box method. The fourth recognition model 110D is optimized by performing machine learning on the neural network using fourth teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which the fourth example data and the fourth correct answer data are associated with each other. In other words, the fourth recognition model 110D is a trained model optimized so that the fourth correct answer data is generated by inputting the fourth example data.

[0160] The fourth example data is an image corresponding to the special light optical image (in other words, an image that is assumed to be a special light optical image). The image corresponding to the special light optical image is an example of the "first information" and "first image" according to the present disclosure.

[0161] The fourth correct answer data is information corresponding to information related to the special light optical image (e.g., information capable of identifying the lesion 36 and the first part shown in the special light optical image). The information corresponding to the information related to the special light optical image is an example of the "second information" and "second image" according to the present disclosure.

[0162] The fourth correct answer data refers to correct answer data (i.e., annotations) for the fourth example data. Here, as an example of the fourth correct answer data, annotations that identify the geometric characteristics, the type of the lesion, the lesion model, and the first site of the lesion appearing in the image used as the fourth example data are used.

[0163] Recognition unit 100B acquires a special light optical image from control unit 100A, and inputs the special light optical image acquired from control unit 100A to fourth recognition model 110D. As a result, every time a special light optical image is input, fourth recognition model 110D recognizes lesion 36 and first region appearing in the input special light optical image, and generates and outputs fourth recognition result 154, which is the recognition result. Fourth recognition result 154 is an example of the "execution result," "related information," and "characteristic region identification information" according to the present disclosure.

[0164] The control unit 100A switches the processing content in the recognition unit 100B according to the light source identification information 114B. That is, the control unit 100A selectively executes the third recognition process 148 and the fourth recognition process 150 according to the light source identification information 114B.

[0165] When the control unit 100A refers to the light source identification information 114B and identifies that the light source currently being used for imaging in the upper gastrointestinal tract 28 is the first light source, the control unit 100A outputs the white light optical image to the recognition unit 100B and causes the recognition unit 100B to execute the third recognition process 148. That is, the recognition unit 100B inputs the white light optical image to the third recognition model 110C.

[0166] When the control unit 100A refers to the light source identification information 114B and identifies that the light source currently being used for imaging in the upper digestive tract 28 is the second light source, it outputs the special light optical image to the recognition unit 100B and causes the recognition unit 100B to execute the fourth recognition process 150. That is, the recognition unit 100B inputs the special light optical image to the fourth recognition model 110D.

[0167] In this way, the server 2 has the third recognition model 110C and the fourth recognition model 110D, and in the processing of the recognition unit 100B, the learned model as the input destination of the optical image 29A is switched according to the light source identification information 114B. That is, the control unit 100A causes the recognition unit 100B to selectively use the third recognition model 110C and the fourth recognition model 110D according to the light source identification information 114B.

[0168] 9, when the third recognition process 148 is executed by the recognition unit 100B, the recognition unit 100B outputs a third recognition result 152 to the control unit 100A. When the fourth recognition process 150 is executed by the recognition unit 100B, the recognition unit 100B outputs a fourth recognition result 154 to the control unit 100A. When the third recognition process 148 is switched to the fourth recognition process 150 by the recognition unit 100B, or when the fourth recognition process 150 is switched to the third recognition process 148 by the recognition unit 100B, the recognition unit 100B generates switching content information 126 and outputs the generated switching content information 126 to the control unit 100A.

[0169] An example of the switching content information 126 when the processing executed by the recognition unit 100B is switched from the fourth recognition processing 150 to the third recognition processing 148 is information capable of identifying an event that the processing executed by the recognition unit 100B is switched from the fourth recognition processing 150 to the third recognition processing 148. Also, an example of the switching content information 126 when the processing executed by the recognition unit 100B is switched from the third recognition processing 148 to the fourth recognition processing 150 is information capable of identifying an event that the processing executed by the recognition unit 100B is switched from the third recognition processing 148 to the fourth recognition processing 150.

[0170] An example of the model switching content information 126A when the trained model used by the recognition unit 100B is switched from the fourth recognition model 110D to the third recognition model 110C is information capable of identifying an event that the trained model to which the medical image 29 is input is switched from the fourth recognition model 110D (i.e., trained model for the second light source) to the third recognition model 110C (i.e., trained model for the first light source). Also, an example of the model switching content information 126A when the trained model used by the recognition unit 100B is switched from the third recognition model 110C to the fourth recognition model 110D is information capable of identifying an event that the trained model to which the medical image 29 is input is switched from the third recognition model 110C (i.e., trained model for the first light source) to the fourth recognition model 110D (i.e., trained model for the second light source).

[0171] The control unit 100A transmits the third recognition result 152, the fourth recognition result 154, and the switching content information 126 input from the recognition unit 100B to the endoscope device 10 via the external I / F 98 (see FIG. 4). In the endoscope device 10, the processor 88 receives the third recognition result 152, the fourth recognition result 154, and the switching content information 126 via the external I / F 86.

[0172] When processor 88 receives third recognition result 152 via external I / F 86, processor 88 generates first site name information 128, first lesion identification information 130, first differentiation information 132, and first size information 134 as information based on third recognition result 152, and displays them in first display area 35A.

[0173] When a white light optical image is displayed in the first display region 35A (i.e., when the white light optical image used in the third recognition process 148 is displayed in the first display region 35A), the processor 88 superimposes and displays a bounding box BB1 generated according to the geometric characteristics of the lesion 36 recognized by performing the third recognition process 148 on the white light optical image (e.g., the white light optical image used in the third recognition process 148) displayed in the first display region 35A. In the example shown in FIG. 9, the bounding box BB1 is superimposed and displayed at the position where the lesion 36 is captured in the white light optical image displayed in the first display region 35A. The display of the bounding box BB1 is updated in synchronization with the display timing of the white light optical image newly displayed in the first display region 35A (i.e., the timing when the white light optical image displayed in the first display region 35A is updated).

[0174] When the trained model used by the recognition unit 100B is switched from the fourth recognition model 110D to the third recognition model 110C, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 156 is displayed as one of the auxiliary information 38. The message 156 is a message indicating an event that the trained model for the second light source has been switched to the trained model for the first light source.

[0175] When processor 88 receives fourth recognition result 154 via external I / F 86, processor 88 generates first site name information 128, first lesion identification information 130, first differentiation information 132, and first size information 134 as information based on fourth recognition result 154, and displays them in first display area 35A.

[0176] When a special light optical image is displayed in the first display area 35A (i.e., when the special light optical image used in the fourth recognition process 150 is displayed in the first display area 35A), the processor 88 superimposes and displays a bounding box BB1 generated according to the geometric characteristics of the lesion 36 recognized by performing the fourth recognition process 150 on the special light optical image (e.g., the special light optical image used in the fourth recognition process 150) displayed in the first display area 35A. In the example shown in FIG. 9, the bounding box BB1 is superimposed and displayed at the position where the lesion 36 is captured in the special light optical image displayed in the first display area 35A. The display of the bounding box BB1 is updated in synchronization with the display timing of the special light optical image newly displayed in the first display area 35A (i.e., the timing when the special light optical image displayed in the first display area 35A is updated).

[0177] When the trained model used by the recognition unit 100B is switched from the third recognition model 110C to the fourth recognition model 110D, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 158 is displayed as one of the auxiliary information 38. The message 158 is a message indicating an event that the trained model for the first light source has been switched to the trained model for the second light source.

[0178] In the example shown in FIG. 9, messages 156 and 158 are illustrated, but modifications similar to the modifications of messages 136 and 146 described in the above embodiment can be applied to messages 156 and 158.

[0179] 8 and 9, the modality related information 114 includes light source identification information 114B, and in the processing in the recognition unit 100B, the trained model as the input destination of the optical image 29A is switched according to the light source identification information 114B. The details of the switching of the trained model are displayed on the screen 35 as a message 156 or 158. Therefore, the doctor 12 can understand the switching details when the processing details in the recognition unit 100B are switched according to the type of light source from the message 156 or 158 displayed on the screen 35.

[0180] In the example shown in Figures 8 and 9, an example is given of a form in which the first light source (i.e., a light source emitting white light) and the second light source (i.e., a light source emitting special light) are switched from one to the other, but this is merely one example, and processing can be performed in a similar manner to the example shown in Figures 8 and 9 when switching between multiple types of special light (e.g., light for BLI and light for LCI).

[0181] 8 and 9, the modality related information 114 includes light source identification information 114B, and the third recognition model 110C and the fourth recognition model 110D are switched according to the light source identification information 114B. However, this is merely an example. For example, as shown in FIG. 10, the modality related information 114 may include magnification specification information 114C, and the trained model used in the recognition unit 100B may be switched according to the magnification specification information 114C.

[0182] The magnification specification information 114C is information capable of specifying the magnification applied to the medical image 29 (here, as an example, the optical image 29A). The information capable of specifying the magnification applied to the medical image 29 refers to, for example, information capable of specifying the magnification currently applied to the zoom lens of the camera 60 (for example, an identifier or a magnification ratio, etc.). For the sake of convenience, two-stage magnification, zoom-in and zoom-out, will be described as an example of magnification. Here, zoom-in refers to zoom-in at a constant first magnification ratio, and zoom-out refers to zoom-out at a constant second magnification ratio.

[0183] The magnification change specific information 114C is held by a computer (not shown) of the endoscope body 16. The content of the magnification change specific information 114C is changed when zooming in or zooming out is changed to the other. The processor 72 of the control device 22 acquires the magnification change specific information 114C from the endoscope body 16 at the timing when the content of the magnification change specific information 114C is changed. The processor 72 transmits the magnification change specific information 114C acquired from the endoscope body 16 to the server 2 via the function extension device 24.

[0184] In the server 2, the control unit 100A outputs the optical image 29A to the recognition unit 100B. The recognition unit 100B recognizes characteristic regions (e.g., lesions, organs, etc.) in the optical image 29A based on the optical image 29A input from the control unit 100A. To achieve this, the recognition unit 100B executes a fifth recognition process 160 and a sixth recognition process 162. The fifth recognition process 160 and the fifth recognition process 162 are examples of the "first process" and "AI process" according to the present disclosure.

[0185] In the fifth recognition process 160, based on an image obtained as an optical image 29A by capturing an image of the inside of the upper gastrointestinal tract 28 by the camera 60 in a zoomed-in state (hereinafter referred to as a "zoomed-in image"), the geometric characteristics of the lesion 36, the type of the lesion 36, the form of the lesion 36, etc., shown in the zoomed-in image are recognized. Also, in the fifth recognition process 160, the first site is recognized based on the zoomed-in image.

[0186] The fifth recognition process 160 is performed by the recognition unit 100B on the acquired zoomed-in image every time the zoomed-in image is acquired. The fifth recognition process 160 is a process for recognizing the lesion 36 and the first region by a method using AI. Here, as the fifth recognition process 160, a process using the fifth recognition model 110E is performed.

[0187] The fifth recognition model 110E is a trained model for object recognition by AI using a bounding box method. The fifth recognition model 110E is optimized by performing machine learning on the neural network using the fifth teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which the fifth example data and the fifth correct answer data are associated with each other. In other words, the fifth recognition model 110E is a trained model optimized so that the fifth correct answer data is generated by inputting the fifth example data.

[0188] The fifth example data is an image corresponding to a zoomed-in image (in other words, an image that is assumed to be a zoomed-in image). The image corresponding to the zoomed-in image is an example of the "first information" and the "first image" according to the present disclosure.

[0189] The fifth correct answer data is information corresponding to information related to the zoomed-in image (e.g., information that can identify the lesion 36 and the first part shown in the zoomed-in image). The information corresponding to the information related to the zoomed-in image is an example of the "second information" and the "second image" according to the present disclosure.

[0190] The fifth correct answer data refers to correct answer data (i.e., annotations) for the fifth example data. Here, as an example of the fifth correct answer data, annotations that identify the geometric characteristics, the type of the lesion, the lesion model, and the first site of the lesion in the image used as the fifth example data are used.

[0191] The recognition unit 100B acquires a zoomed-in image from the control unit 100A, and inputs the zoomed-in image acquired from the control unit 100A to the fifth recognition model 110E. As a result, the fifth recognition model 110E recognizes the lesion 36 and the first region shown in the input zoomed-in image every time the zoomed-in image is input, and generates and outputs the fifth recognition result 164, which is the recognition result. The fifth recognition result 164 is an example of the "execution result," "related information," and "characteristic region identification information" according to the present disclosure.

[0192] Meanwhile, in the sixth recognition process 162, based on an image obtained as optical image 29A by imaging the inside of upper gastrointestinal tract 28 by camera 60 in a zoomed-out state (hereinafter referred to as a "zoomed-out image"), the geometric characteristics of lesion 36, the type of lesion 36, the form of lesion 36, etc., shown in the zoomed-out image are recognized. Also, in the sixth recognition process 162, the first site is recognized based on the zoomed-out image.

[0193] The sixth recognition process 162 is performed by the recognition unit 100B on the acquired zoomed-out image every time the zoomed-out image is acquired. The sixth recognition process 162 is a process for recognizing the lesion 36 and the first region by a method using AI. Here, as the sixth recognition process 162, a process using the sixth recognition model 110F is performed.

[0194] The sixth recognition model 110F is a trained model for object recognition by AI using a bounding box method. The sixth recognition model 110F is optimized by performing machine learning on the neural network using sixth teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which sixth example data and sixth correct answer data are associated with each other. In other words, the sixth recognition model 110F is a trained model optimized so that the sixth correct answer data is generated by inputting the sixth example data.

[0195] The sixth example data is an image corresponding to a zoomed-out image (in other words, an image that is assumed to be a zoomed-out image). The image corresponding to a zoomed-out image is an example of the "first information" and the "first image" according to the present disclosure.

[0196] The sixth correct answer data is information corresponding to information related to the zoomed-out image (e.g., information that can identify the lesion 36 and the first part shown in the zoomed-out image). The information corresponding to the information related to the zoomed-out image is an example of the "second information" and the "second image" according to the present disclosure.

[0197] The sixth correct answer data refers to correct answer data (i.e., annotations) for the sixth example data. Here, as an example of the sixth correct answer data, annotations that identify the geometric characteristics, the type of the lesion, the lesion model, and the first site of the lesion in the image used as the sixth example data are used.

[0198] The recognition unit 100B acquires a zoomed-out image from the control unit 100A, and inputs the zoomed-out image acquired from the control unit 100A to the sixth recognition model 110F. As a result, the sixth recognition model 110F recognizes the lesion 36 and the first region shown in the input zoomed-out image every time a zoomed-out image is input, and generates and outputs the sixth recognition result 166, which is the recognition result. The sixth recognition result 166 is an example of the "execution result," "related information," and "characteristic region identification information" according to the present disclosure.

[0199] The control unit 100A switches the processing contents in the recognition unit 100B in accordance with the magnification specification information 114C. That is, the control unit 100A selectively executes the fifth recognition process 160 and the sixth recognition process 162 in accordance with the magnification specification information 114C.

[0200] When the control unit 100A refers to the magnification specification information 114C and specifies that the magnification currently being used for the camera 60 is zoom-in, it outputs the zoomed-in image to the recognition unit 100B and causes the recognition unit 100B to execute the fifth recognition process 160. That is, the recognition unit 100B inputs the zoomed-in image to the fifth recognition model 110E.

[0201] When the control unit 100A refers to the magnification specification information 114C and specifies that the magnification currently being used for the camera 60 is zoom-out, the control unit 100A outputs the zoom-out image to the recognition unit 100B and causes the recognition unit 100B to execute the sixth recognition process 162. That is, the recognition unit 100B inputs the zoom-out image to the sixth recognition model 110F.

[0202] In this way, the server 2 has the fifth recognition model 110E and the sixth recognition model 110F, and in the processing of the recognition unit 100B, the learned model as the input destination of the optical image 29A is switched according to the magnification specification information 114C. That is, the control unit 100A causes the recognition unit 100B to selectively use the fifth recognition model 110E and the sixth recognition model 110F according to the magnification specification information 114C.

[0203] 11, when the fifth recognition process 160 is executed by the recognition unit 100B, the recognition unit 100B outputs a fifth recognition result 164 to the control unit 100A. When the sixth recognition process 162 is executed by the recognition unit 100B, the recognition unit 100B outputs a sixth recognition result 166 to the control unit 100A. When the fifth recognition process 160 is switched to the sixth recognition process 162 by the recognition unit 100B, or when the sixth recognition process 162 is switched to the fifth recognition process 160 by the recognition unit 100B, the recognition unit 100B generates switching content information 126 and outputs the generated switching content information 126 to the control unit 100A.

[0204] An example of the switching content information 126 when the process executed by the recognition unit 100B is switched from the sixth recognition process 162 to the fifth recognition process 160 is information capable of identifying an event that the recognition unit 100B has switched from the sixth recognition process 162 to the fifth recognition process 160. Also, an example of the switching content information 126 when the process executed by the recognition unit 100B is switched from the fifth recognition process 160 to the sixth recognition process 162 is information capable of identifying an event that the process executed by the recognition unit 100B has been switched from the fifth recognition process 160 to the sixth recognition process 162.

[0205] An example of the model switching content information 126A when the trained model used by the recognition unit 100B is switched from the sixth recognition model 110F to the fifth recognition model 110E is information capable of identifying an event that the trained model to which the medical image 29 is input is switched from the sixth recognition model 110F (i.e., a trained model for zooming out) to the fifth recognition model 110E (i.e., a trained model for zooming in). Also, an example of the model switching content information 126A when the trained model used by the recognition unit 100B is switched from the fifth recognition model 110E to the sixth recognition model 110F is information capable of identifying an event that the trained model to which the medical image 29 is input is switched from the fifth recognition model 110E (i.e., a trained model for zooming in) to the sixth recognition model 110F (i.e., a trained model for zooming out).

[0206] The control unit 100A transmits the fifth recognition result 164, the sixth recognition result 166, and the switching content information 126 input from the recognition unit 100B to the endoscope device 10 via the external I / F 98 (see FIG. 4). In the endoscope device 10, the processor 88 receives the fifth recognition result 164, the sixth recognition result 166, and the switching content information 126 via the external I / F 86.

[0207] When processor 88 receives fifth recognition result 164 via external I / F 86, processor 88 generates first site name information 128, first lesion identification information 130, first differentiation information 132, and first size information 134 as information based on fifth recognition result 164, and displays them in first display area 35A.

[0208] When a zoomed-in image is displayed in the first display area 35A (i.e., when the zoomed-in image used in the fifth recognition process 160 is displayed in the first display area 35A), the processor 88 superimposes and displays a bounding box BB1 generated according to the geometric characteristics of the lesion 36 recognized by performing the fifth recognition process 160 on the zoomed-in image (e.g., the zoomed-in image used in the fifth recognition process 160) displayed in the first display area 35A. In the example shown in FIG. 11, the bounding box BB1 is superimposed and displayed at the position where the lesion 36 is captured in the zoomed-in image displayed in the first display area 35A. The display of the bounding box BB1 is updated in synchronization with the display timing of the zoomed-in image newly displayed in the first display area 35A (i.e., the timing when the zoomed-in image displayed in the first display area 35A is updated).

[0209] When the trained model used by the recognition unit 100B is switched from the sixth recognition model 110F to the fifth recognition model 110E, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 168 is displayed as one of the auxiliary information 38. The message 168 is a message indicating an event that the trained model for zooming out has been switched to the trained model for zooming in.

[0210] When processor 88 receives sixth recognition result 166 via external I / F 86, processor 88 generates first site name information 128, first lesion identification information 130, first differentiation information 132, and first size information 134 as information based on sixth recognition result 166, and displays them in first display area 35A.

[0211] When a zoomed-out image is displayed in the first display area 35A (i.e., when the zoomed-out image used in the sixth recognition process 162 is displayed in the first display area 35A), the processor 88 superimposes and displays a bounding box BB1 generated according to the geometric characteristics of the lesion 36 recognized by performing the sixth recognition process 162 on the zoomed-out image (e.g., the zoomed-out image used in the sixth recognition process 162) displayed in the first display area 35A. In the example shown in FIG. 11, the bounding box BB1 is superimposed and displayed at the position where the lesion 36 is captured in the zoomed-out image displayed in the first display area 35A. The display of the bounding box BB1 is updated in synchronization with the display timing of the zoomed-out image newly displayed in the first display area 35A (i.e., the timing when the zoomed-out image displayed in the first display area 35A is updated).

[0212] When the trained model used by the recognition unit 100B is switched from the fifth recognition model 110E to the sixth recognition model 110F, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 170 is displayed as one of the auxiliary information 38. The message 170 is a message indicating an event that the trained model for zooming in has been switched to the trained model for zooming out.

[0213] In the example shown in FIG. 11, messages 168 and 170 are illustrated, but modifications similar to the modifications of messages 136 and 146 described in the above embodiment can be applied to messages 168 and 170.

[0214] 10 and 11, the modality-related information 114 includes the magnification-change specification information 114C, and in the processing in the recognition unit 100B, the trained model as the input destination of the optical image 29A is switched in accordance with the magnification-change specification information 114C. The content of the switched trained model is displayed on the screen 35 as a message 168 or 170. Therefore, the doctor 12 can grasp the switching content when the processing content in the recognition unit 100B is switched in accordance with the magnification from the message 168 or 170 displayed on the screen 35.

[0215] 10 and 11, two stages of magnification change, zoom-in and zoom-out, are illustrated, but this is merely an example, and even if three or more stages of magnification change are performed, it is sufficient that a trained model corresponding to each stage of magnification change is prepared, and multiple types of trained models are switched according to the magnification change specification information 114C, and information indicating the switching content is displayed on the screen 35. Also, here, the magnification change specification information 114C is illustrated, but instead of the magnification change specification information 114C, information capable of specifying the degree of enlargement of the medical image 29 (for example, the enlargement rate) may be used, or information indicating whether the medical image 29 has been enlarged may be used.

[0216] In the examples shown in Figures 10 and 11, the modality related information 114 includes the magnification specific information 114C, and the fifth recognition model 110E and the sixth recognition model 110F are switched in response to the magnification specific information 114C. However, this is merely an example. For example, as shown in Figure 12, the modality related information 114 includes image quality information 114D, and the processing content of the recognition unit 100B may be switched in response to the image quality information 114D. For example, in the example shown in Figure 12, the seventh recognition process 172 and the eighth recognition process 174 are selectively executed by the recognition unit 100B in response to the image quality information 114D.

[0217] The image quality information 114D is information related to the image quality of the medical image 29 (here, as an example, the optical image 29A) (for example, information that can specify whether the image quality is at or above a certain level). The image quality information 114D includes image quality parameters 114D1. The image quality parameters 114D1 are parameters that define the image quality of the medical image 29 (here, as an example, the optical image 29A). Examples of the image quality parameters 114D1 include one or more parameters including gain, dynamic range, and / or spatial frequency.

[0218] The image quality information 114D is held by a computer (not shown) of the endoscope body 16. The content of the image quality information 114D is changed as the image quality parameter 114D1 is changed from one of the high image quality parameter and the low image quality parameter to the other. Each of the high image quality parameter and the low image quality parameter is one or more parameters. The high image quality parameter is one or more parameters that can realize a higher image quality of the medical image 29 than the low image quality parameter. Note that, for convenience of explanation, two parameters, a high image quality parameter and a low image quality parameter, are exemplified here, but this is merely an example, and the parameters may specify three or more different image qualities.

[0219] The processor 72 of the control device 22 acquires the image quality information 114D from the endoscope body 16 at the timing when the content of the image quality information 114D is changed. The processor 72 transmits the image quality information 114D acquired from the endoscope body 16 to the server 2 via the function extension device 24.

[0220] In the server 2, the control unit 100A outputs the optical image 29A to the recognition unit 100B. The recognition unit 100B recognizes characteristic regions (e.g., lesions, organs, etc.) in the optical image 29A based on the optical image 29A input from the control unit 100A. To achieve this, the recognition unit 100B executes a seventh recognition process 172 and an eighth recognition process 174. The seventh recognition process 172 and the eighth recognition process 174 are examples of the "first process" and "AI process" according to the present disclosure.

[0221] In the seventh recognition process 172, the geometric characteristics, type, and model of the lesion 36 shown in the high-quality image (hereinafter referred to as the "high-quality image") are recognized based on an image obtained as an optical image 29A by imaging the inside of the upper digestive tract 28, the image quality of which has been improved to a certain level or higher by using high-quality parameters.In addition, in the seventh recognition process 172, the first site is recognized based on the high-quality image.

[0222] The seventh recognition process 172 is performed by the recognition unit 100B on the acquired high-quality image every time the high-quality image is acquired. The seventh recognition process 172 is a process for recognizing the lesion 36 and the first region by a method using AI. Here, as the seventh recognition process 172, a process using the seventh recognition model 110G is performed.

[0223] The seventh recognition model 110G is a trained model for object recognition by AI using a bounding box method. The seventh recognition model 110G is optimized by performing machine learning on the neural network using seventh teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which the seventh example data and the seventh correct answer data are associated with each other. In other words, the seventh recognition model 110G is a trained model optimized so that the seventh correct answer data is generated by inputting the seventh example data.

[0224] The seventh example data is an image corresponding to a high-quality image (in other words, an image that is assumed to be a high-quality image). The image corresponding to a high-quality image is an example of the "first information" and the "first image" according to the present disclosure.

[0225] The seventh correct answer data is information corresponding to information related to the high-quality image (e.g., information that can identify the lesion 36 and the first part shown in the high-quality image). The information corresponding to the information related to the high-quality image is an example of the "second information" and "second image" according to the present disclosure.

[0226] The seventh correct answer data refers to correct answer data (i.e., annotations) for the seventh example data. Here, as an example of the seventh correct answer data, annotations that identify the geometric characteristics, the type of the lesion, the lesion model, and the first site of the lesion in the image used as the seventh example data are used.

[0227] The recognition unit 100B acquires a high-quality image from the control unit 100A, and inputs the high-quality image acquired from the control unit 100A to the seventh recognition model 110G. As a result, the seventh recognition model 110G recognizes the lesion 36 and the first part shown in the input high-quality image every time a high-quality image is input, and generates and outputs the seventh recognition result 176, which is the recognition result. The seventh recognition result 176 is an example of the "execution result," "related information," and "characteristic region identification information" according to the present disclosure.

[0228] On the other hand, in the eighth recognition process 174, the geometric characteristics, type, and model of the lesion 36 shown in the low-image-quality image (hereinafter referred to as the "low-image-quality image") are recognized based on an image obtained as an optical image 29A by imaging the inside of the upper digestive tract 28, the image quality of which has been reduced to below a certain level by using low-image-quality parameters. Also, in the eighth recognition process 174, the first site is recognized based on the low-image-quality image.

[0229] The eighth recognition process 174 is performed by the recognition unit 100B on the acquired low-quality image every time the low-quality image is acquired. The eighth recognition process 174 is a process for recognizing the lesion 36 and the first region by a method using AI. Here, the eighth recognition process 174 is a process using the eighth recognition model 110H.

[0230] The eighth recognition model 110H is a trained model for object recognition by AI using a bounding box method. The eighth recognition model 110H is optimized by performing machine learning on the neural network using eighth teacher data, which is a data set including a plurality of data (i.e., data for a plurality of frames) in which the eighth example data and the eighth correct answer data are associated with each other. In other words, the eighth recognition model 110H is a trained model optimized so that the eighth correct answer data is generated by inputting the eighth example data.

[0231] The eighth example data is an image corresponding to a low-quality image (in other words, an image that is assumed to be a low-quality image). The image corresponding to a low-quality image is an example of the "first information" and the "first image" according to the present disclosure.

[0232] The eighth correct answer data is information corresponding to information related to the low-quality image (e.g., information that can identify the lesion 36 and the first part shown in the low-quality image). The information corresponding to the information related to the low-quality image is an example of the "second information" and "second image" according to the present disclosure.

[0233] The eighth correct answer data refers to correct answer data (i.e., annotations) for the eighth example data. Here, annotations that identify the geometric characteristics, the type of the lesion, the lesion model, and the first site of the lesion in the image used as the eighth example data are used as an example of the eighth correct answer data.

[0234] Recognition unit 100B acquires low-quality images from control unit 100A, and inputs the low-quality images acquired from control unit 100A to eighth recognition model 110H. As a result, every time a low-quality image is input, eighth recognition model 110H recognizes lesion 36 and first region appearing in the input low-quality image, and generates and outputs eighth recognition result 178, which is the recognition result. Eighth recognition result 178 is an example of the "execution result," "related information," and "characteristic region identifying information" according to the present disclosure.

[0235] The control unit 100A switches the processing contents in the recognition unit 100B in accordance with the image quality information 114D. That is, the control unit 100A selectively executes the seventh recognition process 172 and the eighth recognition process 174 in accordance with the image quality information 114D.

[0236] When control unit 100A determines with reference to image quality information 114D that the image quality of optical image 29A is at or above a certain level (i.e., when image quality parameter 114D1 included in image quality information 114D is determined to be a high image quality parameter), control unit 100A outputs optical image 29A as a high image quality image to recognition unit 100B and causes recognition unit 100B to execute seventh recognition process 172. That is, recognition unit 100B inputs the high image quality image to seventh recognition model 110G.

[0237] When control unit 100A determines with reference to image quality information 114D that the image quality of optical image 29A is below a certain level (i.e., when image quality parameter 114D1 included in image quality information 114D is determined to be a low image quality parameter), control unit 100A outputs optical image 29A to recognition unit 100B as a low image quality image, and causes recognition unit 100B to execute eighth recognition process 174. That is, recognition unit 100B inputs the low image quality image to eighth recognition model 110H.

[0238] In this way, the seventh recognition model 110G and the eighth recognition model 110H exist in the server 2, and in the processing of the recognition unit 100B, the learned model as the input destination of the optical image 29A is switched according to the image quality information 114D. That is, the control unit 100A causes the recognition unit 100B to selectively use the seventh recognition model 110G and the eighth recognition model 110H according to the image quality information 114D.

[0239] 13, when the seventh recognition process 172 is executed by the recognition unit 100B, the recognition unit 100B outputs a seventh recognition result 176 to the control unit 100A. When the eighth recognition process 174 is executed by the recognition unit 100B, the recognition unit 100B outputs an eighth recognition result 178 to the control unit 100A. When the seventh recognition process 172 is switched to the eighth recognition process 174 by the recognition unit 100B, or when the eighth recognition process 174 is switched to the seventh recognition process 172 by the recognition unit 100B, the recognition unit 100B generates switching content information 126 and outputs the generated switching content information 126 to the control unit 100A.

[0240] An example of the switching content information 126 when the process executed by the recognition unit 100B is switched from the eighth recognition process 174 to the seventh recognition process 172 is information capable of identifying an event that the recognition unit 100B has switched from the eighth recognition process 174 to the seventh recognition process 172. Also, an example of the switching content information 126 when the process executed by the recognition unit 100B is switched from the seventh recognition process 172 to the eighth recognition process 174 is information capable of identifying an event that the process executed by the recognition unit 100B has switched from the seventh recognition process 172 to the eighth recognition process 174.

[0241] An example of the model switching content information 126A when the trained model used by the recognition unit 100B is switched from the eighth recognition model 110H to the seventh recognition model 110G is information capable of identifying an event that the trained model to which the medical image 29 is input is switched from the eighth recognition model 110H (i.e., a trained model for low image quality) to the seventh recognition model 110G (i.e., a trained model for high image quality). Also, an example of the model switching content information 126A when the trained model used by the recognition unit 100B is switched from the seventh recognition model 110G to the eighth recognition model 110H is information capable of identifying an event that the trained model to which the medical image 29 is input is switched from the seventh recognition model 110G (i.e., a trained model for high image quality) to the eighth recognition model 110G (i.e., a trained model for low image quality).

[0242] The control unit 100A transmits the seventh recognition result 176, the eighth recognition result 178, and the switching content information 126 input from the recognition unit 100B to the endoscope device 10 via the external I / F 98 (see FIG. 4). In the endoscope device 10, the processor 88 receives the seventh recognition result 176, the eighth recognition result 178, and the switching content information 126 via the external I / F 86.

[0243] When processor 88 receives seventh recognition result 176 via external I / F 86, processor 88 generates first site name information 128, first lesion identification information 130, first differentiation information 132, and first size information 134 as information based on seventh recognition result 176, and displays them in first display area 35A.

[0244] When a high-quality image is displayed in the first display area 35A (i.e., when the high-quality image used in the seventh recognition process 172 is displayed in the first display area 35A), the processor 88 superimposes and displays a bounding box BB1 generated according to the geometric characteristics of the lesion 36 recognized by performing the seventh recognition process 172 on the high-quality image (e.g., the high-quality image used in the seventh recognition process 172) displayed in the first display area 35A. In the example shown in FIG. 13, the bounding box BB1 is superimposed and displayed at the position where the lesion 36 is captured in the high-quality image displayed in the first display area 35A. The display of the bounding box BB1 is updated in synchronization with the display timing of a new high-quality image displayed in the first display area 35A (i.e., the timing when the high-quality image displayed in the first display area 35A is updated).

[0245] When the trained model used by the recognition unit 100B is switched from the eighth recognition model 110H to the seventh recognition model 110G, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 180 is displayed as one of the auxiliary information 38. The message 180 is a message indicating an event that the trained model for low image quality has been switched to the trained model for high image quality.

[0246] When the processor 88 receives the eighth recognition result 178 via the external I / F 86, the processor 88 generates first site name information 128, first lesion identification information 130, first differentiation information 132, and first size information 134 as information based on the eighth recognition result 178, and displays them in the first display area 35A.

[0247] When a low-quality image is displayed in the first display area 35A (i.e., when the low-quality image used in the eighth recognition process 174 is displayed in the first display area 35A), the processor 88 superimposes and displays a bounding box BB1 generated according to the geometric characteristics of the lesion 36 recognized by performing the eighth recognition process 178 on the low-quality image (e.g., the low-quality image used in the eighth recognition process 174) displayed in the first display area 35A. In the example shown in FIG. 13, the bounding box BB1 is superimposed and displayed at the position where the lesion 36 is captured in the low-quality image displayed in the first display area 35A. The display of the bounding box BB1 is updated in synchronization with the display timing of a new low-quality image displayed in the first display area 35A (i.e., the timing when the low-quality image displayed in the first display area 35A is updated).

[0248] When the trained model used by the recognition unit 100B is switched from the seventh recognition model 110G to the eighth recognition model 110H, the processor 88 converts the model switching content information 126A included in the switching content information 126 into text and displays it in the second display area 35B. In the second display area 35B, a message 182 is displayed as one of the auxiliary information 138. The message 182 is a message indicating an event that the trained model for high image quality has been switched to the trained model for low image quality.

[0249] In the example shown in FIG. 13, messages 180 and 182 are illustrated, but modifications similar to the modifications of messages 136 and 146 described in the above embodiment can be applied to messages 180 and 182.

[0250] 12 and 13, the modality-related information 114 includes image quality information 114D, and in the processing in the recognition unit 100B, the trained model as the input destination of the optical image 29A is switched in accordance with the image quality information 114D. The details of the switching of the trained model are displayed on the screen 35 as a message 180 or 182. Therefore, the doctor 12 can understand the switching details when the processing details in the recognition unit 100B are switched in accordance with the image quality of the optical image 29A from the message 180 or 182 displayed on the screen 35.

[0251] In the examples shown in Figures 12 and 13, high-quality images and low-quality images are illustrated, but this is merely one example. Even if three or more medical images 29 of different image quality are obtained, it is sufficient that a trained model corresponding to each image quality is prepared, and multiple types of trained models can be switched depending on the image quality information 114D.

[0252] In the example shown in Fig. 12, the seventh recognition process 172 and the eighth recognition process 174 are selectively executed by the recognition unit 100B according to the image quality information 114D, but this is merely one example. For example, the recognition unit 100B may execute only the seventh recognition process 172, so that only high-image-quality images are input to the seventh recognition model 110G. Here, the reason why a low-image-quality image is not input to the seventh recognition model 110G is that there is a risk that the accuracy of the seventh recognition result 176 obtained by inputting a low-image-quality image to the seventh recognition model 110G, which is a trained model for high image quality, may decrease below a reference level.

[0253] 14 is a conceptual diagram showing an example of a manner in which a high-quality image is input to the seventh recognition model 110G. As shown in FIG. 14, when the control unit 100A refers to the image quality information 114D and specifies that the image quality of the optical image 29A is at or above a certain level (i.e., when the control unit 100A specifies that the image quality parameter 114D1 included in the image quality information 114D is a high-quality parameter), the control unit 100A outputs the optical image 29A to the recognition unit 100B as a high-quality image, operates the seventh recognition model 110G, and causes the recognition unit 100B to execute the seventh recognition process 172. On the other hand, when the control unit 100A refers to the image quality information 114D and specifies that the image quality of the optical image 29A is below a certain level (i.e., when the control unit 100A specifies that the image quality parameter 114D1 included in the image quality information 114D is a low-quality parameter), the control unit 100A does not output the optical image 29A to the recognition unit 100B, does not operate the seventh recognition model 110G, and does not cause the recognition unit 100B to execute the seventh recognition process 172.

[0254] 15, the switching content information 126 includes ON / OFF information 126B. The ON / OFF information 126B is information indicating that the seventh recognition model 110G has been switched from ON to OFF (i.e., the seventh recognition model 110G has been switched from an operating state to an inoperating state), or that the seventh recognition model 110G has been switched from OFF to ON (i.e., the seventh recognition model 110G has been switched from an inoperating state to an operating state).

[0255] The control unit 100A transmits the seventh recognition result 176 and the switching content information 126 input from the recognition unit 100B to the endoscope device 10 via the external I / F 98 (see FIG. 4). In the endoscope device 10, the processor 88 receives the seventh recognition result 176 and the switching content information 126 via the external I / F 86. The processor 88 executes processing using the seventh recognition result 176 in a manner similar to the example shown in FIG.

[0256] When the on / off information 126B included in the switching content information 126 received by the processor 88 indicates that the seventh recognition model 110G has been switched from off to on, the processor 88 displays a message 188 in the second display area 35B as one of the auxiliary information 38. The message 188 is a message indicating that the trained model used by the recognition unit 100B has been switched from off to on and the reason why the trained model has been switched from off to on.

[0257] In contrast, when the on / off information 126B included in the switching content information 126 received by the processor 88 indicates that the seventh recognition model 110G has been switched from on to off, the processor 88 displays a message 190 in the second display area 35B as one of the auxiliary information 38. The message 190 is a message indicating that the trained model used by the recognition unit 100B has been switched from on to off and the reason why the trained model has been switched from on to off.

[0258] 14 and 15, messages 188 and 190 are selectively displayed in the second display area 35B according to the on / off information 126B included in the switching content information 126. Therefore, the doctor 12 can know from the message 188 or 190 whether the trained model used by the recognition unit 100B has switched from on to off, or the trained model has switched from off to on. In addition, the doctor 12 can know from the message 188 or 190 why the trained model used by the recognition unit 100B has switched from on to off, or the trained model has switched from off to on.

[0259] In the example shown in FIG. 14, a high-quality image is input to the seventh recognition model 110G, but a low-quality image is not input to the seventh recognition model 110G. However, this is merely an example, and for example, as shown in FIG. 16, a low-quality image may also be input to the seventh recognition model 110G. However, when a low-quality image is input to the seventh recognition model 110G (i.e., a trained model for high quality), there is a risk that the accuracy of the seventh recognition result 176 will fall below a reference level. Therefore, the control unit 100A switches the setting value (for example, a threshold value that determines the score of the output layer) of the seventh recognition model 110G so that the accuracy of the seventh recognition result 176 is equal to or higher than the reference level in both cases where a high-quality image is input to the seventh recognition model 110G and where a low-quality image is input to the seventh recognition model 110G. In this case, as shown in FIG. 17 as an example, the switching content information 126 includes setting value switching information 126C. The setting value switching information 126C is information indicating that the setting value applied to the seventh recognition model 110G when a high-image-quality image is input to the seventh recognition model 110G (hereinafter referred to as the "setting value for high image quality") has been switched to the setting value applied to the seventh recognition model 110G when a low-image-quality image is input to the seventh recognition model 110G (hereinafter referred to as the "setting value for low image quality"), or that the setting value for low image quality has been switched to the setting value for high image quality.

[0260] When setting value switching information 126C included in switching content information 126 received by processor 88 indicates that the setting value for low image quality has been switched to the setting value for high image quality, processor 88 displays message 192 in second display area 35B as one piece of auxiliary information 38. Message 192 is a message indicating that the setting value of the trained model has been switched from the setting value for low image quality to the setting value for high image quality.

[0261] In contrast, when setting value switching information 126C included in switching content information 126 received by processor 88 indicates that the setting value for high image quality has been switched to the setting value for low image quality, processor 88 displays message 194 in second display area 35B as a piece of auxiliary information 38. Message 194 is a message indicating that the setting value of the trained model has been switched from the setting value for high image quality to the setting value for low image quality.

[0262] 16 and 17, messages 192 and 194 are selectively displayed in the second display area 35B according to the setting value switching information 126C included in the switching content information 126. Therefore, the doctor 12 can understand from the message 192 or 194 whether the setting value of the trained model used by the recognition unit 100B has been switched from a setting value for low image quality to a setting value for high image quality, or from a setting value for high image quality to a setting value for low image quality.

[0263] In the example shown in Fig. 14, a high-quality image is input to the seventh recognition model 110G, but this is merely one example. For example, as shown in Fig. 18, an ultrasound image 29B obtained when the image mode applied to an ultrasound-type upper endoscope device is B-mode (hereinafter referred to as "B-mode ultrasound image 29B") may be input to the ninth recognition model 110I. The ninth recognition model 110I is a trained model generated in the same manner as the second recognition model 110B. The ninth recognition model 110I is different from the second recognition model 110B in that it is a trained model generated on the assumption that a B-mode ultrasound image 29B is input.

[0264] 18, image quality information 114D includes image mode specifying information 114D2. Image mode specifying information 114D2 is information that specifies an image mode applied to an ultrasonic upper endoscope device.

[0265] When the control unit 100A refers to the image mode specification information 114D2 included in the image quality information 114D and specifies that the ultrasound image 29B is a B-mode ultrasound image 29B, it outputs the ultrasound image 29B to the recognition unit 100B, operates the ninth recognition model 110I, and causes the recognition unit 100B to execute the ninth recognition process 196. On the other hand, when the control unit 100A refers to the image mode specification information 114D2 included in the image quality information 114D and specifies that the ultrasound image 29B is an ultrasound image 29B other than the B-mode (i.e., an ultrasound image 29B obtained when the image mode applied to the ultrasound-type upper endoscope device is an image mode other than the B-mode), it does not output the ultrasound image 29B to the recognition unit 100B, does not operate the ninth recognition model 110I, and does not cause the recognition unit 100B to execute the ninth recognition process 196. The ultrasound image 29B other than the B mode refers to, for example, the Doppler mode ultrasound image 29B, the elastography mode ultrasound image 29B, the THI mode ultrasound image 29B, the CH mode ultrasound image 29B, or the CHI mode ultrasound image 29B. Such ultrasound images 29B other than the B mode are not input to the ninth recognition model 110I because there is a risk that the accuracy of the ninth recognition result 198 obtained by inputting the ultrasound image 29B other than the B mode to the ninth recognition model 110I (i.e., a trained model for the B mode) may decrease below a reference level.

[0266] In this way, when the B-mode ultrasound image 29B is input to the ninth recognition model 110I and the ultrasound image 29B other than the B-mode is not input to the ninth recognition model 110I, as an example, as shown in Fig. 19, the switching content information 126 includes on / off information 126D. The on / off information 126D is information indicating that the ninth recognition model 110I has been switched from on to off (i.e., the ninth recognition model 110I has been switched from an operating state to an inoperating state), or that the ninth recognition model 110I has been switched from off to on (i.e., the ninth recognition model 110I has been switched from an inoperating state to an operating state).

[0267] The control unit 100A transmits the ninth recognition result 198 and the switching content information 126 input from the recognition unit 100B to the endoscope device 10 via the external I / F 98 (see FIG. 4). In the endoscope device 10, the processor 88 receives the ninth recognition result 198 and the switching content information 126 via the external I / F 86. The processor 88 executes processing using the ninth recognition result 198 in a manner similar to the example shown in FIG.

[0268] When the on / off information 126D included in the switching content information 126 received by the processor 88 indicates that the ninth recognition model 110I has been switched from off to on, the processor 88 displays a message 200 in the second display area 35B as one of the auxiliary information 38. The message 200 is a message indicating that the trained model used by the recognition unit 100B has been switched from off to on and the reason why the trained model has been switched from off to on.

[0269] In contrast, when the on / off information 126D included in the switching content information 126 received by the processor 88 indicates that the ninth recognition model 110I has been switched from on to off, the processor 88 displays a message 202 in the second display area 35B as one of the auxiliary information 38. The message 202 is a message indicating that the trained model used by the recognition unit 100B has been switched from on to off and the reason why the trained model has been switched from on to off.

[0270] 18 and 19, messages 200 and 202 are selectively displayed in the second display area 35B according to the on / off information 126D included in the switching content information 126. Therefore, the doctor 12 can know from the message 200 or 202 whether the trained model used by the recognition unit 100B has switched from on to off, or the trained model has switched from off to on. In addition, the doctor 12 can know from the message 200 or 202 why the trained model used by the recognition unit 100B has switched from on to off, or why the trained model has switched from off to on.

[0271] In the above embodiment, an example in which the endoscope device 10 transmits the medical information 112 to the server 2 has been given. However, the endoscope device 10 may transmit the medical information 112 to the server 2 on condition that a change has occurred in the medical information 112. Furthermore, the endoscope device 10 may transmit the changed part of the medical information 112 to the server 2 on condition that a change has occurred in the part included in the medical information 112. For example, as shown in FIG. 20, when a change has occurred in the modality related information 114, the endoscope device 10 may transmit the modality related information 114 to the server 2. Furthermore, when a change has occurred in the modality specific information 114A, the endoscope device 10 may transmit the modality specific information 114A to the server 2. Furthermore, when a change has occurred in the medical image 29, the endoscope device 10 may transmit the medical image 29 in which the change has occurred to the server 2. For example, when a change occurs in the optical image 29A, the endoscope device 10 may transmit the optical image 29A to the server 2, and when a change occurs in the ultrasonic image 29B, the endoscope device 10 may transmit the ultrasonic image 29B to the server 2. This can contribute to reducing the amount of communication between the endoscope device 10 and the server 2. It can also contribute to reducing the processing load on the server 2.

[0272] In the above embodiment, an example was given in which only AI processing (i.e., recognition processing using a trained model) was performed by the recognition unit 100B, but the recognition unit 100B may selectively perform AI processing and non-AI processing (e.g., processing to recognize a feature region by a template matching method). In this case, in the same manner as in the above embodiment, AI processing and non-AI processing may be switched according to the medical information 112, and information indicating the switching content when switching between AI processing and non-AI processing may be used as switching content information 126.

[0273] In the above embodiment, an example of the form in which the recognition unit 100B performs AI processing on the medical image 29 has been described, but this is merely an example, and AI processing may be performed on the modality related information 114. In this case, processing using a so-called generation AI may be performed on the modality related information 114. One example of the generation AI is GPT-4 (Internet search<https: / / openai.com / gpt-4> ) is used in ChatGPT. Other examples of generative AI include Stable Diffusion, Midjourney, and Craiyon.

[0274] For example, the generation AI may be input with instruction data (so-called prompts) including at least a part of the modality-related information 114, and the generation AI may output information indicating how to operate the endoscope device 10, information informing the user of the contents of medical procedures recommended to be performed during an endoscopy, or information informing the user of the contents of medical procedures recommended to be performed after an endoscopy, as text information, audio information, and / or images. The information output from the generation AI may be stored in various storage media (e.g., storage 76, 92, and / or 104, etc.), registered in an electronic medical record, displayed on the display device 18 as auxiliary information 38, printed on a medium by a printer, or output as audio from a speaker.

[0275] When AI processing using the generated AI is performed in this manner, AI processing using the generated AI and processing other than AI processing using the generated AI (for example, AI processing that does not use the generated AI (i.e., AI processing by the recognition unit 100B described in the above embodiment) and / or non-AI processing) may be switched depending on the medical information 112 (for example, medical image 29 and / or modality-related information 114).In this case, too, switching content information 126 may be generated and output by the recognition unit 100B in a manner similar to that of the above embodiment.

[0276] In the above embodiment, lesion 36 is given as an example of a "characteristic region" according to the present disclosure, but the present disclosure is not limited thereto, and the technology of the present disclosure can be applied even if a resection region, a bleeding region, a marking region, an organ, or a treatment tool (e.g., a hemostatic clip placed inside the body) is applied instead of lesion 36.

[0277] In the above embodiment, the endoscope device 10 is exemplified, but the technology of the present disclosure is not limited to this. For example, the technology of the present disclosure can be applied to an X-ray imaging device, an extracorporeal ultrasound probe, an MRI, a CT, or a fundus examination device.

[0278] In the above embodiment, the AI ​​processing using the bounding box method is exemplified, but this is merely an example, and for example, AI processing using the segmentation method may be performed instead of the AI ​​processing using the bounding box method. Also, instead of the AI ​​processing, a non-AI method (for example, a template matching method) recognition process may be performed, or a recognition process that combines the non-AI method and the AI ​​method may be performed.

[0279] In the above embodiment, a hybrid type endoscopic device of an optical upper endoscope device and an ultrasonic upper endoscope device is exemplified as the endoscopic device 10, but the optical upper endoscope device and the ultrasonic upper endoscope device may be separate. In this case, for example, on condition that the optical upper endoscope device is connected to the external I / F 70, the endoscopic device 10 transmits information capable of identifying the optical upper endoscope device as the modality identification information 114A to the server 2. Also, for example, on condition that the ultrasonic upper endoscope device is connected to the external I / F 70, the endoscopic device 10 transmits information capable of identifying the ultrasonic upper endoscope device as the modality identification information 114A to the server 2.

[0280] In the above embodiment, an example in which the medical support program 108 is stored in the storage 104 has been described, but the present disclosure is not limited to this. For example, the medical support program 108 may be stored in a portable computer-readable non-transitory storage medium such as an SSD or a USB memory. The medical support program 108 stored in the non-transitory storage medium is installed in the computer 96 of the server 2. The processor 100 executes the medical support process in accordance with the medical support program 108.

[0281] In addition, the medical support program 108 may be stored in a storage device such as another computer connected to the server 2 via a network, and the medical support program 108 may be downloaded and installed in the computer 96 in response to a request from the server 2.

[0282] It is not necessary to store all of the medical support program 108 in a storage device such as another computer connected to the server 2, or to store all of the medical support program 108 in the storage 104; only a portion of the medical support program 108 may be stored.

[0283] The hardware resources for executing the medical support processing can be various processors as shown below. An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the medical support processing by executing software, i.e., a program. Another example of a processor is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the medical support processing by using the memory.

[0284] The hardware resource for executing the medical support process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the medical support process may be a single processor.

[0285] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute medical support processing on a single IC chip, as typified by SoC. In this way, the medical support processing is realized using one or more of the above various processors as hardware resources.

[0286] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processing may be changed, without departing from the spirit of the invention.

[0287] The above description and illustrations are detailed descriptions of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations within the scope of the gist of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the present disclosure.

[0288] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0289] 1 Medical Support System 2 Server 3. Network 10 Endoscopic device 12. Doctor 16 Endoscope body 18 Display device 20 Light source device 22 Control device 24 Function expansion device 26 Subject 28 Upper gastrointestinal tract 29 Medical Imaging 29A Optical Image 29B Ultrasound image 30 light 32 Inner wall 34 Wagon 35 screens 35A 1st display area 35B 2nd display area 36,120 Lesions 38 Supporting Information 40 Control section 42 Insertion section 44 Tip 46 Curved section 48 Soft part 50 Ultrasound Probe 52 Opening for treatment tools 54 Treatment tools 56 Treatment tool insertion port 58 Lighting Equipment 60 Cameras 62 Universal Code 64 Reception device 66,84,96 Computer 68,94,106 Bus 70,86,98 External I / F 72,88,100 processors 74,90,102 Memory 76,92,104 Storage 78 Transmitting and receiving circuit 80 Ultrasonic emission signal 81 Radiation target area 82 Reflected Wave Signal 100A Control Unit 100B recognition part 108 Medical Assistance Program 110A First Recognition Model 110B Second Recognition Model 110C 3rd Recognition Model 110D 4th Recognition Model 110E 5th Recognition Model 110F 6th Recognition Model 110G 7th Recognition Model 110H 8th Recognition Model 110I 9th Recognition Model 112 Medical Information 114 Modality Related Information 114A Modality Specific Information 114B Light source identification information 114C Magnification specific information 114D Image quality information 114D1 Image quality parameters 116 First Recognition Processing 118 Second Recognition Processing 122 First recognition result 124 Second recognition result 126 Switching Content Information 126A Model Switching Information 126B,126D On / Off Information 126C Setting value switching information 128 1st part name information 130 First Lesion Identification Information 132 First identification information 134 First size information 136,146,156,158,168,170,180,182,188,190,192,194,200,202 Messages 138 2nd part name information 140 Secondary Lesion Identification Information 142 Second identification information 144 Second size information 148 Third Recognition Processing 150 4th Recognition Processing 152 Third recognition result 154 4th recognition result 160 5th Recognition Processing 162 6th Recognition Processing 164 5th recognition result 166 6th recognition result 172 7th Recognition Processing 174 8th Recognition Processing 176 7th recognition result 178 8th recognition result 196 9th Recognition Processing 198 9th recognition result BB1,BB2 bounding boxes

Claims

1. A processor is provided. A medical support device that communicates with an external device that transmits medical information including medical images and / or modality-related information, The medical image is obtained by imaging a subject using a modality; the modality-related information is information about the modality at the time when the medical image is obtained, The processor, receiving the medical information transmitted from the external device; performing a first process on the medical information; Transmitting an execution result obtained by executing the first process; switching the processing content of the first processing depending on the medical information; Transmitting switching content information indicating the switching content when the processing content is switched Medical support equipment.

2. The first processing includes an AI processing for generating the associated information by inputting the medical information to a trained model that generates second information corresponding to associated information related to the medical information by inputting first information corresponding to the medical information. The medical support device according to claim 1 .

3. The switching content information when the processing content of the AI ​​processing is switched includes AI processing switching content information indicating the content to which the processing content of the AI ​​processing is switched. The medical support device according to claim 2.

4. There are multiple types of trained models, In the AI ​​processing, the trained model as an input destination of the medical information is switched according to the medical information, The AI ​​processing switching content information when the trained model is switched includes model switching content information indicating the content in which the trained model is switched. The medical support device according to claim 3.

5. The model switching content information includes trained model identification information capable of identifying the trained model used to obtain the execution result transmitted by the processor. The medical support device according to claim 4.

6. the medical information includes the modality-related information; In the AI ​​processing, the trained model as an input destination of the medical information is switched according to the modality-related information. The medical support device according to claim 4.

7. the medical information includes the medical image; In the AI ​​processing, the trained model as an input destination of the medical image is switched according to the modality-related information. The medical support device according to claim 6.

8. the medical information includes the medical image and the modality-related information; the modality-related information includes light source identification information capable of identifying a light source used in imaging to obtain the medical image; In the AI ​​processing, the trained model as an input destination of the medical image is switched according to the light source identification information. The medical support device according to claim 4.

9. the medical information includes the medical image and the modality-related information; the modality-related information includes magnification-specific information that can specify a magnification to be applied to the medical image; In the AI ​​processing, the trained model as an input destination of the medical image is switched according to the magnification specification information. The medical support device according to claim 4.

10. the medical information includes the modality-related information; the modality-related information includes modality-specific information capable of identifying the modality, In the AI ​​processing, the trained model as an input destination of the medical information is switched according to the modality-specific information. The medical support device according to claim 4.

11. the medical information includes the medical image; In the AI ​​processing, the trained model as an input destination of the medical image is switched according to the modality-specific information. The medical support device according to claim 10.

12. the medical information includes the medical image and the modality-related information; The modality-related information includes image quality information that is information related to image quality of the medical image, The processing content of the AI ​​processing is switched according to the image quality information. The medical support device according to claim 2.

13. The image quality information includes parameters defining the image quality and / or image mode specific information that identifies an image mode applied to the modality to obtain the medical image. The medical support device according to claim 12.

14. the medical information includes the medical image; the first information is a first image corresponding to the medical image, the second information is information capable of identifying a first region corresponding to a characteristic region appearing in the medical image, The trained model generates, as the related information, characteristic region identification information capable of identifying the characteristic region when the medical image is input. The medical support device according to claim 2.

15. the modality is an endoscope device; The medical image is an endoscopic image obtained by imaging the inside of the subject's body using the endoscopic device. The medical support device according to claim 1 .

16. A medical support device according to any one of claims 1 to 15; The external device. Medical support system.

17. The external device transmits the medical information to the medical support device on condition that a change has occurred in the medical information. The medical support system according to claim 16.

18. The external device transmits the part of the medical information to the medical support device on condition that a change has occurred in the part of the medical information. The medical support system according to claim 16.

19. 1. A method for operating a medical support device that communicates with an external device that transmits medical information including medical images and / or modality-related information, comprising: The medical image is obtained by imaging a subject using a modality; the modality-related information is information about the modality at the time when the medical image is obtained, receiving the medical information transmitted from the external device; performing a first process on the medical information; Transmitting an execution result obtained by executing the first process; Switching the processing content of the first processing in accordance with the medical information; and Transmitting switching content information indicating the switching content when the processing content is switched. A method for operating a medical support device.

20. A program for causing a computer to execute a medical support process, the program being applied to a medical support device that communicates with an external device that transmits medical information including medical images and / or modality-related information, The medical image is obtained by imaging a subject using a modality; the modality-related information is information about the modality at the time when the medical image is obtained, The medical support process includes: performing a first process on the medical information; Transmitting an execution result obtained by executing the first process; Switching the processing content of the first processing in accordance with the medical information; and Transmitting switching content information indicating the switching content when the processing content is switched. program.

Images