Medical support device, endoscope system, and medical support method

Abstract

To provide a medical support device, an endoscope system, and a medical support method capable of having a user visually recognize information suitable for the kind of an endoscope scope connected to an image processing device.SOLUTION: A medical support method includes: acquisition of a first optical image and an ultrasonic image when an ultrasonic endoscope scope is connected to an image processing device; acquisition of a second optical image when an optical endoscope scope is connected to the image processing device; outputting of the first optical image and the ultrasonic image to a display device when the ultrasonic endoscope scope is connected to the image processing device; and outputting of the second optical image and a first recognition result that an optical image learned model is caused to recognize a first feature region where the second optical image is shown, to the display device when the optical endoscope scope is connected to the image processing device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a medical support device, an endoscope system, and a medical support method. [Background technology]

[0002] Patent Document 1 discloses a display control device that displays endoscopic images and ultrasound images obtained by an endoscope on the same screen. The display control device described in Patent Document 1 includes an acquisition unit that acquires a determination result from a determination unit that determines whether an endoscopic image or an ultrasound image is the subject of interest of an observer who is observing the inside of a living body using the screen, and a control unit that switches between a first state in which the screen displays endoscopic images with priority over ultrasound images and a second state in which the screen displays ultrasound images with priority over endoscopic images, based on the determination result acquired by the acquisition unit.

[0003] Patent Document 2 discloses a medical image processing device. The medical image processing device described in Patent Document 2 includes an image acquisition unit that acquires medical images, a determination unit that determines the illumination mode used when the medical images were captured, a recognition unit that performs a first recognition on the medical image if the illumination mode is determined to be a first illumination mode and a second recognition on the medical image if the illumination mode is determined to be a second illumination mode, and a display control unit that causes a display device to display a first image according to the result of the first recognition if the illumination mode is determined to be the first illumination mode and a second display according to the result of the second recognition if the illumination mode is determined to be the second illumination mode. The recognition unit includes a first recognizer that is configured by learning to perform the first recognition and detects a region of interest from the medical image, and a second recognizer that is configured by learning to perform the second recognition and classify the medical image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-013690 [Patent Document 2] International Publication No. 2020 / 075578 Summary of the Invention

[0005] One embodiment of the present disclosure provides a medical support device, an endoscopic system, and a medical support method that allow a user to visually recognize information that is suited to the type of endoscope connected to an image processing device. [Means for solving the problem]

[0006] A first aspect of the present disclosure is a medical support device that includes a processor, and when an ultrasonic endoscope scope is connected to an image processing device to which an ultrasonic endoscope scope and an optical endoscope scope can be connected, the processor acquires first image information including a first optical image and an ultrasound image generated by imaging using the ultrasonic endoscope scope, and when an optical endoscope scope is connected to the image processing device, the processor acquires second image information including a second optical image generated by imaging using the optical endoscope scope, and when an ultrasonic endoscope scope is connected to the image processing device, the processor outputs the first optical image and the ultrasound image included in the first image information to a display device, and when an optical endoscope scope is connected to the image processing device, the processor outputs to the display device a second optical image included in the second image information and a first recognition result in which the second optical image is input into a trained model for optical images and the trained model for optical images recognizes a first feature area appearing in the second optical image.

[0007] A second aspect of the present disclosure is the medical support device according to the first aspect, in which the medical support device is separate from the image processing device.

[0008] A third aspect of the present disclosure is a medical support device according to the first or second aspect, in which the medical support device is connected to an image processing device via a cable, and the processor acquires first image information and second image information from the image processing device via the cable.

[0009] A fourth aspect of the present disclosure is a medical support device according to any one of the first to third aspects, wherein the image processing device has a first image processing device and a second image processing device, the first image processing device generates an ultrasound image based on reflected waves from ultrasound emitted from an ultrasound endoscope scope, and the second image processing device generates a first optical image based on a first image signal obtained by optical imaging performed by the ultrasound endoscope scope, and generates a second optical image based on a second image signal obtained by optical imaging performed by the optical endoscope scope.

[0010] A fifth aspect of the present disclosure is a medical support device according to the fourth aspect, in which the first image information is generated by the first image processing device or the second image processing device, the second image information is generated by the first image processing device or the second image processing device, and the processor acquires the first image information from the first image processing device or the second image processing device and acquires the second image information from the first image processing device or the second image processing device.

[0011] A sixth aspect of the present disclosure is the medical support device according to the fourth or fifth aspect, in which the first image processing device, the second image processing device, and the medical support device are separate entities.

[0012] A seventh aspect of the present disclosure is a medical support device according to any one of the first to sixth aspects, wherein the first image information and the second image information each include first identification information and / or second identification information, the first identification information is information capable of identifying the type of endoscopic scope connected to the image processing device, the second identification information is information capable of identifying the type of image included in the first image information or the second image information, the types of endoscopic scope are an ultrasonic endoscopic scope and an optical endoscopic scope, and the types of images are an ultrasonic image, a first optical image, and a second optical image, and the processor identifies whether an ultrasonic endoscopic scope is connected to the image processing device or an optical endoscopic scope is connected to the image processing device based on the first identification information and / or the second identification information.

[0013] An eighth aspect of the present disclosure is a medical support device according to any one of the first to seventh aspects, in which a processor determines whether an ultrasonic endoscope scope is connected to the image processing device or whether an optical endoscope scope is connected to the image processing device by analyzing an image contained in the first image information or the second image information.

[0014] A ninth aspect of the present disclosure is a medical support device according to any one of the first to eighth aspects, in which, when an ultrasonic endoscope scope is connected to the image processing device, the processor does not output to the display device information obtained from the trained model for optical images by inputting the first optical image included in the first image information into the trained model for optical images.

[0015] A tenth aspect of the present disclosure is a medical support device according to any one of the first to ninth aspects, in which, when an ultrasound image is displayed in a first display area of ​​the display device, the processor does not output to the display device information obtained from the trained model for optical images by inputting the first optical image included in the first image information into the trained model for optical images.

[0016] An eleventh aspect of the present disclosure is a medical support device according to the ninth or tenth aspect, in which the trained model for the optical image is a trained model obtained by performing machine learning for the second optical image.

[0017] A twelfth aspect of the present disclosure is a medical support device according to any one of the first to eleventh aspects, wherein the first recognition result includes first position information capable of identifying the position of the first feature region, first feature information capable of identifying medical features of the first feature region, and / or first part information capable of identifying a part of the body to be examined by the optical endoscope.

[0018] A thirteenth aspect of the present disclosure is a medical support device according to the twelfth aspect, in which the trained model for optical images includes a first trained model to which a second optical image is input, and the first trained model generates first position information based on the input second optical image.

[0019] A fourteenth aspect of the present disclosure is a medical support device according to any one of the first to thirteenth aspects, in which, when an ultrasound image is displayed in a first display area of ​​the display device, the processor outputs to the display device a second recognition result in which the trained model for ultrasound images recognizes a second feature area appearing in the ultrasound image by inputting the ultrasound image included in the first image information into the trained model for ultrasound images.

[0020] A fifteenth aspect of the present disclosure is a medical support device according to the fourteenth aspect, in which the second recognition result includes second position information capable of identifying the position of the second feature region, second feature information capable of identifying medical features of the second feature region, and / or second part information capable of identifying the part of the body to be examined by the ultrasonic endoscope.

[0021] A sixteenth aspect of the present disclosure is a medical support device according to the fifteenth aspect, in which the trained model for ultrasound images includes a second trained model to which an ultrasound image is input, and the second trained model generates second position information based on the input ultrasound image.

[0022] A 17th aspect of the present disclosure is an endoscopic system comprising a medical support device according to any one of the first to sixteenth aspects, and an image processing device having a first image processing device and a second image processing device, wherein the first image processing device generates an ultrasound image based on reflected waves from ultrasound emitted from an ultrasound endoscope scope, and the second image processing device generates a first optical image based on a first image signal obtained by optical imaging performed by the ultrasound endoscope scope, and generates a second optical image based on a second image signal obtained by optical imaging performed by the optical endoscope scope.

[0023] An 18th aspect of the present disclosure is a medical support method including: when an ultrasonic endoscope scope is connected to an image processing device to which an ultrasonic endoscope scope and an optical endoscope scope can be connected, acquiring first image information including a first optical image and an ultrasound image generated by imaging using the ultrasonic endoscope scope; when an optical endoscope scope is connected to the image processing device, acquiring second image information including a second optical image generated by imaging using the optical endoscope scope; when an ultrasonic endoscope scope is connected to the image processing device, outputting the first optical image and the ultrasound image included in the first image information to a display device; and when an optical endoscope scope is connected to the image processing device, outputting to the display device the second optical image included in the second image information and a first recognition result in which the second optical image is input into a trained model for optical images to cause the trained model for optical images to recognize a first feature area appearing in the second optical image. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a conceptual diagram showing an example of how the medical support system is used by a doctor. [Figure 2] 1 is a conceptual diagram showing an example of the overall configuration of a medical support system when an ultrasonic endoscope is used. [Figure 3] 1 is a conceptual diagram showing an example of the overall configuration of a medical support system when an optical endoscope is used. [Figure 4] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the medical support system. [Figure 5] 1 is a block diagram showing an example of a hardware configuration of a function expansion device included in a medical support system. [Figure 6] 1 is a conceptual diagram showing an example of an aspect in which ultrasound image information is generated by an ultrasound endoscope processing device and output to an optical endoscope processing device. [Figure 7] 10 is a conceptual diagram showing an example of an aspect in which ultrasonic endoscopic image information is generated by an optical endoscopic processing device and output to a function expansion device. FIG. [Figure 8] 10 is a conceptual diagram showing an example of an aspect in which optical endoscope image information is generated by an optical endoscope processing device and output to a function expansion device. FIG. [Figure 9] FIG. 10 is a conceptual diagram showing an example of the processing content of the function extension device when an ultrasound image and an ultrasound image recognition result are displayed on a screen according to first layout information. [Figure 10] FIG. 10 is a conceptual diagram showing an example of processing contents of the function expansion device when the first optical image is displayed on the screen according to the first layout information. [Figure 11] FIG. 10 is a conceptual diagram showing an example of processing contents of the function expansion device when a second optical image and an optical image recognition result are displayed on the screen according to second layout information. [Figure 12] 10 is a flowchart showing an example of the flow of medical support processing. [Figure 13] 10 is a modified example of ultrasonic endoscopic image information. [Figure 14] 10 is a modified example of optical endoscopic image information. [Figure 15] 1 is a conceptual diagram showing an example of an aspect in which ultrasonic endoscopic image information is generated by an ultrasonic endoscopic processing device and output to a function expansion device. [Figure 16] FIG. 10 is a conceptual diagram showing an example of processing content when the second recognition model includes multiple trained models. [Figure 17] FIG. 10 is a conceptual diagram showing an example of processing content when the first recognition model includes multiple trained models. [Figure 18] FIG. 10 is a block diagram showing an example of the hardware configuration of a function expansion device included in a medical support system when a third recognition model is stored in the storage of the function expansion device. [Figure 19] FIG. 10 is a conceptual diagram showing an example of processing content when AI recognition processing is performed on the first optical image. [Figure 20] This is a conceptual diagram showing an example of the contents of AI processing when the type of endoscopic scope connected to the image processing device is identified by analyzing the images contained in the ultrasonic endoscopic image information and the images contained in the optical endoscopic image information. [Figure 21] A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, exemplary embodiments of a medical support device, an endoscope system, and a medical support method according to the present disclosure will be described with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". I / F is an abbreviation for "Interface". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network".5G is an abbreviation for "5th Generation Mobile Communication System." IC is an abbreviation for "Integrated Circuit."

[0028] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be a single physical or virtual computing device, or a combination of multiple physical or virtual computing devices. Furthermore, a processor may be a single type of computing device, or a combination of multiple types of computing devices. Examples of computing devices include a CPU, a GPU, a GPGPU, an APU, or a TPU.

[0029] In the following description, a signed memory is a memory such as a RAM in which information is temporarily stored, and is used as a work memory by a processor.

[0030] In the following description, the term "storage" refers to one or more nonvolatile storage devices that store various programs, various parameters, etc. Examples of nonvolatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0031] In the following embodiments, the external I / F with a symbol controls the exchange of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor, an antenna, etc. may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] 1 is a conceptual diagram showing an example of a manner in which an endoscope system 10 is used. The endoscope system 10 is used by a user 12 in an endoscopic examination or the like. The user 12 refers to, for example, a doctor.

[0034] The endoscopic system 10 includes an endoscope 15, an image processing device 17, a display device 18, a light source device 22, and a function expansion device 24. In this embodiment, the endoscopic system 10 is an example of an "endoscopic system" according to the present disclosure. Also, in this embodiment, the image processing device 17 is an example of an "image processing device" according to the present disclosure. Also, in this embodiment, the display device 18 is an example of a "display device" according to the present disclosure. Also, in this embodiment, the function expansion device 24 is an example of a "medical support device" according to the present disclosure.

[0035] The endoscope 15 is inserted into a hollow organ of a subject 26 (e.g., a patient) and is operated by a user 12 to perform a medical procedure within the hollow organ. In the example shown in FIG. 1 , an upper gastrointestinal tract 28 is shown as an example of a hollow organ. Medical procedures performed using the endoscope 15 inserted into the upper gastrointestinal tract 28 include observation and treatment. Targets on which medical procedures are performed include the esophagus, stomach, duodenum, bile duct, gallbladder, pancreatic duct, and pancreas.

[0036] The endoscope 15 captures images inside the upper gastrointestinal tract 28. Here, the concept of imaging includes optical imaging and ultrasonic imaging. In other words, imaging here refers to the process of detecting physical energy (e.g., reflected light or reflected ultrasonic waves) from an object to be observed inside the upper gastrointestinal tract 28 and outputting the detection results as an electrical signal that can be visualized.

[0037] The image processing device 17 is a device to which the endoscope 15 can be connected, and acquires electrical signals from the endoscope 15 and converts the acquired electrical signals into images by performing various processes on the electrical signals. That is, the image processing device 17 generates a medical image 29 based on the electrical signals acquired from the endoscope 15. The medical image 29 is displayed on the display device 18. Generally, the medical image 29 is also called an endoscopic image.

[0038] There are two types of endoscopes 15: an ultrasonic endoscope 16 and an optical endoscope 300. The ultrasonic endoscope 16 is a hybrid endoscope that combines an optical upper endoscope and an ultrasonic upper endoscope (i.e., an endoscope that combines an optical upper endoscope and an ultrasonic upper endoscope). The optical upper endoscope is inserted, for example, into the upper gastrointestinal tract 28, irradiates light 30 within the upper gastrointestinal tract 28, detects reflected light from the inner wall 32 of the upper gastrointestinal tract 28, and outputs the detection result as an electrical signal to the image processing device 17. The ultrasonic upper endoscope is inserted, for example, into the upper gastrointestinal tract 28, radiates ultrasound within the upper gastrointestinal tract 28, detects reflected waves, and outputs the detection result as an electrical signal to the image processing device 17. The optical endoscope 300 is an optical upper endoscope. Note that although the ultrasonic endoscope 16 is illustrated here, this is merely an example, and the present disclosure also applies to an ultrasonic endoscope in which the optical endoscope and the ultrasonic endoscope are separate. An example of an ultrasound endoscope in which an optical endoscope and an ultrasound endoscope are separate is an ultrasound endoscope in which a thin ultrasound probe is inserted into the forceps port of the optical endoscope. A second example of an ultrasound endoscope in which an optical endoscope and an ultrasound endoscope are separate is an ultrasound endoscope in which an ultrasound probe is attached externally to the tip of the optical endoscope.

[0039] For the sake of convenience, an upper endoscope used for upper endoscopy is shown here as an example, but this is merely one example, and the present disclosure also applies to a lower endoscope used for lower endoscopy.The present disclosure also applies to an endoscope used for endoscopic examination of organs other than the digestive system (for example, a bronchoscope, an ENT endoscope, an intracranial endoscope, or a laparoscope).

[0040] The image processing device 17 includes an ultrasonic endoscopic processing device 19 and an optical endoscopic processing device 20. The ultrasonic endoscope 16 and the optical endoscope 300 are selectively connected to the optical endoscopic processing device 20. When the ultrasonic endoscope 16 is connected to the optical endoscopic processing device 20, the optical endoscopic processing device 20 generates a first optical image 29A as one of the medical images 29 based on the reflected light detected by the ultrasonic endoscope 16. When the ultrasonic endoscope 16 is connected to the ultrasonic endoscopic processing device 19, the ultrasonic endoscopic processing device 19 generates an ultrasound image 29B as one of the medical images 29 based on the reflected wave detected by the ultrasonic endoscope 16. When the optical endoscope 300 is connected to the optical endoscopic processing device 20, the optical endoscopic processing device 20 generates a second optical image 329A as one of the medical images 29 based on the reflected light detected by the optical endoscope 300.

[0041] In this embodiment, the ultrasonic endoscopic processing device 19 is an example of a "first image processing device" according to the present disclosure. Also, in this embodiment, the optical endoscopic processing device 20 is an example of a "second image processing device" according to the present disclosure. Also, in this embodiment, the first optical image 29A is an example of a "first optical image" according to the present disclosure. Also, in this embodiment, the second optical image 329A is an example of a "second optical image" according to the present disclosure. Also, in this embodiment, the ultrasonic image 29B is an example of an "ultrasonic image" according to the present disclosure.

[0042] The ultrasonic endoscope processing device 19, the optical endoscope processing device 20, the light source device 22, and the function expansion device 24 are installed on a wagon 34. The wagon 34 has multiple stands arranged vertically, and the function expansion device 24, the light source device 22, the optical endoscope processing device 20, and the ultrasonic endoscope processing device 19 are installed from the lower stand to the upper stand. In addition, the display device 18 is installed on the top stand of the wagon 34.

[0043] The display device 18 displays various information including images. Examples of the display device 18 include a liquid crystal display and an EL display. Alternatively, instead of the display device 18, or together with the display device 18, a tablet terminal with a display may be used.

[0044] A screen 35 is displayed on the display device 18. The screen 35 includes a plurality of display areas. In the example shown in FIG. 1, a first display area 35A and a second display area 35B are shown as examples of the plurality of display areas. The size of the first display area 35A is larger than the size of the second display area 35B. The first display area 35A is used as a main display area, and the second display area 35B is used as a sub-display area. The size relationship between the first display area 35A and the second display area 35B is not limited to this, and may be any size relationship that fits within the screen 35, and the size relationship between the first display area 35A and the second display area 35B may be fixed or variable.

[0045] When the ultrasonic endoscope 16 is used by the user 12, the ultrasonic endoscope 16 is connected to the ultrasonic endoscopic processing device 19 and the optical endoscopic processing device 20. When the ultrasonic endoscope 16 is connected to the ultrasonic endoscopic processing device 19 and the optical endoscopic processing device 20, an ultrasonic image 29B is displayed in the first display area 35A, and a first optical image 29A is displayed in the second display area 35B.

[0046] On the other hand, when the optical endoscope 300 is used by the user 12, the optical endoscope 300 is connected to the optical endoscope processing device 20. When the optical endoscope 300 is connected to the optical endoscope processing device 20, a second optical image 329A is displayed in the first display area 35A.

[0047] FIG. 2 is a conceptual diagram showing an example of the overall configuration of the endoscopic system 10 when an ultrasonic endoscope 16 is used. As shown in FIG. 2, the ultrasonic endoscope 16 includes an operation unit 40 and an insertion unit 42. The insertion unit 42 is formed in a tubular shape. The insertion unit 42 has a tip end portion 44, a bending portion 46, and a flexible portion 48. The tip end portion 44, the bending portion 46, and the flexible portion 48 are arranged in this order from the distal end to the proximal end of the insertion unit 42. The flexible portion 48 is formed of a long, flexible material and connects the operation unit 40 and the bending portion 46. The bending portion 46 partially bends and rotates around the axis of the insertion unit 42 when the operation unit 40 is operated. The insertion section 42 is advanced into the depths of the upper digestive tract 28 while bending and rotating around its axis in accordance with the shape of the hollow organ (for example, the shape of the passage of the upper digestive tract 28).

[0048] The distal end portion 44 is provided with an ultrasonic probe 50 and a treatment tool opening 52. The ultrasonic probe 50 is provided on the distal end side of the distal end portion 44. The ultrasonic probe 50 is a convex ultrasonic probe that emits ultrasonic waves and receives reflected waves obtained when the emitted ultrasonic waves are reflected by an area to be irradiated (for example, an organ such as the pancreas). Here, a convex ultrasonic probe is given as an example of the ultrasonic probe 50, but this is merely an example, and a radial ultrasonic probe, for example, may also be used.

[0049] The treatment tool opening 52 is formed closer to the base end of the distal end portion 44 than the ultrasonic probe 50. The treatment tool opening 52 is an opening for allowing the treatment tool 54 to protrude from the distal end portion 44. A treatment tool insertion port 56 is formed in the operation unit 40, and the treatment tool 54 is inserted into the insertion portion 42 from the treatment tool insertion port 56. The treatment tool 54 passes through the insertion portion 42 and protrudes from the treatment tool opening 52 to the outside of the ultrasonic endoscope 16. The treatment tool opening 52 is also used as a suction port for sucking blood, internal waste, etc., and as a delivery port for delivering fluids.

[0050] 2, a puncture needle is shown as the treatment tool 54. Note that this is merely an example, and the treatment tool 54 may also be a grasping forceps, a papillotomy knife, a snare, a catheter, a guide wire, a cannula, a puncture needle with a guide sheath, or the like.

[0051] The tip 44 is provided with an illumination device 58 and a camera 60. The illumination device 58 emits light 30 (see FIG. 1). Types of light 30 emitted from the illumination device 58 include, for example, white light and special light. Examples of special light include light for BLI and / or light for LCI.

[0052] The camera 60 is mounted on the ultrasound endoscope 16. The camera 60 is inserted into the upper gastrointestinal tract 28 of the subject 26, and captures an image of the observation area while the upper gastrointestinal tract 28 is illuminated with light 30 by the illumination device 58. An example of the camera 60 is a camera equipped with a CMOS image sensor. The CMOS image sensor is merely an example, and the camera may be equipped with other types of image sensors, such as a CCD image sensor.

[0053] The ultrasonic endoscope 16 is connected to an image processing device 17 and a light source device 22. The housing of the image processing device 17 and the housing of the function expansion device 24 are physically separate. In other words, the image processing device 17 and the function expansion device 24 are separate entities. The image processing device 17 is connected to the function expansion device 24.

[0054] The housing of the ultrasonic endoscopic processing device 19 and the housing of the optical endoscopic processing device 20 are physically separate. That is, the ultrasonic endoscopic processing device 19 and the optical endoscopic processing device 20 are separate entities. Note that the ultrasonic endoscopic processing device 19 and the optical endoscopic processing device 20 being separate entities is merely an example, and the ultrasonic endoscopic processing device 19 and the optical endoscopic processing device 20 may also be integrated.

[0055] The ultrasonic endoscope 16 is connected to an ultrasonic endoscopic processing device 19, an optical endoscopic processing device 20, and a light source device 22 via a universal cord 62. The ultrasonic endoscopic processing device 19 is connected to the optical endoscopic processing device 20 via a cable C1, and the optical endoscopic processing device 20 is connected to the function expansion device 24 via a cable C2. That is, the image processing device 17 is connected to the function expansion device 24 via cables C1 and C2. In this embodiment, the cables C1 and C2 are examples of "cables" according to the present disclosure. Note that, although a wired connection is illustrated in this embodiment, the present disclosure also applies to wireless connections.

[0056] The optical endoscope processing device 20 is connected to a reception device 64. The function expansion device 24 is also connected to a display device 18. That is, the image processing device 17 is connected to the display device 18 via the function expansion device 24.

[0057] The reception device 64 receives instructions from the user 12 and outputs the received instructions as electrical signals to the optical endoscope processing device 20. Examples of the reception device 64 include a keyboard, a mouse, a touch panel, a foot switch, a microphone, and / or a remote control device.

[0058] The optical endoscope processing device 20 controls the light source device 22, exchanges various signals with the ultrasonic endoscope scope 16, and exchanges various signals with the function expansion device 24 in accordance with instructions received by the reception device 64.

[0059] The light source device 22 emits light under the control of the optical endoscope processing device 20 and supplies light 30 (see FIG. 1) to the illumination device 58. The illumination device 58 has a built-in light guide, and the light 30 supplied from the light source device 22 passes through the light guide and is irradiated from the distal end portion 44. The optical endoscope processing device 20 causes the camera 60 to capture an image in accordance with instructions received by the reception device 64, generates a first optical image 29A (see FIG. 1) based on an electrical signal acquired from the camera 60, and outputs the first optical image 29A to the function expansion device 24. Furthermore, the ultrasonic endoscope processing device 19 causes the ultrasonic probe 50 to emit ultrasonic waves in accordance with instructions received by the reception device 64, and generates an ultrasonic image 29B (see FIG. 1) based on the reflected waves received by the ultrasonic probe 50. The ultrasonic endoscope processing device 19 then outputs the generated image to the function expansion device 24 via the optical endoscope processing device 20.

[0060] The function expansion device 24 is a device that expands the functions of the image processing device 17. The function expansion device 24 supports medical procedures using the endoscope 15 by performing processing using medical images 29 input from the optical endoscopic processing device 20. One example of processing using the medical images 29 is AI-based recognition processing of the medical images 29. As such, if the function expansion device 24 is a device that performs AI-based recognition processing of the medical images 29, even if the image processing device 17 does not have a function for performing AI-based recognition processing of the medical images 29, by retrofitting the function expansion device 24 to the image processing device 17, the function for performing AI-based recognition processing of the medical images 29 is added as a function of the image processing device 17, thereby expanding the functions of the image processing device 17. The function expansion device 24 outputs various information, including the medical images 29 input from the optical endoscopic processing device 20, to the display device 18.

[0061] FIG. 3 is a conceptual diagram showing an example of the overall configuration of the endoscopic system 10 when an optical endoscope 300 is used. As shown in FIG. 3, the optical endoscope 300 includes an operation unit 340 and an insertion unit 342. The insertion unit 342 is formed in a tubular shape. The insertion unit 342 has a tip end portion 344, a bending portion 346, and a flexible portion 348. The tip end portion 344, the bending portion 346, and the flexible portion 348 are arranged in this order from the distal end to the proximal end of the insertion unit 342. The flexible portion 348 is formed of a long, flexible material and connects the operation unit 340 and the bending portion 346. The bending portion 346 partially bends and rotates around the axis of the insertion unit 342 when the operation unit 340 is operated. The insertion section 342 is advanced into the depths of the upper digestive tract 28 while bending and rotating around the axis of the insertion section 342 according to the shape of the hollow organ (for example, the shape of the passage of the upper digestive tract 28).

[0062] The distal end portion 344 is provided with a camera 352, an illumination device 354, and a treatment tool opening 356. The camera 352 and the illumination device 354 are provided on a distal end surface 344A of the distal end portion 344. Note that although an example in which the camera 352 and the illumination device 354 are provided on the distal end surface 344A of the distal end portion 344 is given here, this is merely one example, and the camera 352 and the illumination device 354 may be provided on a side surface of the distal end portion 344, so that the optical endoscope 300 is configured as a side-viewing scope.

[0063] The basic configuration of camera 352 is the same as that of camera 60 (see FIG. 2), but camera 352 is a camera with higher performance than camera 60. Images obtained by imaging with camera 352 have higher image quality than images obtained by imaging with camera 60.

[0064] The illumination device 354 has illumination windows 354A and 354B. The illumination device 354 emits light 30 (see FIG. 1 ) through the illumination windows 354A and 354B. Examples of the light 30 emitted from the illumination device 354 include visible light (e.g., white light) and invisible light (e.g., near-infrared light). The illumination device 354 also emits special light through the illumination windows 354A and 354B. Examples of the special light include light for BLI and / or light for LCI. The camera 352 captures images of the upper gastrointestinal tract 28 by an optical method while the light 30 is being emitted by the illumination device 354 into the upper gastrointestinal tract 28.

[0065] The treatment tool opening 356 is an opening for allowing a treatment tool 358 to protrude from the distal end portion 344. The treatment tool opening 356 is also used as a suction port for sucking blood, internal waste, etc., and as a delivery port for delivering fluid.

[0066] A treatment tool insertion port 360 is formed in the operation section 340, and a treatment tool 358 is inserted into the insertion section 342 from the treatment tool insertion port 360. The treatment tool 358 passes through the insertion section 342 and protrudes to the outside from a treatment tool opening 356. In the example shown in FIG. 3, a puncture needle is shown as the treatment tool 358 protruding from the treatment tool opening 356. Here, a puncture needle is shown as the treatment tool 358, but this is merely one example, and the treatment tool 358 may also be a grasping forceps, a papillotomy knife, a snare, a catheter, a guide wire, a cannula, and / or a puncture needle with a guide sheath, etc.

[0067] The optical endoscope 300 is connected to the optical endoscope processing device 20 and the light source device 22 via a universal cord 362 when in use.

[0068] 4 and 5 are block diagrams showing an example of the hardware configuration of the electrical system of the endoscopic system 10. As shown in Fig. 4, the ultrasonic endoscopic processing device 19 includes a computer 66, a bus 68, and an external I / F 70. The computer 66 includes a processor 72, a memory 74, and a storage 76. The processor 72, the memory 74, the storage 76, and the external I / F 70 are connected to the bus 68. The processor 72 controls the entire ultrasonic endoscopic processing device 19. The memory 74 and the storage 76 are used by the processor 72.

[0069] The external I / F 70 controls the exchange of various information between the processor 72 and one or more devices (hereinafter also referred to as "first external devices") that exist outside the ultrasonic endoscope processing device 19.

[0070] A transmission / reception circuit 78 is connected to the external I / F 70 as one of the first external devices. When the ultrasound endoscope 16 is connected to the ultrasound endoscopic processing device 19, the ultrasound probe 50 is connected to the transmission / reception circuit 78. The transmission / reception circuit 78 generates an ultrasound emission signal 80 of a pulse waveform in accordance with instructions from the processor 72 and outputs the generated signal to the ultrasound probe 50. The ultrasound probe 50 converts the ultrasound emission signal 80 input from the transmission / reception circuit 78 into ultrasound and emits the ultrasound toward an emission target region 81 (for example, an organ such as the pancreas). The ultrasound probe 50 receives a reflected wave obtained when the ultrasound emitted from the ultrasound probe 50 is reflected by the emission target region 81, converts the reflected wave into a reflected wave signal 82, which is an electrical signal, and outputs the reflected wave to the transmission / reception circuit 78. The transmission / reception circuit 78 digitizes the reflected wave signal 82 input from the ultrasound probe 50 and outputs the digitized reflected wave signal 82 to the processor 72 via the external I / F 70. The processor 72 generates an ultrasound image 29B (see FIG. 1) showing the state of the radiation target region 81 based on the reflected wave signal 82 input from the transmission / reception circuit 78 via the external I / F 70.

[0071] The optical endoscope processing device 20 includes a computer 84, a bus 85, and an external I / F 86. The computer 84 includes a processor 88, a memory 90, and a storage 92. The processor 88, the memory 90, the storage 92, and the external I / F 86 are connected to the bus 85. The processor 88 controls the entire optical endoscope processing device 20. The memory 90 and the storage 92 are used by the processor 88.

[0072] The external I / F 86 controls the exchange of various information between the processor 88 and one or more devices (hereinafter also referred to as "second external devices") that exist outside the optical endoscope processing device 20.

[0073] A reception device 64 is connected to the external I / F 86 as one of the second external devices, and the processor 88 acquires instructions accepted by the reception device 64 via the external I / F 86 and executes processing according to the acquired instructions.

[0074] A sensor 83 is connected to the external I / F 86 as one of the second external devices. The sensor 83 is provided in the image processing device 17 and detects whether the ultrasonic endoscope 16 or the optical endoscope 300 is connected to the image processing device 17. Types of the sensor 83 include, for example, a microswitch, a proximity sensor, a magnetic sensor, and a photosensor. The processor 88 acquires the detection result by the sensor 83 via the external I / F 86 and determines, based on the acquired detection result, whether the ultrasonic endoscope 16 or the optical endoscope 300 is connected to the image processing device 17.

[0075] When the ultrasonic endoscope 16 is connected to the optical endoscopic processing device 20, a camera 60 is connected to the external I / F 86 as one of the second external devices. The external I / F 86 controls the exchange of various information between the camera 60 and the processor 88. The processor 88 controls the camera 60 via the external I / F 86. The processor 88 also acquires, via the external I / F 86, an electrical signal obtained when the camera 60 captures an image of the interior of the upper gastrointestinal tract 28, and generates a first optical image 29A based on the acquired electrical signal.

[0076] When the optical endoscope 300 is connected to the image processing device 17, a camera 352 is connected to the external I / F 86 as one of the second external devices. The external I / F 86 controls the exchange of various information between the camera 352 and the processor 88. The processor 88 controls the camera 352 via the external I / F 86. The processor 88 also acquires, via the external I / F 86, an electrical signal obtained when the camera 352 captures an image of the interior of the upper gastrointestinal tract 28, and generates a second optical image 329A based on the acquired electrical signal.

[0077] The light source device 22 is connected to the external I / F 86 as one of the second external devices, and the external I / F 86 controls the exchange of various information between the light source device 22 and the processor 88. When the ultrasonic endoscope 16 is connected to the optical endoscope processing device 20, the light source device 22 supplies light to the illumination device 58 under the control of the processor 88. When the optical endoscope 300 is connected to the optical endoscope processing device 20, the light source device 22 supplies light to the illumination device 354 under the control of the processor 88. The illumination devices 58 and 354 irradiate the light supplied from the light source device 22.

[0078] The external I / F 86 is connected to the external I / F 70 as one of the second external devices, and the processor 88 exchanges various information with the processor 72 via the external I / Fs 70 and 88.

[0079] A reception device 64 is connected to the external I / F 86 as one of the second external devices, and the processor 88 acquires instructions accepted by the reception device 64 via the external I / F 96 and executes processing according to the acquired instructions.

[0080] The function expansion device 24 is connected as one of the second external devices to the external I / F 86. For example, as shown in Fig. 5, the function expansion device 24 includes a computer 96 and an external I / F 98, and the external I / F 86 is connected to the external I / F 98.

[0081] The computer 96 includes a processor 100, a memory 102, and a storage 104. The processor 100, the memory 102, the storage 104, and the external I / F 98 are connected to a bus 106. In this embodiment, the processor 100 is an example of a "processor" according to the present disclosure.

[0082] The external I / F 98 controls the exchange of various information between the processor 100 and one or more devices (hereinafter also referred to as "third external devices") that exist outside the function extension device 24.

[0083] The external I / F 86 controls the exchange of various information between the processor 100 of the function expansion device 24 and the processor 88 (see FIG. 4) of the optical endoscope processing device 20, which serves as one of the third external devices. For example, the processor 100 acquires a medical image 29 (see FIG. 1) from the processor 88 of the optical endoscope processing device 20 via the external I / Fs 86 and 98. Then, the processor 100 executes processing using the medical image 29.

[0084] A display device 18 is connected to the external I / F 98 as one of the third external devices. The processor 100 controls the display device 18 via the external I / F 98 to cause the display device 18 to display various information (e.g., a medical image 29, etc.). For example, the processor 100 displays the medical image 29 on a screen 35 of the display device 18. In the example shown in FIG. 5, an ultrasound image 29B is displayed in the first display region 35A, and a first optical image 29A is displayed in the second display region 35B.

[0085] In recent years, the image processing device 17 may be connected to the display device 18 via an AI-BOX, which is a box-type device equipped with an AI function. The AI-BOX acquires a first optical image 29A and an ultrasound image 29B from the image processing device 17 and outputs the first optical image 29A and the ultrasound image 29B to the display device 18. The AI-BOX also performs AI recognition processing on the ultrasound image 29B and outputs the recognition processing result to the display device 18. The AI-BOX displays the first optical image 29A and the ultrasound image 29B on the screen 35 in a specified layout, and also displays the recognition processing result as a bounding box or the like superimposed on the ultrasound image 29B.

[0086] Conventional AI-BOX specifications include specifications that assume the use of an ultrasonic endoscope 16 and specifications that assume the use of an optical endoscope 300. In the former specifications, when the optical endoscope 300 is connected to the image processing device 17, the AI-BOX cannot display the second optical image 329A input from the image processing device 17 on the screen 35, nor can it perform AI recognition processing on the second optical image 329A and display the recognition processing results on the screen 35. On the other hand, in the latter specifications, when the ultrasonic endoscope 16 is connected to the image processing device 17, the AI-BOX cannot display the first optical image 29A and the ultrasound image 29B input from the image processing device 17 on the screen 35, nor can it perform AI recognition processing on the ultrasound image 29B and display the recognition processing results on the screen 35. As such, conventional AI-BOXes cannot allow the user 12 to visually recognize information suited to the type of endoscope 15 connected to the image processing device 17.

[0087] In view of these circumstances, in this embodiment, as shown in FIG. 5 as an example, medical support processing is performed by a processor 100 of a function expansion device 24, which is an improved version of the conventional AI-BOX. A medical support program 108, a first recognition model 110A, and a second recognition model 110B are stored in a storage 104. The processor 100 reads the medical support program 108 from the storage 104 and executes the read medical support program 108 on a memory 102 to perform medical support processing. The medical support processing is realized by the processor 100 operating as a control unit 100A and a recognition unit 100B in accordance with the medical support program 108 executed on the memory 102. As will be described in detail later, the first recognition model 110A and the second recognition model 110B are used by the recognition unit 100B.

[0088] 6 shows an example of processing performed by the processor 72 of the ultrasonic endoscopic processing device 19 when the processor 72 acquires a reflected wave signal 82 from the transmission / reception circuit 78 while the ultrasonic endoscope 16 is connected to the image processing device 17. As shown in FIG. 6, the processor 72 acquires the reflected wave signal 82 from the transmission / reception circuit 78 and generates an ultrasonic image 29B based on the acquired reflected wave signal 82. The processor 72 generates ultrasonic image information 116. The ultrasonic image information 116 includes the ultrasonic image 29B and first related information 114. The first related information 114 is information related to the ultrasonic image 29B and is assigned to the ultrasonic image 29B. The first related information 114 includes ultrasonic image identification information 118. The ultrasonic image identification information 118 is information that can identify that the image to which the first related information 114 is assigned is the ultrasonic image 29B. The processor 72 outputs the ultrasonic image information 116 to the optical endoscopic processing device 20. Ultrasound image information 116 is generated and output at a first frame rate (for example, 15 frames / second).

[0089] FIG. 7 shows an example of processing performed by the processor 88 of the optical endoscopic processing device 20 when the ultrasonic endoscope 16 is connected to the image processing device 17. As shown in FIG. 7, the processor 88 acquires a first image signal 119A, which is an electrical signal obtained by capturing an image of the interior of the upper gastrointestinal tract 28 with the camera 60, and generates a first optical image 29A based on the acquired first image signal 119A. The processor 88 also generates first optical image information 120. The first optical image information 120 includes the first optical image 29A and second related information 121. The second related information 121 is information related to the first optical image 29A and is assigned to the first optical image 29A. The second related information 121 includes first optical image identification information 122. The first optical image identification information 122 is information that can identify that the image to which the second related information 121 is assigned is the first optical image 29A. The first optical image information 120 is generated at a second frame rate (e.g., 30 frames per second).

[0090] The processor 88 generates ultrasonic endoscopic image information 126 and outputs the generated ultrasonic endoscopic image information 126 to the function expansion device 24. The ultrasonic endoscopic image information 126 is generated and output at the second frame rate.

[0091] The ultrasonic endoscopic image information 126 includes first optical image information 120 and first layout information 124A. The first layout information 124A is received by the reception device 64. The processor 88 acquires the first layout information 124A received by the reception device 64. The first layout information 124A is information indicating the layout of the first optical image 29A and the ultrasound image 29B within the screen 35. An example of the first layout information 124A is information instructing that the ultrasound image 29B be displayed in the first display area 35A, the first optical image 29A be displayed in the second display area 35B, and first reference information (here, as an example, information in text format that represents the recognition result obtained by performing AI recognition processing on the ultrasound image 29B) be displayed around the display location of the ultrasound image 29B within the first display area 35A. Here, an example has been given in which the first layout information 124A is received by the reception device 64, but this is merely one example, and the first layout information 124A may be determined in advance or may be changed depending on the operating status and / or design changes of the endoscopic system 10, etc.

[0092] The processor 88 acquires the ultrasound image information 116 from the ultrasound endoscopic processing device 19. When the timing of acquiring the ultrasound image information 116 coincides with the timing of generating the first optical image information 120, the processor 88 includes the ultrasound image information 116 in the ultrasound endoscopic image information 126. This is because the frame rate at which the ultrasound endoscopic image information 126 is generated (i.e., the first frame rate) is lower than the frame rate at which the first optical image information 120 is generated (i.e., the second frame rate).

[0093] In this embodiment, the first image signal 119A is an example of a "first image signal" according to the present disclosure. In this embodiment, the ultrasonic endoscopic image information 126 is an example of "first image information" according to the present disclosure. In addition, in this embodiment, the ultrasonic image identification information 118 and the first optical image identification information 122 are examples of "second identification information" according to the present disclosure. In addition, in this embodiment, the first optical image 29A included in the first optical image information 126 of the ultrasonic endoscopic image information 126 and the ultrasonic image 29B included in the ultrasonic image information 116 of the ultrasonic endoscopic image information 126 are examples of the "type of image included in the first image information" according to the present disclosure.

[0094] FIG. 8 shows an example of processing performed by the processor 88 of the optical endoscopic processing device 20 when the optical endoscope 300 is connected to the optical endoscopic processing device 20. As shown in FIG. 8, the processor 88 acquires a second image signal 119B, which is an electrical signal obtained by capturing an image of the interior of the upper gastrointestinal tract 28 with the camera 352, and generates a second optical image 329A based on the acquired second image signal 119B. The processor 88 also generates second optical image information 125. The second optical image information 125 includes the second optical image 329A and third related information 127. The third related information 127 is information related to the second optical image 329A and is assigned to the second optical image 329A. The third related information 127 includes second optical image identification information 129. The second optical image identification information 129 is information that can identify that the image to which the third related information 127 is assigned is the second optical image 329A. The second optical image information 125 is generated at a third frame rate (eg, 30 frames per second).

[0095] The processor 88 generates optical endoscopic image information 131 and outputs the generated optical endoscopic image information 131 to the function expansion device 24. The optical endoscopic image information 131 is generated and output at a third frame rate.

[0096] The optical endoscopic image information 131 includes second optical image information 125 and second layout information 124B. The second layout information 124B is received by the reception device 64. The processor 88 acquires the second layout information 124B received by the reception device 64. The second layout information 124B is information indicating the layout of the second optical image 329A within the screen 35. An example of the second layout information 124B is information instructing that the second optical image 329A be displayed in the first display region 35A, and that second reference information (here, as an example, information in text format representing a recognition result obtained by performing AI recognition processing on the second optical image 329A) be displayed in the second display region 35B. Here, an example has been given in which the second layout information 124B is received by the reception device 64, but this is merely one example, and the second layout information 124B may be determined in advance or may be changed depending on the operating status and / or design changes of the endoscopic system 10, etc.

[0097] In this embodiment, the second image signal 119B is an example of a "second image signal" according to the present disclosure. In this embodiment, the optical endoscopic image information 131 is an example of a "second image information" according to the present disclosure. Also, in this embodiment, the second optical image identification information 129 is an example of a "second identification information" according to the present disclosure. Also, in this embodiment, the second optical image 329A included in the second optical image information 125 of the optical endoscopic image information 131 is an example of a "type of image included in the second image information" according to the present disclosure.

[0098] 9 shows an example of processing content when processing using the ultrasound image information 116 and first layout information 124A included in the ultrasound endoscopic image information 126 is performed by the function expansion device 24. As shown in Fig. 9, the control unit 100A acquires the ultrasound endoscopic image information 126 from the optical endoscopic processing device 20 (see Fig. 7) via the cable C2 (see Fig. 3), and acquires ultrasound image identification information 118 from the first related information 114 included in the ultrasound image information 116 of the acquired ultrasound endoscopic image information 126. The control unit 100A identifies that the ultrasound endoscopic scope 16 is connected to the image processing device 17 based on the ultrasound image identification information 118.

[0099] 9, the control unit 100A acquires first layout information 124A included in the ultrasonic endoscopic image information 126, and displays the ultrasonic image 29B included in the ultrasonic image information 116 of the ultrasonic endoscopic image information 126 on the screen 35 in accordance with the acquired first layout information 124A. In this embodiment, the first layout information 124A instructs that the ultrasonic image 29B be displayed in the first display region 35A, and therefore the control unit 100A displays the ultrasonic image 29B included in the ultrasonic image information 116 of the ultrasonic endoscopic image information 126 in the first display region 35A. In this embodiment, the first display region 35A is an example of a "first display region" according to the present disclosure.

[0100] The recognition unit 100B acquires an ultrasound image 29B included in the ultrasound image information 116 of the ultrasound endoscopic image information 126, and recognizes a characteristic region 200 appearing in the ultrasound image 29B based on the ultrasound image 29B. Examples of the characteristic region 200 include a lesion 200A (e.g., a cyst) and an internal body part 200B (e.g., the pancreas) that is to be inspected by the ultrasound endoscope 16.

[0101] In order to realize recognition of the characteristic region 200 based on the ultrasound image 29B, the recognition unit 100B recognizes the lesion 200A and the part 200B using an AI method. Here, the recognition process is performed using the first recognition model 110A.

[0102] The first recognition model 110A is a trained model for object recognition using an AI bounding box method. The first recognition model 110A is optimized by performing machine learning on a neural network using first training data, which is a data set including a plurality of data (i.e., a plurality of frames of data) in which first example data and first supervised answer data are associated with each other. In other words, the first recognition model 110A is a trained model optimized so that first supervised answer data is generated when the first example data is input.

[0103] The first example data is an image corresponding to ultrasound image 29B (in other words, a sample image that simulates ultrasound image 29B). A first example of an image corresponding to ultrasound image 29B is an ultrasound image actually obtained using an ultrasound-type upper endoscope. A second example of an image corresponding to ultrasound image 29B is a virtually created image (for example, an image generated by a generation AI).

[0104] The first supervised data is supervised data (i.e., annotations) for the first example data. That is, the first supervised data is information that can distinguish and identify lesions and internal body parts shown in the images used as the first example data. Here, as an example of the first supervised data, annotations that identify the geometric characteristics (e.g., position, size, and shape) of the lesions shown in the images used as the first example data, medical characteristics of the lesions (e.g., type and form of the lesion), and internal body parts are used.

[0105] The recognition unit 100B inputs the ultrasound image 29B to the first recognition model 110A. As a result, the first recognition model 110A recognizes the lesion 200A and the region 200B shown in the input ultrasound image 29B every time the ultrasound image 29B is input, generates an ultrasound image recognition result 128 as the recognition result, and outputs the result to the display device 18.

[0106] The ultrasound image recognition result 128 includes bounding boxes BB1 and BB2, which are displayed in different ways. An ultrasound image 29B is displayed in the first display region 35A of the display device 18. In this state, the bounding boxes BB1 and BB2 are superimposed on the ultrasound image 29B. The bounding box BB1 is a frame that can identify the position of a lesion 200A within the ultrasound image 29B, and the bounding box BB2 is a frame that can identify the position of a region 200B within the ultrasound image 29B. In the example shown in FIG. 9 , the bounding box BB1 is indicated by a solid line, and the bounding box BB2 is indicated by a dashed line. This display manner is merely an example. The bounding boxes BB1 and BB2 may be displayed in different colors or intensities, or only one of the bounding boxes BB1 and BB2 may be displayed flashing. There are various ways in which the display manners of the bounding boxes BB1 and BB2 may be differentiated.

[0107] The ultrasound image recognition result 128 also includes text 202. The text 202 has first text 202A and second text 202B. The first text 202A is text that can identify the medical characteristics of the lesion 200A. The second text 202B is text that indicates the name of the region 200B.

[0108] The control unit 100A displays the text 202 on the screen 35 in accordance with the first layout information 124A included in the ultrasonic endoscopic image information 126. In this embodiment, the first layout information 124A instructs that the text 202 be displayed as first reference information around the display location of the ultrasonic image 29B in the first display area 35A, so the control unit 100A displays the text 202 around the display location of the ultrasonic image 29B in the first display area 35A. As a result, the first text 202A and the second text 202B are displayed around the ultrasonic image 29B in the first display area 35A.

[0109] In this embodiment, the feature region 200 is an example of a "second feature region" according to the present disclosure. Also, in this embodiment, the first recognition model 110A is an example of a "trained model for ultrasound images" according to the present disclosure. Also, in this embodiment, the ultrasound image recognition result 128 is an example of a "second recognition result" according to the present disclosure. Also, in this embodiment, the bounding boxes BB1 and BB2 are an example of "second position information" according to the present disclosure. Also, in this embodiment, the first text 202A is an example of "second feature information" according to the present disclosure. Also, in this embodiment, the bounding box BB2 and the second text 202B are an example of "second region information" according to the present disclosure.

[0110] 10 shows an example of processing content when processing using the first optical image information 120 and first layout information 124A included in the ultrasonic endoscopic image information 126 is performed by the function expansion device 24. As shown in Fig. 10, the control unit 100A acquires the ultrasonic endoscopic image information 126 from the optical endoscopic processing device 20 (see Fig. 7) via the cable C2 (see Fig. 2), and acquires first optical image identification information 122 from the second related information 121 included in the first optical image information 120 of the acquired ultrasonic endoscopic image information 126. The control unit 100A identifies that the ultrasonic endoscopic scope 16 is connected to the image processing device 17 based on the first optical image identification information 122.

[0111] 10, the control unit 100A acquires first layout information 124A included in the ultrasonic endoscopic image information 126, and displays the first optical image 29A included in the first optical image information 120 of the ultrasonic endoscopic image information 126 on the screen 35 in accordance with the acquired first layout information 124A. In this embodiment, the first layout information 124A instructs that the first optical image 29A be displayed in the second display region 35B, and therefore the control unit 100A displays the first optical image 29A included in the first optical image information 120 of the ultrasonic endoscopic image information 126 in the second display region 35B.

[0112] 11 shows an example of processing content when processing using the second optical image information 125 and second layout information 124B included in the optical endoscopic image information 131 is performed by the function expansion device 24. As shown in Fig. 11, the control unit 100A acquires the optical endoscopic image information 131 from the optical endoscopic processing device 20 (see Fig. 8) via the cable C2 (see Fig. 2), and acquires second optical image identification information 129 from the third related information 127 included in the second optical image information 125 of the acquired optical endoscopic image information 131. The control unit 100A identifies that the optical endoscope 300 is connected to the image processing device 17 based on the second optical image identification information 129.

[0113] 11 , the control unit 100A acquires second layout information 124B included in the optical endoscopic image information 131, and displays the second optical image 329A included in the second optical image information 125 of the optical endoscopic image information 131 on the screen 35 in accordance with the acquired second layout information 124B. In this embodiment, the second layout information 124B instructs that the second optical image 329A be displayed in the first display region 35A, and therefore the control unit 100A displays the second optical image 329A included in the second optical image information 125 of the optical endoscopic image information 131 in the first display region 35A.

[0114] Furthermore, the recognition unit 100B acquires a second optical image 329A included in the second optical image information 125 of the optical endoscopic image information 131, and recognizes a characteristic region 250 appearing in the second optical image 329A based on the second optical image 329A. Examples of the characteristic region 250 include a lesion 250A (e.g., cancer) and an internal body region 250B (e.g., a stomach region) that is to be inspected by the optical endoscope 300.

[0115] To realize the recognition of the characteristic region 250 based on the second optical image 329A, the recognition unit 100B recognizes the lesion 250A and the site 250B using an AI method. Here, the recognition process is performed using the second recognition model 110B.

[0116] The second recognition model 110B is a trained model for object recognition using an AI bounding box method, and is obtained by performing machine learning for the second optical image 329A. The second recognition model 110B is optimized by performing machine learning on a neural network using second training data, which is a data set including multiple pieces of data (i.e., multiple frames of data) in which second example data and second supervised data are associated with each other. In other words, the second recognition model 110B is a trained model optimized so that second supervised data is generated when the second example data is input.

[0117] The second example data is an image corresponding to the second optical image 329A (in other words, a sample image that simulates the second optical image 329A). A first example of an image corresponding to the second optical image 329A is an optical image actually obtained by an optical upper endoscope (for example, an optical endoscope having the same configuration as the optical endoscope 300). A second example of an image corresponding to the second optical image 329A is a virtually created image (for example, an image generated by a generation AI).

[0118] The second supervised answer data is supervised answer data (i.e., annotations) for the second example data. That is, the second supervised answer data is information that can distinguish and identify lesions and internal body parts shown in the images used as the second example data. Here, an example of the second supervised answer data is annotations that identify the geometric characteristics (e.g., position, size, and shape) of the lesions shown in the images used as the second example data, medical characteristics of the lesions (e.g., type and form of the lesion), and internal body parts.

[0119] Recognition unit 100B inputs second optical image 329A to second recognition model 110B. As a result, every time second optical image 329A is input, second recognition model 110B recognizes lesion 250A and region 250B shown in input second optical image 329A, generates optical image recognition result 139 as the recognition result, and outputs it to display device 18.

[0120] The optical image recognition result 139 includes a bounding box BB3. The second optical image 329A is displayed in the first display region 35A of the display device 18, and in this state, the bounding box BB3 is superimposed on the second optical image 329A. The bounding box BB3 is a frame that can identify the position of the lesion 250A within the second optical image 329A.

[0121] The optical image recognition result 139 also includes text 252. The text 252 has third text 252A and fourth text 252B. The third text 252A is text that can identify the medical characteristics of the lesion 250A. The fourth text 252B is text that indicates the name of the region 250B.

[0122] The control unit 100A displays the text 252 on the screen 35 in accordance with the second layout information 124B included in the optical endoscopic image information 131. In this embodiment, the second layout information 124B instructs that the text 252 be displayed as second reference information in the second display area 35B, and therefore the control unit 100A displays the text 252 in the second display area 35B. As a result, the third text 252A and the fourth text 252B are displayed in the second display area 35B.

[0123] In this embodiment, the feature region 250 is an example of a "first feature region" according to the present disclosure. Also, in this embodiment, the second recognition model 110B is an example of a "trained model for optical images" according to the present disclosure. Also, in this embodiment, the optical image recognition result 139 is an example of a "first recognition result" according to the present disclosure. Also, in this embodiment, the bounding box BB3 is an example of "first position information" according to the present disclosure. Also, in this embodiment, the third text 252A is an example of "first feature information" according to the present disclosure. Also, in this embodiment, the fourth text 252B is an example of "first part information" according to the present disclosure.

[0124] 11 shows an example in which text 252 is displayed in second display area 35B, but text 252 may be displayed in an area other than second display area 35B (for example, first display area 35A). In this case, information instructing that text 252 be displayed in an area other than second display area 35B (for example, first display area 35A) may be included in second layout information 124B. In this way, control unit 100A displays text 252 at a position (for example, first display area 35A) specified by second layout information 124B.

[0125] In the present embodiment, as described above, when the optical endoscope 300 is connected to the image processing device 17, the second optical image 329A is input to the second recognition model 110B, and the optical image recognition result 139 obtained from the second recognition model 110B is output to the display device 18. In contrast, when the ultrasonic endoscope 16 is connected to the image processing device 17, the first optical image 29A included in the ultrasonic endoscopic image information 126 is not input to the second recognition model 110B. However, the embodiment in which the first optical image 29A is not input to the second recognition model 110B is merely an example, and when the ultrasonic endoscope 16 is connected to the image processing device 17 (in other words, when the ultrasonic image 29B is displayed in the first display region 35A), the first optical image 29A included in the ultrasonic endoscopic image information 126 may be input to the second recognition model 110B, and information obtained from the second recognition model 110B may not be output to the display device 18. Here, the concept of "information not being output to the display device 18" also includes the concept of information being displayed on the screen 35 at a display level that is not visually perceptible by the user 12 or the like.

[0126] Next, the operation of the portion of the endoscope system 10 according to the present disclosure will be described with reference to Fig. 12. The flow of medical support processing shown in Fig. 12 is an example of the "medical support method" according to the present disclosure.

[0127] In the medical support processing shown in Fig. 12, first, in step ST10, the control unit 100A determines whether or not the ultrasonic endoscopic image information 126 or the optical endoscopic image information 131 has been received by the external I / F 98 (see Fig. 5). In step ST10, if neither the ultrasonic endoscopic image information 126 nor the optical endoscopic image information 131 has been received by the external I / F 98, the determination is negative, and the medical support processing proceeds to step ST30. In step ST10, if the ultrasonic endoscopic image information 126 or the optical endoscopic image information 131 has been received by the external I / F 98, the determination is positive, and the medical support processing proceeds to step ST12.

[0128] In step ST12, the control unit 100A determines whether or not the ultrasonic endoscope 16 is connected to the image processing device 17. If the ultrasonic endoscopic image information 126 received by the external I / F 98 in step ST10 contains the ultrasonic image identification information 118, it is determined that the ultrasonic endoscope 16 is connected to the image processing device 17. On the other hand, if the optical endoscopic image information 131 received by the external I / F 98 in step ST10 contains the second optical image identification information 129, it is determined that the optical endoscope 300 is connected to the image processing device 17.

[0129] In step ST12, if the ultrasonic endoscope 16 is not connected to the image processing device 17, that is, if the optical endoscope 300 is connected to the image processing device 17, the determination is negative and the medical support processing proceeds to step ST22. In step ST12, if the ultrasonic endoscope 16 is connected to the image processing device 17, the determination is positive and the medical support processing proceeds to step ST14.

[0130] In step ST14, the control unit 100A acquires the first optical image 29A and the ultrasound image 29B from the ultrasound endoscopic image information 126 received in step ST10 by the external I / F 98. After the processing of step ST14 is executed, the medical support processing proceeds to step ST16.

[0131] In step ST16, the control unit 100A acquires the first layout information 124A from the ultrasonic endoscopic image information 126 received in step ST10 by the external I / F 98. After the processing of step ST16 is executed, the medical support processing proceeds to step ST18.

[0132] In step ST18, the control unit 100A displays the first optical image 29A and the ultrasound image 29B acquired in step ST14 on the screen 35 in accordance with the first layout information 124A acquired in step ST16. In this embodiment, the ultrasound image 29B is displayed in the first display area 35A, and the first optical image 29A is displayed in the second display area 35B. After the processing of step ST18 is executed, the medical support processing proceeds to step ST20.

[0133] In step ST20, the recognition unit 100B inputs the ultrasound image 29B to the first recognition model 110A, thereby acquiring an ultrasound image recognition result 128 from the first recognition model 110A. Then, the control unit 100A outputs the ultrasound image recognition result 128 acquired from the first recognition model 110A by the recognition unit 100B to the display device 18. As a result, a bounding box BB1, a bounding box BB2, a first text 202A, and a second text 202B are displayed as the ultrasound image recognition result 128 on the screen 35 (in this embodiment, the first display region 35A in the screen 35) (see FIGS. 9 and 10). Note that the first text 202A and the second text 202B are displayed around the display location of the ultrasound image 29B in the first display region 35A in accordance with the first layout information 124A.

[0134] In step ST22, the control unit 100A acquires the second optical image 329A from the optical endoscopic image information 131 received in step ST10 by the external I / F 98. After the processing of step ST22 is executed, the medical support processing proceeds to step ST24.

[0135] In step ST24, the control unit 100A acquires the second layout information 124B from the optical endoscope image information 131 received in step ST10 by the external I / F 98. After the processing of step ST24 is executed, the medical support processing proceeds to step ST26.

[0136] In step ST26, the control unit 100A displays the second optical image 329A acquired in step ST22 on the screen 35 in accordance with the second layout information 124B acquired in step ST24. In this embodiment, the second optical image 329A is displayed in the first display area 35A (see FIG. 11). After the processing of step ST26 is executed, the medical support processing proceeds to step ST28.

[0137] In step ST28, the recognition unit 100B inputs the second optical image 329A to the second recognition model 110B, thereby acquiring the optical image recognition result 139 from the second recognition model 110B. Then, the control unit 100A outputs the optical image recognition result 139 acquired from the second recognition model 110B by the recognition unit 100B to the display device 18. As a result, the bounding box BB3, the third text 252A, and the fourth text 252B are displayed on the screen 35 as the optical image recognition result 139 (see FIG. 11). Note that the third text 252A and the fourth text 252B are displayed in the second display area 35B in accordance with the second layout information 124B. After the processing of step ST28 is executed, the medical support processing proceeds to step ST30.

[0138] In step ST30, the control unit 100A determines whether the conditions for terminating the medical support process are satisfied. A first example of the conditions for terminating the medical support process is that a predetermined time (e.g., 10 seconds) has elapsed since the execution of the medical support process was initiated. A second example of the conditions for terminating the medical support process is that an instruction to terminate the medical support process is received by the reception device 64. If the conditions for terminating the medical support process are not satisfied in step ST30, the determination is negative, and the medical support process proceeds to step ST10. If the conditions for terminating the medical support process are satisfied in step ST30, the determination is positive, and the medical support process is terminated.

[0139] As described above, in this embodiment, when the ultrasonic endoscope 16 is connected to the image processing device 17, the control unit 100A acquires ultrasonic endoscopic image information 126 including the first optical image 29A and the ultrasonic image 29B generated by imaging using the ultrasonic endoscope 16. Then, when the ultrasonic endoscope 16 is connected to the image processing device 17, the first optical image 29A and the ultrasonic image 29B included in the ultrasonic endoscopic image information 126 are output to the display device 18.

[0140] On the other hand, when the optical endoscope 300 is connected to the image processing device 17, the control unit 100A acquires optical endoscopic image information 131 including a second optical image 329A generated by imaging by the optical endoscope 300. Then, when the optical endoscope 300 is connected to the image processing device 17, the second optical image 329A included in the optical endoscopic image information 131 is output to the display device 18. In addition, the second optical image 329A is input to the second recognition model 110B, and an optical image recognition result 139 acquired from the second recognition model 110B is also output to the display device 18.

[0141] This allows the user 12 to visually recognize information suited to the type of endoscope 15 connected to the image processing device 17. For example, when an ultrasonic endoscope 16 is connected to the image processing device 17, an ultrasonic image 29B is displayed in the first display area 35A, and a first optical image 29A is displayed in the second display area 35B. Therefore, the user 12 can visually recognize the ultrasonic image 29B displayed in the first display area 35A and the first optical image 29A displayed in the second display area 35B as information suited to the ultrasonic endoscope 16 connected to the image processing device 17. Furthermore, when an optical endoscope 300 is connected to the image processing device 17, a second optical image 329A is displayed in the first display area 35A, and an optical image recognition result 139 is displayed on the screen 35. Therefore, the user 12 can visually recognize the second optical image 329A displayed in the first display area 35A and the optical image recognition result 139 displayed on the screen 35 as information suited to the optical endoscope 300 connected to the image processing device 17.

[0142] In this embodiment, the function expansion device 24 and the image processing device 17 are separate entities. The image processing device 17 has an ultrasonic endoscopic processing device 19 and an optical endoscopic processing device 20, and the ultrasonic endoscopic processing device 19 and the optical endoscopic processing device 20 are separate entities. Even if the function expansion device 24 and the image processing device 17 are separate entities, or even if the function expansion device 24, the ultrasonic endoscopic processing device 19, and the optical endoscopic processing device 20 are separate entities, the user 12 can visually recognize information that is compatible with the type of endoscope 15 connected to the image processing device 17.

[0143] Furthermore, in this embodiment, the function expansion device 24 is connected to the image processing device 17 via a cable C2. Therefore, the user 12 can selectively connect the ultrasonic endoscope 16 and the optical endoscope 300 to the function expansion device 24 by simply connecting the ultrasonic endoscope 16 and the optical endoscope 300 to the image processing device 17, without having to selectively connect them directly to the function expansion device 24. In this way, by connecting the function expansion device 24 to the image processing device 17 via the cable C2, when the ultrasonic endoscope 16 is connected to the image processing device 17, the control unit 100A of the function expansion device 24 acquires ultrasonic endoscopic image information 126 from the image processing device 17 via the cable C2. When the optical endoscope 300 is connected to the image processing device 17, the control unit 100A of the function expansion device 24 acquires optical endoscopic image information 131 from the image processing device 17 via the cable C2. Therefore, even if the user 12 does not have to selectively connect the ultrasonic endoscope 16 and the optical endoscope 300 directly to the function expansion device 24, the control unit 100A of the function expansion device 24 can acquire ultrasonic endoscopic image information 126 when the ultrasonic endoscope 16 is connected to the image processing device 17, and can acquire optical endoscopic image information 131 when the optical endoscope 300 is connected to the image processing device 17.

[0144] Furthermore, in this embodiment, the ultrasonic endoscopic processing device 19 generates an ultrasonic image 29B based on reflected waves of ultrasonic waves irradiated from the ultrasonic endoscope 16. Furthermore, the optical endoscopic processing device 20 generates a first optical image 29A based on a first image signal 119A obtained by optical imaging performed by the ultrasonic endoscope 16, and generates a second optical image 329A based on a second image signal 119B obtained by optical imaging performed by the optical endoscope 300. Therefore, the ultrasonic image 29B, the first optical image 29A, and the second optical image 329A can be generated with higher accuracy than when a general-purpose processing device capable of generating both ultrasonic images and optical images generates the first optical image 29A, the ultrasonic image 29B, and the second optical image 329A.

[0145] Furthermore, in this embodiment, the optical endoscope processing device 20 can be used both when the user 12 uses the ultrasonic endoscope 16 and when the user 12 uses the optical endoscope 300. This contributes to improved usability.

[0146] Furthermore, in this embodiment, the ultrasonic endoscopic image information 126 and the optical endoscopic image information 131 are generated by the optical endoscopic processing device 20. Then, the processor 100 of the function expansion device 24 acquires the ultrasonic endoscopic image information 126 and the optical endoscopic image information 131 from the optical endoscopic processing device 20. Therefore, whether the ultrasonic endoscope 16 is connected to the image processing device 17 or the optical endoscope 300 is connected to the image processing device 17, the processor 100 of the function expansion device 24 can appropriately acquire the ultrasonic endoscopic image information 126 and the optical endoscopic image information 131.

[0147] Furthermore, in this embodiment, the control unit 100A of the function expansion device 24 refers to the ultrasound image identification information 118 included in the ultrasound endoscopic image information 126 and the second optical image identification information 129 included in the optical endoscopic image information 131 to identify whether the endoscope 15 connected to the image processing device 17 is the ultrasound endoscope 16 or the optical endoscope 300. Therefore, compared to a case where information capable of identifying whether the endoscope 15 connected to the image processing device 17 is the ultrasound endoscope 16 or the optical endoscope 300 is manually input to the function expansion device 24, the control unit 100A of the function expansion device 24 can easily identify whether the endoscope 15 connected to the image processing device 17 is the ultrasound endoscope 16 or the optical endoscope 300.

[0148] Furthermore, in the present embodiment, when the ultrasonic endoscope 16 is connected to the image processing device 17, the first optical image 29A included in the ultrasonic endoscopic image information 126 is not input to the second recognition model 110B. The second recognition model 110B is a trained model obtained by performing machine learning for the second optical image 329A. Therefore, information obtained from the second recognition model 110B by inputting the first optical image 29A to the second recognition model 110B is less reliable than information obtained from the second recognition model 110B by inputting the second optical image 329A to the second recognition model 110B. Therefore, by preventing the first optical image 29A from being input to the second recognition model 110B, it is possible to prevent unreliable information from being provided to the user 12.

[0149] Furthermore, when the ultrasonic endoscope 16 is connected to the image processing device 17 (in other words, when the ultrasonic image 29B is displayed in the first display area 35A), the first optical image 29A included in the ultrasonic endoscopic image information 126 is input to the second recognition model 110B, and information obtained from the second recognition model 110B may be prevented from being output to the display device 18. In this way, when the ultrasonic endoscope 16 is connected to the image processing device 17 (in other words, when the ultrasonic image 29B is displayed in the first display area 35A), the first optical image 29A included in the ultrasonic endoscopic image information 126 is input to the second recognition model 110B, and information obtained from the second recognition model 110B (i.e., information with low reliability) can be prevented from being recognized by the user 12.

[0150] Furthermore, in this embodiment, when ultrasound image 29B is input to first recognition model 110A, bounding box BB1, bounding box BB2, first text 202A, and second text 202B are displayed on screen 35 as ultrasound image recognition result 128 obtained from first recognition model 110A. Bounding box BB1 is information that enables identification of the position of lesion 200A within ultrasound image 29B, bounding box BB2 is information that enables identification of the position of region 200B within ultrasound image 29B, first text 202A is information that indicates medical characteristics of lesion 200A, and second text 202B is information that indicates the name of region 200B. In this manner, user 12 can visually recognize the position of lesion 200A, the position of region 200B, the medical characteristics of lesion 200A, and the name of region 200B within ultrasound image 29B displayed in first display region 35A.

[0151] Furthermore, in this embodiment, when second optical image 329A is input to second recognition model 110B, a bounding box BB3, third text 252A, and fourth text 252B are displayed on screen 35 as optical image recognition result 139 obtained from second recognition model 110B. Bounding box BB3 is information that enables the position of lesion 205A to be identified within second optical image 329A, third text 252A is information that enables the medical characteristics of lesion 250A to be identified, and fourth text 252B is information that indicates the name of region 250B. In this manner, user 12 can visually recognize the position of lesion 250A, the position of region 250B, the medical characteristics of lesion 250A, and the name of region 250B within second optical image 329A displayed in first display region 35A.

[0152] [Other variations] In the above embodiment, the control unit 100A of the function extension device 24 identifies that the ultrasonic endoscope 16 is connected to the image processing device 17 from the first optical image identification information 122 included in the second related information 121. However, the present disclosure is not limited to this. For example, as shown in FIG. 13 , the second related information 121 may include ultrasonic endoscope identification information 132, and the control unit 100A may identify that the ultrasonic endoscope 16 is connected to the image processing device 17 from the ultrasonic endoscope identification information 132. The ultrasonic endoscope identification information 132 is information that can identify that the type of the endoscope 15 connected to the image processing device 17 is the ultrasonic endoscope 16. The ultrasonic endoscope identification information 132 is generated by the processor 83 based on a detection result 134 by the sensor 83 and is included in the second related information 121. Note that the ultrasonic endoscope identification information 132 may also be included in the first related information 114. In this case, the control unit 100A may identify that the ultrasonic endoscope 16 is connected to the image processing device 17 from the ultrasonic endoscope identification information 132 included in the first related information 114. In the example shown in FIG. 13, the ultrasonic endoscope identification information 132 is an example of the "first identification information" according to the present disclosure.

[0153] In the above embodiment, the control unit 100A of the function expansion device 24 identifies that the optical endoscope 300 is connected to the image processing device 17 from the second optical image identification information 129 included in the third related information 127. However, the present disclosure is not limited to this. For example, as shown in FIG. 14 , the third related information 127 may include optical endoscope identification information 136, and the control unit 100A may identify that the optical endoscope 300 is connected to the image processing device 17 from the optical endoscope identification information 136. The optical endoscope identification information 136 is information that can identify that the type of the endoscope 15 connected to the image processing device 17 is the optical endoscope 300. The optical endoscope identification information 136 is generated by the processor 83 based on the detection result 134 by the sensor 83 and is included in the third related information 127. In the example shown in FIG. 14 , the optical endoscope identification information 136 is an example of “first identification information” according to the present disclosure.

[0154] In the above embodiment, an example was given in which the ultrasonic image information 116 is input from the ultrasonic endoscopic processing device 19 to the optical endoscopic processing device 20, and the processor 88 of the optical endoscopic processing device 20 generates ultrasonic endoscopic image information 126 including the ultrasonic image information 116, the first optical image information 120, and the first layout information 124A, and outputs it to the function expansion device 24. However, the present disclosure is not limited to this. For example, as shown in FIG. 15 , the first optical image information 120 may be input from the optical endoscopic processing device 20 to the ultrasonic endoscopic processing device 19, and the processor 72 of the ultrasonic endoscopic processing device 19 may generate ultrasonic endoscopic image information 126 including the ultrasonic image information 116, the first optical image information 120, and the first layout information 124A, and output it to the function expansion device 24.

[0155] In addition, the second optical image information 125 may be input from the optical endoscopic processing device 20 to the ultrasonic endoscopic processing device 19, and the processor 72 of the ultrasonic endoscopic processing device 19 may generate optical endoscopic image information 131 including the second optical image information 125 and the second layout information 124B and output it to the function expansion device 24.

[0156] In the above embodiment, an example was given in which second optical image 329A is input to second recognition model 110B, causing second recognition model 110B to recognize feature region 250 appearing in second optical image 329A and output optical image recognition result 139, but the present disclosure is not limited to this. For example, as shown in FIG. 16 , second recognition model 110B may include first position recognition model 110B1, first medical feature recognition model 110B2, and first body part recognition model 110B3. In the example shown in FIG. 16 , first position recognition model 110B is an example of a “first trained model” according to the present disclosure.

[0157] The first position recognition model 110B1 is a trained model that receives a second optical image 329A, and generates and outputs first position information 139A, which is information that can identify the position of a feature region 250 in the second optical image 329A. The first medical feature recognition model 110B2 is a trained model that receives a second optical image 329A, and generates and outputs first feature information 139B, which is information that can identify medical features of the feature region 250 appearing in the second optical image 329A. The first part recognition model 110B3 is a trained model that receives a second optical image 329A, and generates and outputs first part information 139C, which is information that can identify a part 250B appearing in the second optical image 329A. In the example shown in FIG. 16, the first position information 139A is an example of “first position information” according to the present disclosure. 16, first characteristic information 139B is an example of "first characteristic information" according to the present disclosure. Also, in the example shown in Fig. 16, first part information 139C is an example of "first part information" according to the present disclosure.

[0158] Optical image recognition result 139 includes first position information 139A, first feature information 139B, and first part information 139C. Control unit 100A acquires first position information 139A, first feature information 139B, and first part information 139C, and outputs first position information 139A, first feature information 139B, and first part information 139C to display device 18. As a result, first position information 139A is displayed as bounding box BB3 in first display region 35A, first feature information 139B is displayed as third text 252A in second display region 35B, and first part information 139C is displayed as fourth text 252B in second display region 35B.

[0159] In this way, by using first position recognition model 110B1, first medical feature recognition model 110B2, and first part recognition model 110B3, first position information 139A, first feature information 139B, and first part information 139C can be generated with higher accuracy than when a single trained model generates first position information 139A, first feature information 139B, and first part information 139C.

[0160] In the above embodiment, an example was given in which an ultrasound image 29B is input to the first recognition model 110A, causing the first recognition model 110A to recognize a feature region 200 appearing in the ultrasound image 29B and output an ultrasound image recognition result 128. However, the present disclosure is not limited to this. For example, as shown in FIG. 17 , the first recognition model 110A may include a second position recognition model 110A1, a second medical feature recognition model 110A2, and a second body part recognition model 110A3. In the example shown in FIG. 17 , the second position recognition model 110A1 is an example of a “second trained model” according to the present disclosure. The second position recognition model 110A1 is a trained model that, when an ultrasound image 29B is input, generates and outputs second position information 128A, which is information capable of identifying the position of the feature region 200 within the ultrasound image 29B. The second medical feature recognition model 110A2 is a trained model that receives an ultrasound image 29B, and generates and outputs second feature information 128B, which is information capable of identifying medical features of a feature region 200 shown in the ultrasound image 29B. The second part recognition model 110A3 is a trained model that receives an ultrasound image 29B, and generates and outputs second part information 128C, which is information capable of identifying a part 200B shown in the ultrasound image 29B. In the example shown in FIG. 17, the second position information 128A is an example of "second position information" according to the present disclosure. Also, in the example shown in FIG. 17, the second feature information 128B is an example of "second feature information" according to the present disclosure. Also, in the example shown in FIG. 17, the second part information 128C is an example of "second part information" according to the present disclosure.

[0161] Ultrasound image recognition result 128 includes second position information 128A, second feature information 128B, and second region information 128C. Controller 100A acquires second position information 128A, second feature information 128B, and second region information 128C, and outputs second position information 128A, second feature information 128B, and second region information 128C to display device 18. As a result, second position information 128A is displayed as bounding boxes BB1 and BB2 in first display region 35A, second feature information 128B is displayed as first text 202A in first display region 35A, and second region information 128C is displayed as second text 202B in first display region 35A.

[0162] In this way, by using second position recognition model 110A1, second medical feature recognition model 110A2, and second part recognition model 110A3, second position information 128A, second feature information 128B, and second part information 128C can be generated with higher accuracy than when a single trained model generates second position information 128A, second feature information 128B, and second part information 128C.

[0163] In the above embodiment, an example was given in which recognition processing using first recognition model 110A is performed on ultrasound image 29B, and recognition processing using second recognition model 110B is performed on second optical image 329A, but the present disclosure is not limited to this. For example, AI-based recognition processing may also be performed on first optical image 29A. In this case, for example, as shown in FIG. 18 , third recognition model 110C is stored in storage 104, and recognition processing using third recognition model 110C is performed on first optical image 29A. Second recognition model 110B is a trained model obtained by performing machine learning for second optical image 329A, while third recognition model 110C is a trained model obtained by performing machine learning for first optical image 29A.

[0164] As an example, as shown in FIG. 19 , when the control unit 100A determines that the ultrasonic endoscope 16 is connected to the image processing device 17, the recognition unit 100B acquires a first optical image 29A included in the first optical image information 120 of the ultrasonic endoscopic image information 126. The recognition unit 100B then inputs the first optical image 29A to a third recognition model 110C, causing the third recognition model 110C to generate a recognition result 140. The recognition result 140 is a result of recognition of a characteristic region (e.g., a lesion) captured in the first optical image 29A. For example, the recognition result 140 includes information that can identify the position of the characteristic region within the first optical image 29A, information that can identify the medical characteristics of the characteristic region captured in the first optical image 29A, and information that can identify a region (e.g., an organ) captured in the first optical image 29A. The control unit 100A outputs the recognition result 140 to the display device 18. As a result, for example, a bounding box BB4 is displayed in the second display area 35B as information that can identify the position of the characteristic area in the first optical image 29A. In the example shown in Fig. 19, the bounding box BB4 is superimposed on the first optical image 29A displayed in the second display area 35B.

[0165] In the above embodiment, a convex ultrasonic endoscope (e.g., the ultrasonic endoscopic processing device 19) is exemplified. However, the present disclosure can also be applied to a radial ultrasonic endoscope (i.e., an ultrasonic endoscope equipped with a radial ultrasonic probe). In a radial ultrasonic endoscope (e.g., an ultrasonic endoscope in which a radial ultrasonic probe is provided on the proximal end side of an optical camera (in other words, an ultrasonic endoscope in which an optical camera is provided on the distal end side of a cylindrical radial ultrasonic probe), it is conceivable that the appearance of the optical image acquired by the optical camera provided on the radial ultrasonic endoscope will be almost the same as that of the optical image (e.g., the second optical image 329A) acquired by an optical endoscope scope (e.g., the optical endoscope scope 300). In such a case, the second recognition model 110B used as a recognizer for the optical image acquired by the optical endoscope scope may be applied to the optical image acquired by the radial ultrasonic endoscope.

[0166] In the above embodiment, an example was given in which the control unit 100A of the function expansion device 24 identifies that the ultrasonic endoscope 16 is connected to the image processing device 17 from the first optical image identification information 122 included in the second related information 121, or identifies that the optical endoscope 300 is connected to the image processing device 17 from the second optical image identification information 129 included in the third related information 127, but the present disclosure is not limited to this. For example, it may be configured to identify whether the ultrasonic endoscope 16 is connected to the image processing device 17 or whether the optical endoscope 300 is connected to the image processing device 17 by analyzing the images included in the ultrasonic endoscopic image information 126 or the optical endoscopic image information 131.

[0167] Fig. 20 shows an example in which the type of endoscope 15 connected to the image processing device 17 is identified by analysis using AI. In the example shown in Fig. 20, the recognition unit 100B uses a fourth recognition model 110D. The fourth recognition model 110D is a trained model that recognizes the type of endoscope 15 used to obtain the input image, and generates and outputs a scope type recognition result 142 that is the recognition result.

[0168] The fourth recognition model 110D receives the first optical image 29A, the second optical image 329A, or the ultrasound image 29B. For example, when the first optical image 29A or the ultrasound image 29B is input, the fourth recognition model 110D recognizes that the type of the endoscope 15 connected to the image processing device 17 is an ultrasound endoscope 16. When the second optical image 329A is input, the fourth recognition model 110D recognizes that the type of the endoscope 15 connected to the image processing device 17 is an optical endoscope 300. The control unit 100A acquires the scope type recognition result 142 and identifies the type of the endoscope 15 connected to the image processing device 17 from the acquired scope type recognition result 142.

[0169] By doing this, the control unit 100A of the function expansion device 24 can easily identify whether the endoscope 15 connected to the image processing device 17 is an ultrasonic endoscope 16 or an optical endoscope 300, compared to when information that can identify whether the endoscope 15 connected to the image processing device 17 is an ultrasonic endoscope 16 or an optical endoscope 300 is manually input to the function expansion device 24.

[0170] Note that, although an example has been given here of a process in which the type of endoscope 15 connected to the image processing device 17 is recognized using an AI method, this is merely one example, and the present disclosure also applies to a process in which the type of endoscope 15 connected to the image processing device 17 is recognized using a non-AI method (for example, a process that does not use AI, such as template matching).

[0171] In the above embodiment, a trained model for object recognition using an AI bounding box method is exemplified, but the present disclosure can also be implemented by using a trained model for object recognition using an AI segmentation method instead of or together with the trained model for object recognition using an AI bounding box method.

[0172] In the above embodiment, an example in which the medical support processing is performed by the computer 96 has been described, but the present disclosure is not limited to this, and at least a part of the processing included in the medical support processing may be performed by a device provided outside the computer 96. An example of this case will be described below with reference to FIG.

[0173] 21 is a conceptual diagram showing an example of the configuration of an endoscope system 144. The endoscope system 144 is an example of the "endoscope system" according to the present disclosure. The endoscope system 144 differs from the endoscope system 10 described in the above embodiment in that it includes an external device 146.

[0174] The external device 146 is communicatively coupled to the computer 96 via a network 148 (eg, a WAN and / or a LAN, etc.).

[0175] An example of the external device 146 is at least one server that directly or indirectly transmits and receives data to and from the computer 96 via the network 148. The external device 146 receives a processing execution instruction provided from the processor 100 of the computer 96 via the network 148. The external device 146 then executes processing in accordance with the received processing execution instruction and transmits the processing result to the computer 96 via the network 148. In the computer 96, the processor 100 receives the processing result transmitted from the external device 146 via the network 148 and executes processing using the received processing result.

[0176] An example of the processing execution instruction is an instruction to cause the external device 146 to execute at least a part of the medical support processing. A first example of at least a part of the medical support processing (i.e., processing to be executed by the external device 146) is recognition processing using the first recognition model 110A, the second recognition model 110B, the third recognition model 110C, and / or the fourth recognition model 110D described above. In this case, the external device 146 executes the recognition processing in accordance with the processing execution instruction provided from the processor 100 via the network 148, and transmits the recognition results (e.g., the ultrasound image recognition result 128, the optical image recognition result 139, the recognition result 140, and / or the scope type recognition result 142) to the computer 96 via the network 148. In the computer 96, the processor 100 receives the recognition results and executes processing similar to that of the above-described embodiment using the received recognition results.

[0177] A second example of at least a part of the medical support processing (i.e., processing to be executed by the external device 146) is identification processing for identifying the type of endoscope 15 connected to the image processing device 17. In this case, the external device 146 executes the identification processing in accordance with a processing execution instruction given from the processor 100 via the network 148, and transmits the processing result (e.g., information that can identify the type of endoscope 15 connected to the image processing device 17) to the computer 96 via the network 148. In the computer 96, the processor 100 receives the processing result and executes processing similar to that in the above embodiment using the received processing result.

[0178] For example, the external device 146 is realized by cloud computing. Note that cloud computing is merely one example, and the external device 146 may be realized by network computing such as fog computing, edge computing, or grid computing. Instead of a server, at least one personal computer or the like may be used as the external device 146. Alternatively, the external device 146 may be a computing device with a communication function and equipped with multiple types of AI functions.

[0179] In the above embodiment, an example has been described in which the medical support program 108 is stored in the storage 104, but the present disclosure is not limited to this. For example, the medical support program 108 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or a USB memory. The medical support program 108 stored in the non-transitory storage medium is installed in the computer 96 of the endoscope system 10. The processor 100 executes medical support processing in accordance with the medical support program 108.

[0180] Alternatively, the medical support program 108 may be stored in a storage device such as another computer or server connected to the endoscopic system 10 via a network, and the medical support program 108 may be downloaded and installed on the computer 96 in response to a request from the endoscopic system 10.

[0181] It is not necessary to store the entire medical support program 108 in a storage device such as another computer or server device connected to the endoscopic system 10, or to store the entire medical support program 108 in the storage 104; only a portion of the medical support program 108 may be stored.

[0182] The hardware resources that execute the medical support processing can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes medical support processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to execute specific processes. Each processor has built-in or connected memory, and executes medical support processing by using the memory.

[0183] The hardware resource that executes the medical support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the medical support processing may be a single processor.

[0184] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes medical support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute medical support processing, on a single IC chip, as typified by SoCs. In this way, medical support processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0185] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above medical support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0186] The above-described description and illustrations are a detailed explanation of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or elements may be replaced with other parts from the above-described description and illustrations, as long as they do not deviate from the gist of the present disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the parts related to the present disclosure, the above-described description and illustrations omit explanations of common general technical knowledge that do not require particular explanation to enable the implementation of the present disclosure.

[0187] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0188] The following additional notes are provided regarding the above-described embodiments.

[0189] (Appendix 1) a processor; The processor is acquiring a first optical image and an ultrasonic image generated by imaging using the ultrasonic endoscope from an image processing device connectable to the ultrasonic endoscope; outputting the first optical image and the ultrasound image to a display device; When an instruction to display the ultrasound image in the first display area of ​​the display device is given, the ultrasound image is input to a trained model for ultrasound images, and a recognition result obtained from the trained model for ultrasound images is output to the display device. Medical support equipment.

[0190] (Appendix 2) When an instruction is given to display the first optical image in the first display area, The processor outputs to the display device a recognition result obtained from the trained model for optical images by inputting the first optical image into the trained model for optical images. 2. The medical support device of claim 1.

[0191] (Appendix 3) When the display of the ultrasound image is instructed in the first display area The processor does not output to the display device a recognition result obtained from the trained model for optical images by inputting the first optical image into the trained model for optical images. 10. The medical support device according to claim 1 or 2. [Explanation of symbols]

[0192] 10,144 Endoscopy Systems 12 Doctors 15 Endoscope 16. Ultrasound Endoscope 17 Image processing device 18 Display device 19. Ultrasound Endoscope Processing Device 20 Optical endoscope processing device 22 Light source device 24 Function expansion device 26 Subject 28 Upper gastrointestinal tract 29 Medical Imaging 29A First optical image 29B Ultrasound image 30 light 32 Inner wall 34 Wagon 35 screens 35A 1st display area 35B 2nd display area 40,340 Operation unit 42,342 Insertion section 44,344 Tip 46,346 curved section 48,348 Soft part 50 Ultrasound Probe 52,356 Opening for treatment instruments 54,358 Treatment tools 56,360 Treatment tool insertion port 58,354 lighting equipment 60,352 cameras 62,362 Universal Code 64 Reception device 66,84,96 Computer 68, 85, 106 buses 70,86,98 External I / F 72,88,100 processors 74,90,102 memory 76,92,104 Storage 78 Transmitting and receiving circuit 80 Ultrasonic emission signal 81 Radiation target area 82 Reflected Wave Signal 83 Sensors 100A control unit 100B recognition part 108 Medical Assistance Program 110A First Recognition Model 110B Second Recognition Model 110C Third Recognition Model 110D 4th Recognition Model 114 Related Information No. 1 116 Ultrasound Image Information 118 Ultrasound image specific information 119A 1st image signal 119B Second image signal 120 First Optical Image Information 121 Second Related Information 122 First optical image specific information 124A First Layout Information 124B Second layout information 125 Second Optical Image Information 126 Endoscopic Ultrasound Image Information 127 Third Related Information 128 Ultrasound image recognition results 128A 2nd location information 128B Second characteristic information 128C 2nd part information 129 Second optical image specific information 131 Optical Endoscope Image Information 132 Ultrasound Endoscope Specific Information 134 Detection Results 136 Optical Endoscope Specific Information 139 Optical Image Recognition Results 139A 1st location information 139B First characteristic information 139C 1st part information 140 Recognition results 142 Scope type recognition results 146 External device 148 Network 200,250 feature areas 200A, 250A lesions 200B,250B part 202,252 text 202A First Text 202B Second Text 252A Third Text 252B Fourth Text 300 Optical Endoscope 329A Second Optical Image 344 Tip 344A Tip surface 354A, 354B Lighting window BB1,BB2,BB3,BB4 bounding boxes C1, C2, C3 cables

Claims

1. a processor; The processor: When the ultrasonic endoscope is connected to an image processing device to which an ultrasonic endoscope and an optical endoscope can be connected, first image information including a first optical image and an ultrasonic image generated by imaging using the ultrasonic endoscope is acquired, When the optical endoscope is connected to the image processing device, second image information including a second optical image generated by imaging using the optical endoscope is acquired, When the ultrasonic endoscope is connected to the image processing device, the first optical image and the ultrasonic image included in the first image information are output to a display device; When the optical endoscope is connected to the image processing device, the second optical image included in the second image information and a first recognition result in which the trained model for optical images recognizes a first feature region appearing in the second optical image by inputting the second optical image into the trained model for optical images are output to the display device. Medical support equipment.

2. The medical support device is separate from the image processing device. The medical support device according to claim 1 .

3. the medical support device is connected to the image processing device by a cable; The processor acquires the first image information and the second image information from the image processing device via the cable. The medical support device according to claim 1 .

4. the image processing device includes a first image processing device and a second image processing device; the first image processing device generates the ultrasonic image based on a reflected wave of ultrasonic waves emitted from the ultrasonic endoscope; The second image processing device generates the first optical image based on a first image signal obtained by optically imaging the ultrasonic endoscope, and generates the second optical image based on a second image signal obtained by optically imaging the optical endoscope. The medical support device according to claim 1 .

5. the first image information is generated by the first image processing device or the second image processing device, the second image information is generated by the first image processing device or the second image processing device, The processor: acquiring the first image information from the first image processing device or the second image processing device; Acquiring the second image information from the first image processing device or the second image processing device The medical support device according to claim 4.

6. The first image processing device, the second image processing device, and the medical support device are separate entities. The medical support device according to claim 4.

7. each of the first image information and the second image information includes first identification information and / or second identification information; the first identification information is information capable of identifying a type of endoscope connected to the image processing device, the second identification information is information capable of identifying a type of image included in the first image information or the second image information, the types of the endoscope are the ultrasonic endoscope and the optical endoscope, the types of images are the ultrasound image, the first optical image, and the second optical image; The processor identifies whether the ultrasonic endoscope is connected to the image processing device or whether the optical endoscope is connected to the image processing device based on the first identification information and / or the second identification information. The medical support device according to claim 1 .

8. The processor analyzes the image included in the first image information or the second image information to identify whether the ultrasonic endoscope is connected to the image processing device or whether the optical endoscope is connected to the image processing device. The medical support device according to claim 1 .

9. When the ultrasonic endoscope is connected to the image processing device, the processor does not output to the display device information obtained from the trained model for optical images by inputting the first optical image included in the first image information into the trained model for optical images. The medical support device according to claim 1 .

10. When the ultrasound image is displayed in a first display area of ​​the display device, the processor does not output, to the display device, information obtained from the trained model for optical images by inputting the first optical image included in the first image information into the trained model for optical images. The medical support device according to claim 1 .

11. The trained model for the optical image is a trained model obtained by performing machine learning for the second optical image. The medical support device according to claim 9.

12. The first recognition result includes first position information capable of identifying a position of the first feature region, first feature information capable of identifying a medical feature of the first feature region, and / or first site information capable of identifying a site within the body to be inspected by the optical endoscope. The medical support device according to claim 1 .

13. The trained model for the optical image includes a first trained model to which the second optical image is input, The first trained model generates the first position information based on the input second optical image. The medical support device according to claim 12.

14. When the ultrasound image is displayed in a first display area of ​​the display device, the processor outputs to the display device a second recognition result in which the ultrasound image included in the first image information is input to a trained model for ultrasound images, thereby causing the trained model for ultrasound images to recognize a second feature area appearing in the ultrasound image. The medical support device according to claim 1 .

15. The second recognition result includes second position information capable of identifying the position of the second feature region, second feature information capable of identifying medical features of the second feature region, and / or second site information capable of identifying a site within the body to be inspected by the ultrasonic endoscope. The medical support device according to claim 14.

16. The trained model for ultrasound images includes a second trained model to which the ultrasound image is input, The second trained model generates the second position information based on the input ultrasound image. The medical support device according to claim 15.

17. A medical support device according to any one of claims 1 to 16; an image processing device having a first image processing device and a second image processing device; the first image processing device generates the ultrasonic image based on a reflected wave of ultrasonic waves emitted from the ultrasonic endoscope; The second image processing device generates the first optical image based on a first image signal obtained by optically imaging the ultrasonic endoscope, and generates the second optical image based on a second image signal obtained by optically imaging the optical endoscope. Endoscopy system.

18. When the ultrasonic endoscope is connected to an image processing device to which an ultrasonic endoscope and an optical endoscope can be connected, first image information including a first optical image and an ultrasonic image generated by imaging using the ultrasonic endoscope is acquired; acquiring second image information including a second optical image generated by imaging using the optical endoscope when the optical endoscope is connected to the image processing device; outputting the first optical image and the ultrasonic image included in the first image information to a display device when the ultrasonic endoscope is connected to the image processing device; and When the optical endoscope is connected to the image processing device, outputting to the display device the second optical image included in the second image information and a first recognition result in which the trained model for optical images recognizes a first feature region appearing in the second optical image by inputting the second optical image into the trained model for optical images. Medical support methods.

Images