Ultrasonic endoscope system and working method of ultrasonic endoscope system

Abstract

In the ultrasonic endoscope system and the working method of the ultrasonic endoscope system of the present invention, the ultrasonic image recognition unit establishes a corresponding association between the position of the front end of the ultrasonic endoscope in the body cavity of the subject and the label number based on the observation order of the observation object part, and pre-learns the relationship between the learning ultrasonic images and the label numbers corresponding to the positions of the front end of the ultrasonic endoscope when the learning ultrasonic images are taken for multiple learning ultrasonic images, and recognizes the label numbers corresponding to the positions of the front end of the ultrasonic endoscope when the diagnostic ultrasonic images are taken based on the learning results, and the display control unit displays the position of the front end of the ultrasonic endoscope corresponding to the label number recognized by the ultrasonic image recognition unit on the display.

Description

Technical Field

[0001] The present invention relates to an ultrasonic endoscope system for observing the state of an observation target site in a subject's body using ultrasonic waves, and an operating method of the ultrasonic endoscope system. Background Art

[0002] The ultrasonic endoscope system is mainly used to observe the pancreas or gallbladder through the digestive tract. An ultrasonic endoscope having an endoscopic observation unit and an ultrasonic observation unit at the front end is inserted into the digestive tract of the subject, and endoscopic images of the inside of the digestive tract and ultrasonic images of the area outside the digestive tract wall are taken.

[0003] In the ultrasonic endoscope system, the illumination unit at the front end of the ultrasonic endoscope irradiates the adjacent part of the observation object in the digestive tract with illumination light, and the imaging unit at the front end of the ultrasonic endoscope receives the reflected light, and generates an endoscopic image based on the imaging signal of the reflected light. In addition, the plurality of ultrasonic transducers at the front end of the ultrasonic endoscope cause the organs and other observation object parts located outside the digestive tract wall to transmit and receive ultrasonic waves, and generates an ultrasonic image based on the received ultrasonic signals.

[0004] When observing the endoscopic image, if the ultrasonic endoscope is inserted into the digestive tract of the subject, for example, the inner wall of the esophagus can be seen. If the front end of the ultrasonic endoscope is further advanced, the inner wall of the stomach can be seen. In this way, the practicing physician (user of the ultrasonic endoscope system) can easily know where the front end of the ultrasonic endoscope is currently located in the subject's digestive organ, in which direction it is facing, and which part is being observed.

[0005] In contrast, it is extremely difficult for a medical practitioner who is not familiar with ultrasonic images to distinguish what is shown in an ultrasonic image, such as the pancreas or gallbladder, or a blood vessel, bile duct, or pancreatic duct. Furthermore, when observing an ultrasonic image, a medical practitioner who is not familiar with ultrasonic images finds it difficult to distinguish where the front end of the ultrasonic endoscope is currently located, in which direction it is facing, and which part is being observed, and thus may get lost in the body of the subject.

[0006] Here, as prior art documents related to the present invention, there are Patent Documents 1 to 4.

[0007] Patent document 1 describes the following: a target part in an image of a diagnostic part in a subject is roughly extracted, overall information for identifying the target part using a neural network is predicted, the contour of the target part is identified using the overall information, and the identification result is displayed together with the original image.

[0008] Patent document 2 describes the following: based on the electrical signal from the coil, position / orientation data of the front end portion of the ultrasonic endoscope are generated, and based on the position / orientation data, insertion shape data for representing the insertion shape of the ultrasonic endoscope are generated, and a guide image is generated by combining it with three-dimensional living tissue model data of the tissue structure of the subject's organ group, etc., and a video signal of a composite image combining the ultrasonic image and the guide image is generated and displayed on a display.

[0009] Furthermore, Patent Document 2 describes that a stereo guide image and a cross-sectional guide image are arranged and displayed in a left area of ​​a screen, and an ultrasonic image is arranged and displayed in a right area of ​​the screen.

[0010] Furthermore, Patent Document 2 describes a button for enlarging or reducing the display range of an ultrasonic image.

[0011] Patent Document 3 describes that an ultrasonic tomographic image of a subject and an optical image thereof are displayed adjacent to each other at one position on a screen of a display device so that both images can be observed simultaneously.

[0012] Patent Document 4 describes that an ultrasonic image and a schematic diagram are displayed on the same screen, the schematic diagram being a diagram or an actual optical image of a human body, and the scanning surface and insertion shape of an ultrasonic endoscope are also displayed in the schematic diagram.

[0013] Furthermore, Patent Document 4 discloses that the scanning position area of ​​the ultrasonic endoscope is detected from a signal of the position and direction of the ultrasonic endoscope detected using a coil and ultrasonic scanning area data is output, part name data corresponding to the ultrasonic scanning area data is read from a part name storage unit, and the part name is displayed superimposed on the ultrasonic image.

[0014] Previous technical literature

[0015] Patent Literature

[0016] Patent Document 1: Japanese Patent Application Laid-Open No. 06-233761

[0017] Patent Document 2: Japanese Patent Application Publication No. 2010-069018

[0018] Patent Document 3: Japanese Patent Application Laid-Open No. 02-045045

[0019] Patent Document 4: Japanese Patent Application Publication No. 2004-113629 Summary of the invention

[0020] Technical issues to be solved by the invention

[0021] The first object of the present invention is to provide an ultrasonic endoscope system and an operating method of the ultrasonic endoscope system that solve the problems of the above-mentioned prior art and enable even a practicing physician who is not familiar with ultrasonic images to reliably grasp the position and direction of the front end of the ultrasonic endoscope.

[0022] The second purpose of the present invention is, in addition to the above-mentioned first purpose, to provide an ultrasonic endoscope system and an operating method of the ultrasonic endoscope system, in which even a practicing physician who is not familiar with ultrasonic images will not get lost in the body of the subject and can accurately move the front end of the ultrasonic endoscope from the current observation object part to the next observation object part.

[0023] Means for solving technical problems

[0024] In order to achieve the above-mentioned object, the present invention provides an ultrasonic endoscope system, which comprises: an ultrasonic endoscope having an ultrasonic transducer at a front end;

[0025] An ultrasonic observation device that transmits and receives ultrasonic waves through an ultrasonic transducer and generates a diagnostic ultrasonic image based on the received ultrasonic signal;

[0026] an ultrasonic image recognition unit that establishes a corresponding association between the position of the front end portion of the ultrasonic endoscope in the body cavity of the subject and a label number based on the observation order of the observation object part, and pre-learns the relationship between the learning ultrasonic images and the label numbers corresponding to the positions of the front end portion of the ultrasonic endoscope when the learning ultrasonic images were taken for a plurality of learning ultrasonic images, and recognizes the label number corresponding to the position of the front end portion of the ultrasonic endoscope when the diagnostic ultrasonic image was taken from the diagnostic ultrasonic image based on the learning result; and

[0027] The display control unit displays the position of the distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit on the display.

[0028] Here, it is preferable that the display control unit displays the name of the observation target site corresponding to the recognized tag number on the display as character information.

[0029] Furthermore, it is preferable that the display control unit displays, on the display, an anatomical illustration in which the position of the distal end portion of the ultrasonic endoscope corresponding to the recognized tag number is superimposed and displayed as image information.

[0030] Furthermore, it is preferred that the ultrasonic image recognition unit establishes a correspondence between the position and orientation of the front end portion of the ultrasonic endoscope and the label number, and pre-learns the relationship between the learning ultrasonic images and the label numbers corresponding to the position and orientation of the front end portion of the ultrasonic endoscope when the learning ultrasonic images were captured for a plurality of learning ultrasonic images, and recognizes the label number corresponding to the position and orientation of the front end portion of the ultrasonic endoscope when the diagnostic ultrasonic image was captured based on the learning result from the diagnostic ultrasonic image.

[0031] The display control unit displays the position and orientation of the distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit on the display.

[0032] Furthermore, it is preferable that the display control unit displays the name of the observation target site corresponding to the recognized tag number on the display as character information.

[0033] Furthermore, it is preferable that the display control unit displays, on the display, an anatomical illustration in which the position and orientation of the distal end portion of the ultrasonic endoscope corresponding to the recognized tag number are superimposed and displayed as image information.

[0034] Preferably, the apparatus further comprises an operation process storage unit for storing an operation process for moving the front end portion of the ultrasonic endoscope from an observation target portion corresponding to one label number to an observation target portion corresponding to a label number next to the one label number in the observation order.

[0035] The display control unit obtains the operation process for moving the front end of the ultrasonic endoscope from the observation target part corresponding to the current label number to the observation target part corresponding to the next label number in the observation order from the current label number from the operation process storage unit, and displays the obtained operation process on the display.

[0036] Preferably, the display control unit further displays, on the display as character information, a name of an observation target site corresponding to a tag number next to the current tag number in the observation order.

[0037] Furthermore, it is preferable that the display control unit displays the operation process on the display as character information.

[0038] Furthermore, it is preferable that the operation process includes the names of one or more organs that are depicted when the distal end portion of the ultrasonic endoscope is moved.

[0039] Furthermore, it is preferable that the display control unit displays the anatomical illustration on the display as image information with the operation process superimposed thereon.

[0040] Furthermore, it is preferable that the display control unit colors the regions of the one or more organs depicted when the distal end portion of the ultrasonic endoscope is moved on the anatomical illustration, and displays the anatomical illustration with the colored regions on the display.

[0041] Furthermore, it is preferred that the display control unit colors the area of ​​the observation object part corresponding to the label number next to the current label number in the observation order and the area of ​​one or more organs depicted when the front end portion of the ultrasonic endoscope is moved in different colors on the anatomical illustration, and displays the anatomical illustration with the areas colored in different colors on the display.

[0042] Preferably, there is further provided a warning generating unit for generating a warning when the distal end portion of the ultrasonic endoscope moves from the observation target site corresponding to the current tag number to an observation target site other than the observation target site corresponding to the tag number next to the current tag number in the observation order.

[0043] Furthermore, it is preferable that the warning issuing unit issues the warning as sound information, or issues both character information and sound information as the warning.

[0044] Furthermore, preferably, the display control unit assigns a check mark to the label number corresponding to the arrived observation target part each time the distal end portion of the ultrasonic endoscope reaches the observation target part corresponding to each label number, and displays the label number assigned the check mark as character information on the display.

[0045] Furthermore, it is preferred that the display control unit colors the area of ​​the reached observation object part on the anatomical illustration each time the front end of the ultrasonic endoscope reaches the observation object part corresponding to each label number, and displays the colored anatomical illustration of the reached observation object part on the display.

[0046] Furthermore, preferably, the display control unit emphasizes a region of the observation target part corresponding to a label number next to the current label number in observation order on the anatomical illustration, and displays the anatomical illustration with the emphasized region of the observation target part on the display.

[0047] Furthermore, it is preferred that the display control unit colors the area of ​​the observation object part corresponding to the label number next to the current label number in the observation order on the anatomical illustration with a color different from the color of the areas of the observation object part other than the observation object part corresponding to the label number next to the current label number in the observation order, and displays the anatomical illustration with the colored areas of the observation object part on the display.

[0048] Furthermore, the display control unit preferably arranges the movement path of the ultrasonic endoscope's distal end portion when it ideally moves according to the observation order of the observation target parts and the movement path of the ultrasonic endoscope's distal end portion when it actually moves according to actual operation as image information on the anatomical diagram and displays them on the display.

[0049] Preferably, the apparatus further includes a movement path registration unit that registers in advance a movement path when the distal end portion of the ultrasonic endoscope is ideally moved according to the observation order of the observation target site.

[0050] Furthermore, it is preferable that the ultrasonic image recognition unit is built in the ultrasonic observation device.

[0051] Preferably, the ultrasonic endoscope further includes a lighting unit and an imaging unit at the front end.

[0052] The apparatus further comprises an endoscope processor for receiving reflected light of the illumination light emitted from the illumination unit by the imaging unit and generating a diagnostic endoscopic image based on an imaging signal of the reflected light.

[0053] The ultrasonic image recognition unit is built into the endoscope processor.

[0054] Preferably, the ultrasonic endoscope further includes a lighting unit and an imaging unit at the front end.

[0055] The apparatus further comprises an endoscope processor for receiving reflected light of the illumination light emitted from the illumination unit by the imaging unit and generating a diagnostic endoscopic image based on an imaging signal of the reflected light.

[0056] The ultrasonic image recognition unit is provided outside the ultrasonic observation device and the endoscope processor.

[0057] Furthermore, the present invention provides an operating method of an ultrasonic endoscope system, comprising the following steps: an ultrasonic image recognition unit establishes a corresponding association between the position of the front end portion of the ultrasonic endoscope in the body cavity of the subject and a label number based on the observation order of the observation object part, and pre-learns the relationship between the learning ultrasonic images and the label number corresponding to the position of the front end portion of the ultrasonic endoscope when the learning ultrasonic images were captured for a plurality of learning ultrasonic images;

[0058] The ultrasonic observation device transmits and receives ultrasonic waves by means of an ultrasonic transducer provided at the distal end of the ultrasonic endoscope, and generates a diagnostic ultrasonic image based on the received ultrasonic signal;

[0059] The ultrasonic image recognition unit recognizes, from the diagnostic ultrasonic image based on the learning result, a label number corresponding to the position of the distal end portion of the ultrasonic endoscope when the diagnostic ultrasonic image was captured; and

[0060] A step of causing the display control unit to display, on a display, the position of the distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit.

[0061] Here, it is preferred to establish a correspondence between the position and orientation of the front end portion of the ultrasonic endoscope and the label number, and to pre-learn the relationship between the learning ultrasonic images and the label number corresponding to the position and orientation of the front end portion of the ultrasonic endoscope when the learning ultrasonic images were captured for a plurality of learning ultrasonic images.

[0062] Based on the learning result, the label number corresponding to the position and orientation of the front end of the ultrasonic endoscope when the ultrasonic image for diagnosis is recognized and photographed is used.

[0063] The position and orientation of the distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit are displayed on the display.

[0064] Preferably, the method further includes a step in which the operation process storage unit stores an operation process for moving the front end portion of the ultrasonic endoscope from an observation target site corresponding to one tag number to an observation target site corresponding to a tag number next to the one tag number in observation order,

[0065] The identified tag number is used as the current tag number, and the operation process for moving the front end of the ultrasonic endoscope from the observation target part corresponding to the current tag number to the observation target part corresponding to the tag number next to the current tag number in the observation order is obtained from the operation process storage unit.

[0066] The acquired operation process is displayed on the display.

[0067] Furthermore, the ultrasonic image recognition unit, the display control unit, and the warning generation unit are preferably hardware or a processor that executes a program, and the operation process storage unit and the movement path registration unit are preferably hardware or a memory.

[0068] Effects of the Invention

[0069] In the present invention, the position and orientation of the front end of the ultrasonic endoscope are displayed on the display. Thus, according to the present invention, for example, even a practicing physician who is not familiar with ultrasonic images can reliably understand where the front end of the ultrasonic endoscope is currently located, in which direction it is facing, and which part is being observed.

[0070] Furthermore, in the present invention, the operation process for moving the front end of the ultrasonic endoscope can be displayed on the display. Thus, according to the present invention, for example, even a practicing physician who is not familiar with ultrasonic images can accurately move the front end of the ultrasonic endoscope from the current observation target site to the next observation target site without getting lost in the body of the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a diagram showing a schematic configuration of an ultrasonic endoscope system according to an embodiment of the present invention.

[0072] Figure 2 It is a plan view showing the distal end portion of the insertion portion of the ultrasonic endoscope and its surroundings.

[0073] Figure 3 It means use Figure 2 The II cross section shown in the figure is a cross-sectional view when the distal end portion of the insertion portion of the ultrasonic endoscope is cut.

[0074] Figure 4 This is a block diagram showing one embodiment of the structure of an endoscope image recognition unit.

[0075] Figure 5 It is a block diagram showing the structure of an ultrasonic observation apparatus.

[0076] Figure 6 This is a block diagram showing an embodiment of the configuration of an ultrasonic image recognition unit.

[0077] Figure 7 This is a diagram showing the flow of diagnostic processing using the ultrasonic endoscope system.

[0078] Figure 8 This is a diagram showing the process of diagnosis steps in the diagnosis process.

[0079] Fig. 9 This is a conceptual diagram showing an embodiment of an endoscopic image, an ultrasonic image, and an anatomical illustration.

[0080] Fig.10 This is a conceptual diagram showing an embodiment of label numbers to which check marks are given.

[0081] Fig.11 This is a block diagram showing an embodiment of the configuration of an ultrasonic endoscope system in which an ultrasonic image recognition unit is built into an ultrasonic observation apparatus.

[0082] Fig.12 This is a block diagram showing an embodiment of the configuration of an ultrasonic endoscope system in which an ultrasonic image recognition unit is built into an endoscope processor.

[0083] Fig.13 This is a block diagram showing an embodiment of the configuration of an ultrasonic endoscope system in which an ultrasonic image recognition unit is provided outside an ultrasonic observation apparatus and an endoscope processor. DETAILED DESCRIPTION

[0084] An ultrasonic endoscope system according to one embodiment (this embodiment) of the present invention will be described in detail with reference to preferred embodiments shown in the drawings.

[0085] In addition, the present embodiment is a representative embodiment of the present invention, but is merely an example and does not limit the present invention.

[0086] <<Overview of Ultrasonic Endoscope System>>

[0087] Regarding the ultrasonic endoscope system 10 according to this embodiment, refer to Figure 1 The outline thereof will be described. Figure 1 1 is a diagram showing a schematic configuration of the ultrasonic endoscope system 10 .

[0088] The ultrasonic endoscope system 10 is used to observe the state of the observation target part in the patient's body using ultrasonic waves (hereinafter also referred to as ultrasonic diagnosis). Here, the observation target part is a part that is difficult to examine from the patient's body surface, such as the pancreas or gallbladder. By using the ultrasonic endoscope system 10, the state of the observation target part and the presence or absence of abnormalities can be ultrasonically diagnosed through the patient's body cavity, that is, the digestive tract such as the esophagus, stomach, duodenum, small intestine and large intestine.

[0089] The ultrasonic endoscope system 10 acquires ultrasonic images and endoscope images, such as Figure 1 As shown, the ultrasonic endoscope 12 , the ultrasonic observation device 14 , the endoscope processor 16 , the light source device 18 , the display 20 , the water supply tank 21 a , the suction pump 21 b , and the operation console 100 are provided.

[0090] The ultrasonic endoscope 12 includes an insertion portion 22 inserted into a patient's body cavity, an operation portion 24 operated by a practitioner (user) such as a doctor or a technician, and an ultrasonic transducer unit 46 (see FIG. 1 ) mounted on a distal end portion 40 of the insertion portion 22. Figure 2 and Figure 3 The ultrasonic endoscope 12 has a plurality of ultrasonic transducers 48 provided by an ultrasonic transducer unit 46 at the front end as the ultrasonic observation unit 36 ​​(see Figure 2 and Figure 3 ). In addition, the ultrasonic endoscope 12 has, as an endoscope observation unit 38, an illumination unit including an illumination window 88 and the like at the front end, and an imaging unit including an observation window 82, an objective lens 84, and a solid imaging element 86 and the like (refer to Figure 2 and Figure 3 ). The practicing physician obtains endoscopic images and ultrasonic images through the function of the ultrasonic endoscope 12.

[0091] Here, an "endoscopic image" is an image obtained by optically photographing the inner wall of a patient's body cavity, and an "ultrasound image" is an image obtained by receiving the reflected wave (echo) of ultrasound transmitted from the patient's body cavity toward the observation target site and imaging the received signal.

[0092] In addition, the ultrasonic endoscope 12 will be described in detail later.

[0093] The ultrasonic observation device 14 is connected to the ultrasonic endoscope 12 via the universal cord 26 and the ultrasonic connector 32a provided at the end thereof. The ultrasonic observation device 14 controls the ultrasonic transducer unit 46 of the ultrasonic endoscope 12 and transmits ultrasonic waves. Furthermore, the ultrasonic observation device 14 generates an ultrasonic image by imaging the reception signal when the ultrasonic transducer unit 46 receives the reflected wave (echo) of the transmitted ultrasonic wave. In other words, the ultrasonic observation device 14 generates a diagnostic ultrasonic image (hereinafter also referred to as an ultrasonic image) based on the ultrasonic reception signal by transmitting and receiving ultrasonic waves through the plurality of ultrasonic transducers 48 provided in the ultrasonic transducer unit 46.

[0094] In addition, the ultrasonic observation device 14 will be described in detail later.

[0095] The endoscope processor 16 is connected to the ultrasonic endoscope 12 via the universal cord 26 and the endoscope connector 32b provided at the end thereof. The endoscope processor 16 acquires image data of a portion adjacent to an observation object captured by the ultrasonic endoscope 12 (specifically, the solid imaging element 86 described later), and performs a predetermined image processing on the acquired image data to generate an endoscopic image. In other words, the endoscope processor 16 receives the reflected light of the illumination light irradiated from the illumination unit provided at the front end of the ultrasonic endoscope 12 through the imaging unit provided at the front end of the ultrasonic endoscope 12, and generates a diagnostic endoscopic image (hereinafter also referred to as an endoscopic image) based on the imaging signal of the reflected light.

[0096] Here, the “observation target adjacent site” refers to a portion located adjacent to the observation target portion in the inner wall of the patient's body cavity.

[0097] In the present embodiment, the ultrasonic observation device 14 and the endoscope processor 16 are configured by two separate devices (computers), but the present invention is not limited thereto, and both the ultrasonic observation device 14 and the endoscope processor 16 may be configured by one device.

[0098] The light source device 18 is connected to the ultrasonic endoscope 12 via the universal cord 26 and the light source connector 32c provided at the end thereof. When the ultrasonic endoscope 12 is used to photograph the adjacent part of the observation object, the light source device 18 irradiates white light composed of three primary colors of red light, green light, and blue light or light of a specific wavelength. The light irradiated by the light source device 18 propagates in the ultrasonic endoscope 12 through the light guide (not shown) included in the universal cord 26, and is emitted from the ultrasonic endoscope 12 (specifically, the illumination window 88 described later). As a result, the adjacent part of the observation object is irradiated with the light from the light source device 18.

[0099] The display 20 is connected to the ultrasonic observation device 14 and the endoscope processor 16 , and displays ultrasonic images generated by the ultrasonic observation device 14 , endoscopic images generated by the endoscope processor 16 , and other anatomical illustrations.

[0100] As a display method of the ultrasonic image and the endoscopic image, one image may be switched to one of the other images and displayed on the display 20, or two or more images may be displayed in parallel at the same time.

[0101] In addition, in the present embodiment, the ultrasonic image and the endoscopic image are displayed on a single display 20, but a display for displaying ultrasonic images, a display for displaying endoscopic images, and a display for displaying anatomical diagrams may be provided separately. Furthermore, the ultrasonic image and the endoscopic image may be displayed in a display other than the display 20, for example, on a display of a terminal carried by a practicing physician.

[0102] The operation console 100 is an example of a command acquisition unit that acquires commands input from a practicing physician (user), and is a device provided for the practicing physician to input necessary information when performing ultrasonic diagnosis, or to issue a command to start ultrasonic diagnosis to the ultrasonic observation device 14. The operation console 100 is composed of, for example, a keyboard, a mouse, a trackball, a touch pad, and a touch panel. When the operation console 100 is operated, the CPU (control circuit) 152 (refer to the control circuit 152 of the ultrasonic observation device 14) is controlled according to the operation content. Figure 5 ) control device components (for example, the receiving circuit 142 and the transmitting circuit 144 described later).

[0103] Specifically, before starting the ultrasonic diagnosis, the practicing physician inputs the examination information (for example, examination sequence information including the year, month, day and sequence number, and patient information including the patient ID and patient name, etc.) through the operation console 100. After completing the input of the examination information, if the practicing physician issues a command to start the ultrasonic diagnosis through the operation console 100, the CPU 152 of the ultrasonic observation device 14 controls each part of the ultrasonic observation device 14 in such a way that the ultrasonic diagnosis is performed according to the input examination information.

[0104] Furthermore, when performing ultrasonic diagnosis, the medical practitioner can set various control parameters through the console 100. Examples of the control parameters include the selection result of the live mode and the freeze mode, the setting value of the display depth (depth), and the selection result of the ultrasonic image generation mode.

[0105] Here, the "live mode" is a mode for sequentially displaying (displaying in real time) ultrasonic images (moving images) acquired at a predetermined frame rate. The "freeze mode" is a mode for reading a single frame image (still image) of an ultrasonic image (moving image) generated in the past from the cine memory 150 described later and displaying it.

[0106] In the present embodiment, there are a plurality of selectable ultrasonic image generation modes, specifically, B (Brightness) mode, CF (Color Flow) mode, and PW (Pulse Wave) mode. The B mode is a mode for converting the amplitude of ultrasonic echoes into brightness to display a tomographic image. The CF mode is a mode for mapping the average blood flow velocity, blood flow fluctuation, the intensity of blood flow signals, or blood flow dynamics into various colors and displaying them superimposed on the B mode image. The PW mode is a mode for displaying the velocity of the ultrasonic echo source (e.g., the velocity of the blood flow) detected by the transmission and reception of the pulse wave.

[0107] The above-mentioned ultrasonic image generation modes are merely examples, and modes other than the above-mentioned three modes, such as an A (Amplitude) mode, an M (Motion) mode, and a contrast enhancement mode, may be included.

[0108] <<Structure of Ultrasonic Endoscope 12>>

[0109] Next, refer to the Figure 1 and Figure 2 , Figure 3 and Figure 5 The structure of the ultrasonic endoscope 12 will be described. Figure 2 It is a plan view showing an enlarged view of the distal end portion of the insertion portion 22 of the ultrasonic endoscope 12 and its surroundings. Figure 3 It means use Figure 2 The illustrated diagram is a cross-sectional view of a case where the distal end portion 40 of the insertion portion 22 of the ultrasonic endoscope 12 is cut along the II cross section. Figure 5 It is a block diagram showing the configuration of the ultrasonic observation device 14 .

[0110] As described above, the ultrasonic endoscope 12 includes the insertion portion 22 and the operation portion 24. Figure 1 As shown in FIG. 2 , the insertion portion 22 includes a front end portion 40, a bending portion 42, and a soft portion 43 in order from the front end side (free end side). Figure 2 As shown in FIG. 4 , an ultrasonic observation unit 36 ​​and an endoscope observation unit 38 are provided at the distal end portion 40. Figure 3 As shown, an ultrasound transducer unit 46 including a plurality of ultrasound transducers 48 is disposed in the ultrasound observation section 36 .

[0111] And, if Figure 2 As shown, the front end portion 40 is provided with a treatment tool outlet 44. The treatment tool outlet 44 serves as an outlet for treatment tools (not shown) such as forceps, puncture needles or high-frequency knives. In addition, the treatment tool outlet 44 also serves as a suction port when suctioning blood, body waste and other suction objects.

[0112] The bending portion 42 is a bendable portion connected to the proximal side (the side opposite to the side where the ultrasonic transducer unit 46 is provided) of the distal end portion 40. The soft portion 43 is a portion connecting the bending portion 42 and the operation portion 24, is flexible, and is provided in an elongated state.

[0113] A plurality of air and water supply conduits and a plurality of suction conduits are formed inside the insertion portion 22 and the operation portion 24. Also, a treatment instrument channel 45 having one end connected to a treatment instrument outlet 44 is formed inside the insertion portion 22 and the operation portion 24.

[0114] Next, among the structural elements of the ultrasonic endoscope 12 , the ultrasonic observation section 36 , the endoscope observation section 38 , the water supply tank 21 a , the suction pump 21 b , and the operation section 24 will be described in detail.

[0115] (Ultrasonic Observation Section 36)

[0116] The ultrasonic observation section 36 is a portion provided for acquiring ultrasonic images, and is disposed on the distal end side of the distal end portion 40 of the insertion section 22. Figure 3 As shown, the ultrasonic observation section 36 includes an ultrasonic transducer unit 46 , a plurality of coaxial cables 56 , and an FPC (Flexible Printed Circuit) 60 .

[0117] The ultrasonic transducer unit 46 is equivalent to an ultrasonic probe (probe), and transmits ultrasonic waves in the patient's body cavity using an ultrasonic transducer array 50 in which a plurality of ultrasonic transducers 48 described later are arranged, and receives reflected waves (echoes) of ultrasonic waves reflected at the observation target site and outputs a reception signal. The ultrasonic transducer unit 46 involved in this embodiment is convex and transmits ultrasonic waves in a radial (arc) shape. However, the type (type) of the ultrasonic transducer unit 46 is not particularly limited to this, and as long as it can transmit and receive ultrasonic waves, it can be of other types, such as radial type and linear type.

[0118] like Figure 3 As shown, the ultrasonic transducer unit 46 is constituted by laminating a backing material layer 54 , an ultrasonic transducer array 50 , an acoustic matching layer 74 , and an acoustic lens 76 .

[0119] The ultrasonic vibrator array 50 is composed of a plurality of ultrasonic vibrators 48 (ultrasonic transducers) arranged in a one-dimensional array. More specifically, the ultrasonic vibrator array 50 is composed of N (for example, N=128) ultrasonic vibrators 48 arranged at equal intervals in a convex curved shape along the axial direction of the distal end portion 40 (the longitudinal direction of the insertion portion 22). In addition, the ultrasonic vibrator array 50 may also be configured by arranging a plurality of ultrasonic vibrators 48 in a two-dimensional array.

[0120] Each of the N ultrasonic vibrators 48 is formed by placing electrodes on both surfaces of a piezoelectric element (piezoelectric body). As the piezoelectric element, barium titanate (BaTiO3), lead zirconate titanate (PZT), potassium niobate (KNbO3) and the like can be used.

[0121] The electrode is composed of an individual electrode (not shown) provided for each of the plurality of ultrasonic vibrators 48 and a vibrator ground electrode (not shown) common to the plurality of ultrasonic vibrators 48. The electrode is electrically connected to the ultrasonic observation device 14 via a coaxial cable 56 and an FPC 60.

[0122] A pulsed driving voltage is supplied from the ultrasonic observation device 14 to each ultrasonic vibrator 48 via a coaxial cable 56 as an input signal (transmission signal). When the driving voltage is applied to the electrode of the ultrasonic vibrator 48, the piezoelectric element expands and contracts, and the ultrasonic vibrator 48 is driven (vibrated). As a result, a pulsed ultrasonic wave is output from the ultrasonic vibrator 48. At this time, the amplitude of the ultrasonic wave output from the ultrasonic vibrator 48 becomes a size corresponding to the intensity (output intensity) when the ultrasonic vibrator 48 outputs the ultrasonic wave. Here, the output intensity is defined as the size of the sound pressure of the ultrasonic wave output from the ultrasonic vibrator 48.

[0123] Then, when the reflected wave (echo) of the ultrasonic wave is received, each ultrasonic vibrator 48 vibrates (drives) accordingly, and the piezoelectric element of each ultrasonic vibrator 48 generates an electrical signal. The electrical signal is output from each ultrasonic vibrator 48 toward the ultrasonic observation device 14 as an ultrasonic reception signal. At this time, the magnitude (voltage value) of the electrical signal output from the ultrasonic vibrator 48 becomes a magnitude corresponding to the reception sensitivity when the ultrasonic vibrator 48 receives the ultrasonic wave. Here, the reception sensitivity is defined as the ratio of the amplitude of the electrical signal of the ultrasonic wave received and output by the ultrasonic vibrator 48 to the amplitude of the ultrasonic wave transmitted by the ultrasonic vibrator 48.

[0124] In this embodiment, the multiplexer 140 (refer to Figure 5) or other electronic switches sequentially drive the N ultrasonic vibrators 48, thereby scanning the ultrasonic wave in a scanning range along the curved surface on which the ultrasonic vibrator array 50 is arranged, for example, a range of about tens of mm from the center of curvature of the curved surface. More specifically, when a B-mode image (tomographic image) is acquired as an ultrasonic image, a driving voltage is supplied to m (for example, m=N / 2) ultrasonic vibrators 48 (hereinafter referred to as driven vibrators) arranged continuously among the N ultrasonic vibrators 48 by selecting the open channel of the multiplexer 140. As a result, the m driven vibrators are driven, and ultrasonic waves are output from each of the driven vibrators in the open channel. The ultrasonic waves output from the m driven vibrators are immediately synthesized, and the synthesized wave (ultrasonic beam) is transmitted toward the observation target site. Then, each of the m driven vibrators receives the ultrasonic wave (echo) reflected at the observation target site, and outputs an electrical signal (received signal) corresponding to the receiving sensitivity at that point in time.

[0125] Furthermore, the above series of processes (i.e., supply of driving voltage, reception and transmission of ultrasonic waves, and output of electrical signals) are repeated while shifting the position of the driven vibrator among the N ultrasonic vibrators 48 by one at a time (shifting one ultrasonic vibrator 48 at a time). Specifically, the above series of processes are started from the m driven vibrators on both sides of the ultrasonic vibrator 48 located at one end as the center among the N ultrasonic vibrators 48. Furthermore, the above series of processes are repeated each time the position of the driven vibrator is shifted by switching the opening channel by the multiplexer 140. Finally, the above series of processes are repeated N times in total until the m driven vibrators on both sides of the ultrasonic vibrator 48 located at the other end as the center among the N ultrasonic vibrators 48 are reached.

[0126] The backing material layer 54 supports each ultrasonic transducer 48 of the ultrasonic transducer array 50 from the back side. The backing material layer 54 has the function of attenuating the ultrasonic wave propagating to the backing material layer 54 side among the ultrasonic wave emitted from the ultrasonic transducer 48 or the ultrasonic wave (echo) reflected by the observation target part. The backing material is made of a rigid material such as hard rubber, and an ultrasonic wave attenuating material (ferrite, ceramic, etc.) is added as needed.

[0127] The acoustic matching layer 74 is superimposed on the ultrasonic transducer array 50 and is provided to obtain acoustic impedance matching between the patient's human body and the ultrasonic transducer 48. The acoustic matching layer 74 is provided to improve the transmittance of ultrasonic waves. As the material of the acoustic matching layer 74, an organic material having an acoustic impedance value closer to the acoustic impedance value of the patient's human body than the piezoelectric element of the ultrasonic transducer 48 can be used. Specifically, as the material of the acoustic matching layer 74, epoxy resin, silicone rubber, polyimide, polyethylene, etc. can be cited.

[0128] The acoustic lens 76 superimposed on the acoustic matching layer 74 converges the ultrasonic waves emitted from the ultrasonic transducer array 50 toward the observation target site. The acoustic lens 76 is composed of, for example, a silicone resin (kneading type silicone rubber (HTV rubber), liquid silicone rubber (RTV rubber) or the like), a butadiene resin or a polyurethane resin, and is mixed with powders such as titanium oxide, aluminum oxide or silicon dioxide as needed.

[0129] The PPC 60 is electrically connected to the electrodes provided in each ultrasonic vibrator 48. Each of the plurality of coaxial cables 56 is wired to the FPC 60 at one end thereof. Furthermore, when the ultrasonic endoscope 12 is connected to the ultrasonic observation device 14 via the ultrasonic connector 32a, each of the plurality of coaxial cables 56 is electrically connected to the ultrasonic observation device 14 at the other end thereof (the side opposite to the PPC 60 side).

[0130] (Endoscope Observation Section 38)

[0131] The endoscopic observation section 38 is a portion provided for acquiring an endoscopic image, and is disposed at a position closer to the proximal end than the ultrasonic observation section 36 in the distal end portion 40 of the insertion section 22. Figure 2 and Figure 3 As shown, the endoscope observation section 38 is composed of an observation window 82, an objective lens 84, a solid-state imaging element 86, an illumination window 88, a cleaning nozzle 90, a wiring cable 92, and the like.

[0132] The observation window 82 is installed in the distal end portion 40 of the insertion portion 22 in a tilted state relative to the axial direction (the longitudinal axis direction of the insertion portion 22). Light reflected from the observation window 82 near the observation object is imaged on the imaging surface of the solid imaging element 86 by the objective lens 84.

[0133] The solid imaging element 86 performs photoelectric conversion on the reflected light of the observation object adjacent part that is imaged on the imaging surface through the observation window 82 and the objective lens 84, and outputs an imaging signal. As the solid imaging element 86, a CCD (charge coupled device) and a CMOS (Complementary Metal Oxide Semiconductor) can be used. The image signal output by the solid imaging element 86 is transmitted to the endoscope processor 16 through the wiring cable 92 extending from the insertion part 22 to the operation part 24 through the universal cord 26.

[0134] The illumination windows 88 are provided at both sides of the observation window 82. The emission end of the light guide (not shown) is connected to the illumination window 88. The light guide extends from the insertion portion 22 to the operation portion 24, and its incident end is connected to the light source device 18 connected via the universal cord 26. The illumination light emitted by the light source device 18 propagates in the light guide and shines from the illumination window 88 toward the adjacent portion of the observation object.

[0135] The cleaning nozzle 90 is a discharge hole formed in the front end portion 40 of the insertion portion 22 in order to clean the surface of the observation window 82 and the lighting window 88, and air or cleaning liquid is sprayed from the cleaning nozzle 90 toward the observation window 82 and the lighting window 88. In addition, in the present embodiment, the cleaning liquid sprayed from the cleaning nozzle 90 is water, in particular, degassed water. However, there is no particular limitation on the cleaning liquid, and it may be other liquids, such as ordinary water (non-degassed water).

[0136] (Water delivery tank 21a and suction pump 21b)

[0137] The water supply tank 21a is a tank for storing deaerated water, and is connected to the light source connector 32c via an air and water supply hose 34a. In addition, the deaerated water can be used as the cleaning liquid sprayed from the cleaning nozzle 90.

[0138] The suction pump 21b sucks the suction material (including degassed water supplied for washing) in the body cavity through the treatment instrument outlet 44. The suction pump 21b is connected to the light source connector 32c through the suction hose 34b. In addition, the ultrasonic endoscope system 10 may also include an air supply pump that supplies air to a predetermined air supply destination.

[0139] A treatment instrument channel 45 and air and water supply conduits (not shown) are provided in the insertion portion 22 and the operation portion 24 .

[0140] The treatment instrument channel 45 communicates between the treatment instrument insertion port 30 provided in the operation portion 24 and the treatment instrument outlet 44. The treatment instrument channel 45 is connected to the suction button 28b provided in the operation portion 24. The suction button 28b is connected to the suction pump 21b in addition to the treatment instrument channel 45.

[0141] The air and water supply conduit communicates with the cleaning nozzle 90 at one end thereof and is connected at the other end to an air and water supply button 28a provided on the operation unit 24. The air and water supply button 28a is connected to the water supply tank 21a in addition to the air and water supply conduit.

[0142] (Operation unit 24)

[0143] The operation unit 24 is a part operated by the practicing physician when starting, during, or ending the ultrasonic diagnosis, and one end of the operation unit 24 is connected to one end of the universal cord 26. Figure 1As shown, the operation portion 24 includes an air and water supply button 28a, a suction button 28b, a pair of angle buttons 29, and a treatment instrument insertion port (forceps port) 30.

[0144] When each of the pair of angle knobs 29 is rotated, the bending portion 42 is remotely operated to bend and deform. By this deformation operation, the distal end portion 40 of the insertion portion 22 provided in the ultrasonic observation portion 36 and the endoscope observation portion 38 can be directed to a desired direction.

[0145] The treatment instrument insertion port 130 is a hole formed for inserting a treatment instrument (not shown) such as forceps, and communicates with the treatment instrument outlet 44 via the treatment instrument channel 45. The treatment instrument inserted into the treatment instrument insertion port 130 passes through the treatment instrument channel 45 and then is introduced into the body cavity from the treatment instrument outlet 44.

[0146] The air and water supply button 28 a and the suction button 28 b are two-stage switch buttons, and are operated to switch the opening and closing of the pipelines provided inside the insertion portion 22 and the operation portion 24 , respectively.

[0147] <<Schematic Configuration of Endoscope Processor 16>>

[0148] Here, the detailed configuration of the endoscope processor 16 is omitted, but the endoscope processor 16 includes an endoscope image recognition unit 170 in addition to conventionally known general components for capturing an endoscope image.

[0149] The endoscopic image recognition unit 170 pre-learns the relationship between the learning endoscopic images and the lesion areas displayed on the learning endoscopic images for a plurality of learning endoscopic images, and recognizes the lesion areas displayed on the diagnostic endoscopic images from the diagnostic endoscopic images generated by the endoscope processor 16 based on the learning results.

[0150] The learning endoscopic image is an existing endoscopic image used by the endoscopic image recognition unit 170 to learn the relationship between the endoscopic image and the lesion area displayed in the endoscopic image, and for example, various endoscopic images captured in the past can be used.

[0151] like Figure 4 As shown, the endoscopic image recognition unit 170 includes a lesion region detection unit 102 , a position information acquisition unit 104 , a selection unit 106 , and a lesion region detection control unit 108 .

[0152] The lesion region detection unit 102 detects the lesion region from the diagnostic endoscopic image based on the learning result. The lesion region detection unit 102 includes a plurality of detection units corresponding to a plurality of positions in the body cavity. Figure 4As shown, the apparatus includes first to eleventh detection sections 102A to 102K. The first detection section 102A corresponds to the rectum, the second detection section 102B corresponds to the sigmoid colon, the third detection section 102C corresponds to the descending colon, the fourth detection section 102D corresponds to the transverse colon, the fifth detection section 102E corresponds to the ascending colon, the sixth detection section 102F corresponds to the cecum, the seventh detection section 102G corresponds to the ileum, the eighth detection section 102H corresponds to the jejunum, the ninth detection section 102I corresponds to the duodenum, the tenth detection section 102J corresponds to the stomach, and the eleventh detection section 102K corresponds to the esophagus.

[0153] The first to eleventh detection units 102A to 102K are respectively learned models. The plurality of learned models are respectively models that have been learned using data sets consisting of different learning endoscopic images. Specifically, the plurality of learned models are models that have previously learned the relationship between the learning endoscopic images and the lesion areas displayed in the learning endoscopic images using data sets consisting of learning endoscopic images taken at different positions in the body cavity.

[0154] That is, the first detection unit 102A is a model that uses a data set consisting of learning endoscopic images of the rectum for learning, the second detection unit 102B is a model that uses a data set consisting of learning endoscopic images of the sigmoid colon for learning, the third detection unit 102C is a model that uses a data set consisting of learning endoscopic images of the descending colon for learning, the fourth detection unit 102D is a model that uses a data set consisting of learning endoscopic images of the transverse colon for learning, the fifth detection unit 102E is a model that uses a data set consisting of learning endoscopic images of the ascending colon for learning, and the sixth detection unit 102F is a model that uses a data set consisting of learning endoscopic images of the cecum for learning. The seventh detection unit 102G is a model that learns using a data set consisting of learning endoscopic images of the ileum, the eighth detection unit 102H is a model that learns using a data set consisting of learning endoscopic images of the jejunum, the ninth detection unit 102I is a model that learns using a data set consisting of learning endoscopic images of the duodenum, the tenth detection unit 102J is a model that learns using a data set consisting of learning endoscopic images of the stomach, and the eleventh detection unit 102K is a model that learns using a data set consisting of learning endoscopic images of the esophagus.

[0155] The learning method is not particularly limited as long as it is a method that can learn the relationship between the endoscopic image and the lesion area from a plurality of endoscopic images for learning and generate a learned model.

[0156] As a learning method, for example, deep learning using a hierarchical neural network, which is an example of machine learning, which is one of artificial intelligence (AI) technologies, can be used.

[0157] In addition, machine learning other than deep learning can be utilized, artificial intelligence techniques other than machine learning can be utilized, and learning methods other than artificial intelligence techniques can be utilized.

[0158] Furthermore, the learned model may be generated using only the learning endoscopic image. In this case, the learned model is not updated, and the same learned model can be always used.

[0159] Alternatively, the learned model may be generated using diagnostic endoscopic images in addition to the learning endoscopic images. In this case, the learned model is updated at any time by learning the relationship between the diagnostic endoscopic images and the lesion areas displayed in the diagnostic endoscopic images.

[0160] Next, the position information acquisition unit 104 acquires the position information in the body cavity of the endoscopic image. Here, a medical practitioner such as a doctor inputs the position information using the console 100. The position information acquisition unit 104 acquires the position information input from the console 100.

[0161] As the position information in the body cavity of the image, information of the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, cecum, ileum, jejunum, duodenum, stomach, esophagus, etc. may be input. These position candidates may be displayed on the display 20 in a selectable manner and selected by a medical practitioner such as a doctor using the console 100.

[0162] Next, the selection unit 106 selects the detection unit corresponding to the position information acquired by the position information acquisition unit 104 from the lesion region detection unit 102. That is, the selection unit 106 selects the first detection unit 102A when the position information is the rectum, selects the second detection unit 102B when the position information is the sigmoid colon, selects the third detection unit 102C when the position information is the descending colon, selects the fourth detection unit 102D when the position information is the transverse colon, selects the fifth detection unit 102E when the position information is the ascending colon, selects the sixth detection unit 102F when the position information is the cecum, selects the seventh detection unit 102G when the position information is the ileum, selects the eighth detection unit 102H when the position information is the jejunum, selects the ninth detection unit 102I when the position information is the duodenum, selects the tenth detection unit 102J when the position information is the stomach, and selects the eleventh detection unit 102K when the position information is the esophagus.

[0163] Next, the lesion area detection control unit 108 detects the lesion area from the endoscopic image through the detection unit selected by the selection unit 106. The lesion area here is not limited to the area caused by the disease, but also includes an area in a state different from the normal state in appearance. Examples of the lesion area include polyps, cancer, colon diverticula, inflammation, EMR (Endoscopic Mucosal Resection) scars or ESD (Endoscopic Submucosal Dissection) scars and other treatment marks, clipping sites, bleeding points, perforations, and vascular atypia.

[0164] <<Configuration of Ultrasonic Observation Device 14>>

[0165] The ultrasonic observation device 14 generates an ultrasonic image by causing the ultrasonic transducer unit 46 to transmit and receive ultrasonic waves, and by imaging the reception signal output by the ultrasonic transducer 48 (more specifically, the driven element) when receiving the ultrasonic waves. In addition to the generated ultrasonic image, the ultrasonic observation device 14 displays the endoscopic image and anatomical illustrations transmitted from the endoscope processor 16 on the display 20.

[0166] like Figure 5 As shown, the ultrasonic observation device 14 has a multiplexer 140, a receiving circuit 142, a transmitting circuit 144, an A / D converter 146, an ASIC (Application Specific Integrated Circuit) 148, a cine memory 150, a CPU (Central Processing Unit) 152, a DSC (Digital Scan Converter) 154, an ultrasonic image recognition unit 168, an operation process storage unit 174, a warning generation unit 176, a movement path registration unit 178 and a display control unit 172.

[0167] The receiving circuit 142 and the transmitting circuit 144 are electrically connected to the ultrasonic transducer array 50 of the ultrasonic endoscope 12. The multiplexer 140 selects at most m drive target transducers from the N ultrasonic transducers 48 and opens the channels thereof.

[0168] The transmission circuit 144 is composed of an FPGA (field programmable gate array), a pulse generator (pulse generating circuit 158), and a SW (switch), etc., and is connected to the MUX (multiplexer 140). In addition, an ASIC (application-specific integrated circuit) may be used instead of the FPGA.

[0169] The transmission circuit 144 is a circuit for supplying a driving voltage for ultrasonic transmission to the driven vibrator selected by the multiplexer 140 in accordance with a control signal transmitted from the CPU 152 in order to transmit ultrasonic waves from the ultrasonic vibrator unit 46. The driving voltage is a pulsed voltage signal (transmission signal) and is applied to the electrodes of the driven vibrator via the universal cord 26 and the coaxial cable 56.

[0170] The transmission circuit 144 has a pulse generating circuit 158 ​​that generates a transmission signal based on a control signal. Under the control of the CPU 152, the transmission circuit 144 generates a transmission signal that drives the plurality of ultrasonic vibrators 48 to generate ultrasonic waves using the pulse generating circuit 158 ​​and supplies the transmission signal to the plurality of ultrasonic vibrators 48. More specifically, when ultrasonic diagnosis is performed under the control of the CPU 152, the transmission circuit 144 generates a transmission signal having a driving voltage for performing ultrasonic diagnosis using the pulse generating circuit 158.

[0171] The receiving circuit 142 is a circuit for receiving an electrical signal output from the driven object transducer that has received the ultrasonic wave (echo), that is, a received signal. The receiving circuit 142 amplifies the received signal received from the ultrasonic transducer 48 according to the control signal transmitted from the CPU 152, and transmits the amplified signal to the A / D converter 146. The A / D converter 146 is connected to the receiving circuit 142, and converts the received signal received from the receiving circuit 142 from an analog signal to a digital signal, and outputs the converted digital signal to the ASIC 148.

[0172] ASIC 148 is connected to A / D converter 146, such as Figure 5 As shown, a phase matching unit 160 , a B-mode image generating unit 162 , a PW-mode image generating unit 164 , a CF-mode image generating unit 166 , and a memory controller 151 are configured.

[0173] In addition, in the present embodiment, the above functions (specifically, the phase matching unit 160, the B-mode image generating unit 162, the PW-mode image generating unit 164, the CF-mode image generating unit 166, and the storage controller 151) are realized by a hardware circuit such as the ASIC 148, but the present invention is not limited thereto. The above functions may also be realized by linking a central processing unit (CPU) with software (computer program) for executing various data processing.

[0174] The phase matching unit 160 performs a process of adding a delay time to the received signal (received data) digitized by the A / D converter 146 and performing phase matching and addition (adding after matching the phase of the received data). The phase matching and addition process generates a sound line signal with a narrowed focus of the ultrasonic echo.

[0175] The B-mode image generating unit 162 , the PW-mode image generating unit 164 , and the CF-mode image generating unit 166 generate ultrasonic images based on electrical signals (strictly speaking, acoustic line signals generated by phasing and adding received data) output by the driven transducers among the plurality of ultrasonic transducers 48 when the ultrasonic transducer unit 46 receives ultrasonic waves.

[0176] The B-mode image generating unit 162 is an image generating unit that generates a tomographic image of the inside of the patient (inside the body cavity), namely, a B-mode image. The B-mode image generating unit 162 corrects the attenuation caused by the propagation distance according to the depth of the reflection position of the ultrasound wave on the acoustic line signal generated sequentially by STC (Sensitivity Time Gain Control). In addition, the B-mode image generating unit 162 performs envelope detection processing and Log (logarithmic) compression processing on the corrected acoustic line signal to generate a B-mode image (image signal).

[0177] The PW mode image generating unit 164 is an image generating unit that generates an image showing the velocity of the blood flow in a predetermined direction. The PW mode image generating unit 164 performs a fast Fourier transform on a plurality of acoustic line signals in the same direction, among the acoustic line signals sequentially generated by the phase matching unit 160, to extract frequency components. Then, the PW mode image generating unit 164 calculates the velocity of the blood flow based on the extracted frequency components, and generates a PW mode image (image signal) showing the calculated velocity of the blood flow.

[0178] The CF mode image generating unit 166 is an image generating unit that generates an image showing information about blood flow in a predetermined direction. The CF mode image generating unit 166 obtains the autocorrelation of a plurality of acoustic line signals in the same direction among the acoustic line signals sequentially generated by the phase matching unit 160, thereby generating an image signal showing information about blood flow. Then, based on the image signal, the CF mode image generating unit 166 generates a CF mode image (image signal) as a color image in which information about blood flow is superimposed on the B mode image signal generated by the B mode image generating unit 162.

[0179] The memory controller 151 stores the image signal generated by the B-mode image generating unit 162 , the PW-mode image generating unit 164 , or the CF-mode image generating unit 166 in the cine memory 150 .

[0180] DSC154 is connected to ASIC148, and converts the signal of the image generated by the B-mode image generating unit 162, the PW-mode image generating unit 164 or the CF-mode image generating unit 166 into an image signal (raster conversion) in accordance with the scanning method of a conventional television signal, and performs various required image processing such as grayscale processing on the image signal, and then outputs it to the ultrasonic image recognition unit 168.

[0181] The ultrasonic image recognition unit 168 establishes a corresponding association between the position and orientation of the front end portion of the ultrasonic endoscope 12 in the body cavity of the subject and a label number based on the observation order of the observation target part such as an organ, and pre-learns the relationship between the learning ultrasonic image and the label number corresponding to the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 when the learning ultrasonic image was taken for a plurality of learning ultrasonic images, and recognizes the label number corresponding to the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 when the diagnostic ultrasonic image was taken from the ultrasonic image raster-converted by the DSC 154, that is, the diagnostic ultrasonic image generated by the ultrasonic observation device 14 based on the learning result. The label number recognized by the ultrasonic image recognition unit 168 is output to the display control unit 172 and the warning generation unit 176 described later (refer to Figure 5 ).

[0182] The observation order of the observation target site refers to the order in which the ultrasonic images of the observation target sites are captured (observed) in the body cavity of the subject. The observation order of the observation target sites will be described below with examples.

[0183] The label number is assigned according to the observation order of the observation object part. For example, if the observation object part with the first observation order is the left lobe of the liver, the left lobe of the liver is assigned the first label number. As long as the order is known, the label number can be any label, not necessarily a "number".

[0184] The learning ultrasonic image is an existing ultrasonic image used by the ultrasonic image recognition unit 168 to learn the relationship between the ultrasonic image and the label number corresponding to the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 when the ultrasonic image was captured. For example, various ultrasonic images captured in the past can be used.

[0185] like Figure 6 As shown, the ultrasonic image recognition unit 168 includes a label number detection unit 112 , an organ name detection unit 120 , and a position and orientation detection unit 122 .

[0186] The label number detection unit 112 detects, from the diagnostic ultrasonic image, a label number corresponding to the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 when the diagnostic ultrasonic image was captured based on the learning result.

[0187] The label number detection unit 112 is a learned model. The learned model is a model that uses a data set composed of learning ultrasonic images that each capture different positions of the observation target site in the subject, assigns label numbers corresponding to the positions and orientations of the front end portion 40 of the ultrasonic endoscope 12 when the learning ultrasonic images were captured, and learns in advance the relationship between the learning ultrasonic images and the label numbers corresponding to the positions and orientations of the front end portion 40 of the ultrasonic endoscope 12 when the learning ultrasonic images were captured.

[0188] The learning method is not particularly limited as long as it is a method that can learn the relationship between ultrasonic images and label numbers corresponding to the position and orientation of the distal end portion 40 when the ultrasonic endoscope 12 is used to capture images from a plurality of learning ultrasonic images and generate a learned model.

[0189] As a learning method, for example, deep learning using a hierarchical neural network, which is an example of machine learning, which is one of artificial intelligence (AI) technologies, can be used.

[0190] In addition, machine learning other than deep learning can be utilized, artificial intelligence techniques other than machine learning can be utilized, and learning methods other than artificial intelligence techniques can be utilized.

[0191] Furthermore, a learned model may be generated using only the learning ultrasonic image. In this case, the learned model is not updated, and the same learned model can be always used.

[0192] Alternatively, the learned model may be generated by using diagnostic ultrasonic images in addition to the learning ultrasonic images. In this case, the learned model is updated at any time by learning the relationship between the diagnostic ultrasonic images and the label numbers corresponding to the positions and orientations of the distal end portion 40 of the ultrasonic endoscope 12 when the diagnostic ultrasonic images were captured.

[0193] The observation order in the body when taking ultrasonic images (the movement path of the front end portion 40 of the ultrasonic endoscope 12) and the representative observation points (the position and orientation of the front end portion 40 of the ultrasonic endoscope 12) are roughly determined. Therefore, the ultrasonic image at the representative observation point and the tag number corresponding to the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 when the ultrasonic image at the observation point is taken can be associated and learned.

[0194] Hereinafter, representative observation points in the body (position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 ) when taking an ultrasonic image will be described in the order of observation.

[0195] Representative observation points in the body include, for example, the following (1) to (12).

[0196] (1) Left lobe of liver

[0197] (2) Confluence of the aorta, celiac artery, and superior mesenteric artery

[0198] (3) Pancreatic body

[0199] (4) Pancreatic tail

[0200] (5) Confluence of the splenic vein, superior mesenteric vein, and portal vein (junction point)

[0201] (6) Pancreatic head

[0202] (7) Gallbladder

[0203] (8) Portal vein

[0204] (9) Common bile duct

[0205] (10) Gallbladder

[0206] (11) Uncinate process of pancreatic head

[0207] (12) Papillary protrusions

[0208] Here, (1) the left lobe of the liver, (2) the confluence of the aorta, celiac artery and superior mesenteric artery, (3) the body of the pancreas, (4) the tail of the pancreas, (5) the confluence of the splenic vein, superior mesenteric vein and portal vein, (6) the head of the pancreas and (7) the gallbladder are representative observation points from the stomach, (8) the portal vein, (9) the common bile duct and (10) the gallbladder are representative observation points from the duodenal bulb, and (11) the uncinate process of the pancreatic head and (12) the papillary processes are representative observation points from the descending duodenum.

[0209] (1) to (12) correspond to the label numbers (the order of observation of the observation target parts), and the observation points of (1) to (12) correspond to the observation target parts (organs).

[0210] In addition, the above-mentioned observation order of the observation object part is an example, and there is a case where the observation order of the observation object part is slightly different depending on the practicing physician. Therefore, it can be set as follows, that is, according to the practicing physician, a plurality of lists of observation orders of the observation object parts that are different from each other are prepared, and about each list, the relationship between the learning ultrasonic image and the label number corresponding to the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 when the learning ultrasonic image is taken is learned for a plurality of learning ultrasonic images, and the practicing physician switches the list used, that is, the observation order of the observation object part. Alternatively, it can be set as follows, that is, the practicing physician can register a desired list.

[0211] Furthermore, the number of observation target parts in the list may be greater than the observation order of the observation target parts, or may be less than the observation order. That is, one or more other observation target parts may be added between one observation target part and the observation target part that is next to the one observation target part in the observation order, or one or more observation target parts may be deleted from a plurality of observation target parts that are continuous in the observation order.

[0212] Next, the organ name detection unit 120 detects the organ names corresponding to the label numbers (1) to (12) detected by the label number detection unit 112. The label numbers are associated with the order of observation of the observation object parts, so the name of the observation object part (organ) corresponding to the label number can be obtained based on the label number.

[0213] Next, the position and orientation detection unit 122 detects the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 when capturing the diagnostic ultrasonic image based on the tag numbers (1) to (12) detected by the tag number detection unit 112. Since the tag number is associated with the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 when capturing the ultrasonic image, the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 corresponding to the tag number can be obtained based on the tag number.

[0214] In the case of the present embodiment, as the position and orientation of the front end portion 40 of the ultrasonic endoscope 12, the observation points corresponding to the above-mentioned label numbers (1) to (12) are detected, that is, the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 when taking ultrasonic images of (1) the left lobe of the liver, (2) the confluence of the aorta, the celiac artery and the superior mesenteric artery, (3) the body of the pancreas, (4) the tail of the pancreas, (5) the confluence of the splenic vein, the superior mesenteric vein and the portal vein, (6) the head of the pancreas and (7) the gallbladder (representative observation points from the stomach), (8) the portal vein, (9) the common bile duct and (10) the gallbladder (representative observation points of the duodenal bulb), (11) the uncinate process of the head of the pancreas and (12) the papillary process (representative observation points of the duodenal bulb).

[0215] When the ultrasound transducer unit 46 is convex as in the present embodiment, it is preferable to make the position and orientation of the distal end portion 40 of the ultrasound endoscope 12 correspond to the tag number and detect the position and orientation corresponding to the tag number.

[0216] On the other hand, when the ultrasonic transducer unit 46 is a radial type, there is no need to detect the orientation of the distal end portion 40 of the ultrasonic endoscope 12, so it is preferred to make only the position of the distal end portion 40 of the ultrasonic endoscope 12 correspond to the tag number and detect only the position corresponding to the tag number.

[0217] The operation procedure storage unit 174 stores an operation procedure for moving the distal end portion 40 of the ultrasonic endoscope 12 from an observation target site corresponding to one tag number to an observation target site corresponding to a tag number next to the one tag number in observation order.

[0218] One label number is a label number corresponding to any one of the observation target parts observed in the observation order, and the label number next to the one label number in the observation order is a label number corresponding to the observation target part next to the any one observation target part.

[0219] For example, assume that the first observation object site corresponding to the first label number is the left lobe of the liver, and the second observation object site corresponding to the second label number is the confluence of the aorta, the celiac artery, and the superior mesenteric artery. In this case, the operation process storage unit 174 stores the operation process for moving the front end portion 40 of the ultrasonic endoscope 12 from the left lobe of the liver corresponding to the first label number to the confluence of the aorta, the celiac artery, and the superior mesenteric artery corresponding to the second label number. The operation process storage unit 174 also stores the operation process for the observation object sites corresponding to each label number after the second. In addition, the operation process for the observation object site corresponding to the last label number in the observation order is not stored.

[0220] The operation process includes various commands for moving the distal end portion 40 of the ultrasonic endoscope 12. For example, commands for moving the ultrasonic endoscope 12 forward, rotating the ultrasonic endoscope 12 clockwise or counterclockwise, and bending the distal end portion 40 of the ultrasonic endoscope 12 are included.

[0221] When the distal end portion 40 of the ultrasonic endoscope 12 moves from the observation target site corresponding to the current tag number to an observation target site other than the observation target site corresponding to the tag number next to the current tag number in observation order, the warning generating unit 176 issues a warning.

[0222] The current label number is a label number corresponding to the currently observed observation target part among the observation target parts observed in the observation order, in other words, a label number currently recognized by the ultrasonic image recognition unit 168. The label number next to the current label number in the observation order is a label number corresponding to the observation target part observed next to the currently observed observation target part.

[0223] Similarly, it is assumed that the first observation target site corresponding to the first label number is the left lobe of the liver, and the second observation target site corresponding to the second label number is the confluence of the aorta, celiac artery, and superior mesenteric artery. In this case, if the left lobe of the liver is currently observed, the first label number corresponding to the left lobe of the liver is the current label number, and the second label number corresponding to the confluence of the aorta, celiac artery, and superior mesenteric artery is the label number next to the current label number in the observation order.

[0224] As described above, the observation order of the observation target part is roughly determined. Therefore, when the front end portion 40 of the ultrasonic endoscope 12 moves from the observation target part corresponding to the current label number to the observation target part other than the observation target part corresponding to the label number next to the current label number in the observation order, the warning generating unit 176 determines that the movement direction of the front end portion 40 of the ultrasonic endoscope 12 operated by the practicing physician is wrong, and issues a warning.

[0225] The warning generating unit 176 generates a warning, whereby the medical practitioner can notice that the distal end portion 40 of the ultrasonic endoscope 12 is being moved in a wrong direction, and can move the distal end portion 40 of the ultrasonic endoscope 12 in a correct direction.

[0226] In the case of the present embodiment, the warning issued by the warning generating unit 176 is output to the display control unit 172, and under the control of the display control unit 172, a warning such as "Wrong moving direction!" is displayed as character information on the display 20. In addition, the mechanism for issuing the warning is not particularly limited, and for example, the warning may be issued as sound information from a speaker, or both character information and sound information may be issued as a warning.

[0227] The movement path registration unit 178 registers in advance a movement path when the distal end portion 40 of the ultrasonic endoscope 12 ideally moves according to the observation order of the observation target site. The ideal movement path is output to the display control unit 172 .

[0228] The ideal movement path refers to a movement path when the distal end portion 40 of the ultrasonic endoscope 12 is correctly operated and moved according to the observation order of the observation target site.

[0229] Next, the display control unit 172 displays the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 corresponding to the tag number recognized by the ultrasonic image recognition unit 168 on the display 20 .

[0230] Furthermore, the display control unit 172 displays the lesion area superimposed on the endoscopic image, displays the name of the organ superimposed on the ultrasonic image, or displays the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 superimposed on the anatomical illustration, according to an instruction from the practicing physician.

[0231] In other words, the display control unit 172 displays one or more images in parallel on the screen of the display 20 based on an order from a practicing physician, including an endoscopic image in which the lesion area is not displayed, an endoscopic image in which the lesion area is displayed in an overlapping manner, an ultrasonic image in which the name of an organ is not displayed, an ultrasonic image in which the name of an organ is displayed in an overlapping manner, an anatomical diagram in which the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 is not displayed, and an anatomical diagram in which the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 is displayed in an overlapping manner.

[0232] The name of the organ is displayed near the organ, for example, by being superimposed on the ultrasonic image, or on the organ, and the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 is displayed, for example, by being superimposed on the anatomical diagram. The lesion area is displayed, for example, by being superimposed on the endoscopic image and enclosing the lesion area with a frame line.

[0233] The cine memory 150 has a capacity for storing image signals of one frame or several frames. The image signal generated by the ASIC 148 is output to the DSC 154, and is also stored in the cine memory 150 through the memory controller 151. In the freeze mode, the memory controller 151 reads the image signal stored in the cine memory 150 and outputs it to the DSC 154. Thus, an ultrasonic image (still image) based on the image signal read from the cine memory 150 can be displayed on the display 20.

[0234] The CPU 152 functions as a control unit for controlling each part of the ultrasonic observation device 14, and is connected to the receiving circuit 142, the transmitting circuit 144, the A / D converter 146, and the ASIC 148, and controls these devices. Specifically, the CPU 152 is connected to the console 100, and controls each part of the ultrasonic observation device 14 according to the inspection information and control parameters input through the console 100.

[0235] Furthermore, when the ultrasonic endoscope 12 is connected to the ultrasonic observation apparatus 14 via the ultrasonic connector 32 a , the CPU 152 automatically recognizes the ultrasonic endoscope 12 by a method such as PnP (Plug and Play).

[0236] <<Operation Example of Ultrasonic Endoscope System 10>>

[0237] Next, as an operation example of the ultrasonic endoscope system 10, refer to Figure 7 and Figure 8 The flow of a series of processes related to ultrasonic diagnosis (hereinafter also referred to as diagnostic processes) will be described. Figure 7 1 is a diagram showing the flow of diagnostic processing using the ultrasonic endoscope system 10 . Figure 8 This is a diagram showing the process of diagnosis steps in the diagnosis process.

[0238] When the ultrasonic endoscope 12 is connected to the ultrasonic observation device 14, the endoscope processor 16, and the light source device 18, the power of each part of the ultrasonic endoscope system 10 is turned on, and the diagnostic process is started with this as a trigger. Figure 7 As shown, the input step (S001) is first implemented. In the input step, the practicing physician inputs the examination information and control parameters through the operation console 100. If the input step is completed, the standby step (S002) is implemented until a command to start diagnosis is issued.

[0239] Next, when a diagnosis start command is issued from the practicing physician (Yes in S003), the CPU 152 controls each part of the ultrasonic observation device 14 to perform a diagnosis step (S004). Figure 8, when the designated image generation mode is the B mode ("Yes" in S031), the various parts of the ultrasonic observation device 14 are controlled so as to generate a B mode image (S032). Furthermore, when the designated image generation mode is not the B mode ("No" in S031) but the CF mode ("Yes" in S033), the various parts of the ultrasonic observation device 14 are controlled so as to generate a CF mode image (S034). Furthermore, when the designated image generation mode is not the CF mode ("No" in S033) but the PW mode ("Yes" in S035), the various parts of the ultrasonic observation device 14 are controlled so as to generate a PW mode image (S036). In addition, when the designated image generation mode is not the PW mode ("No" in S035), the process proceeds to step S037.

[0240] Next, the CPU 152 determines whether the ultrasonic diagnosis has been completed (S037). If the ultrasonic diagnosis has not been completed ("No" in S037), the process returns to the diagnostic step S031 and repeatedly generates ultrasonic images based on each image generation mode until the diagnosis end condition is met. As the diagnosis end condition, for example, the practicing physician instructs the diagnosis to end through the console 100.

[0241] On the other hand, if the diagnosis end condition is satisfied and the ultrasonic diagnosis is ended (YES in S037 ), the diagnosis step is ended.

[0242] Next, return to Figure 7 If the power of each part of the ultrasonic endoscope system 10 is turned off ("Yes" in S006), the diagnostic processing is terminated. On the other hand, when the power of each part of the ultrasonic endoscope system 10 remains on ("No" in S005), the process returns to the input step S001 and the steps of the above-mentioned diagnostic processing are repeated.

[0243] <<Display method of endoscopic images, ultrasonic images, and anatomical illustrations>>

[0244] Next, a method for displaying endoscopic images, ultrasonic images, and anatomical illustrations will be described.

[0245] The practicing physician can display at least one of an endoscopic image, an ultrasonic image, and an anatomical illustration on the screen of the display 20 by issuing a command through the operation console 100 .

[0246] In this case, through the display control unit 172 and based on an order from a practicing physician, one image or two or more images are displayed side by side on the screen of the display 20 from the endoscopic image (with / without displaying the lesion area), the ultrasonic image (with / without displaying the name of the organ) and the anatomical illustration (with / without displaying the position and orientation of the front end portion 40 of the ultrasonic endoscope 12).

[0247] Furthermore, in response to an instruction from a medical practitioner, the display control unit 172 can display one image as an image of interest from among two or more images displayed on the display 20 so as to be larger than the other images.

[0248] In the ultrasonic endoscope system 10 , when an ultrasonic image or an anatomical diagram is displayed on the screen of the display 20 , the ultrasonic image recognition unit 168 operates, and when an endoscopic image is displayed on the screen of the display 20 , the endoscopic image recognition unit 170 operates.

[0249] Thus, according to an order from a practicing physician, an endoscopic image with a lesion area superimposed thereon can be displayed on the display 20, or an ultrasonic image with the name of an organ superimposed thereon can be displayed on the display 20, or an anatomical diagram with the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 superimposed thereon can be displayed on the display 20.

[0250] For example, Fig. 9 As shown, the medical practitioner can display endoscopic images, ultrasonic images, and anatomical illustrations on the screen of the display 20 .

[0251] In from Fig. 9 In the screen of the display 20 shown, an ultrasonic image is displayed in the center from the left, and the names of organs Panc, PD, SV, and SA are superimposed on the ultrasonic image. Panc represents the name of the pancreas, PD represents the name of the pancreatic duct, SV represents the name of the splenic vein, and SA represents the name of the splenic artery. An anatomical diagram is displayed in the upper right part of the screen of the display 20, and the position and direction of the front end portion 40 of the ultrasonic endoscope 12 are superimposed on the anatomical diagram. An endoscopic image that does not display the lesion area is displayed in the lower right part of the screen of the display 20. In the case of this example, the ultrasonic image is displayed as a focus image larger than the anatomical diagram and the endoscopic image. In addition, between the anatomical diagram and the endoscopic image in the right center of the screen of the display 20, the operation process for moving the front end portion 40 of the ultrasonic endoscope 12 is displayed as a text message "Next observation target site: Please rotate clockwise along the 4th pancreatic tail SV."

[0252] In addition, it is not limited to Fig. 9 For example, the practicing physician can arbitrarily combine one image or two or more images to arrange and display them on the screen of the display 20. In addition, the practicing physician can arbitrarily set the position of the endoscopic image, ultrasonic image, anatomical illustration and operation process. In addition, the practicing physician can switch the display of the image of interest from the images displayed on the display 20.

[0253] <<Display control method by display control unit>>

[0254] Next, a method of controlling various displays by the display control unit 172 will be described.

[0255] First, a method of displaying the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 will be described.

[0256] The display control unit 172 can display the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 corresponding to the tag number recognized by the ultrasonic image recognition unit 168 as character information on the display 20. In this case, the display control unit 172 displays character information describing the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12, such as, for example, "Currently, the distal end portion of the ultrasonic endoscope is tracing the direction from the stomach to the left lobe of the liver." on the display 20. In addition, the display control unit 172 can display the name of the observation target site corresponding to the tag number recognized by the ultrasonic image recognition unit 168 on the display 20 as character information.

[0257] Furthermore, the display control unit 172 can display an anatomical diagram on the display 20, which is superimposed with the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 recognized by the ultrasonic image recognition unit 168 as image information. Fig. 9 As shown in the upper right part of , the anatomical illustration is displayed on the display 20, and the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 are superimposed and displayed as image information on the anatomical illustration.

[0258] In this way, by displaying the position and orientation of the front end portion 40 of the ultrasonic endoscope 12 on the display 20, for example, even a practicing physician who is not familiar with ultrasonic images can reliably understand where the front end portion 40 of the ultrasonic endoscope 12 is currently located, in which direction it is facing, and which part is being observed.

[0259] Furthermore, the display control unit 172 may display the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 as character information on the display 20 and as image information on the display 20. That is, both the character information and the image information may be displayed simultaneously.

[0260] Next, a method of displaying the operation progress is described.

[0261] The display control unit 172 can display the operation process for moving the distal end portion 40 of the ultrasonic endoscope 12 on the display 20. In this case, the display control unit 172 acquires the operation process for moving the distal end portion 40 of the ultrasonic endoscope 12 from the observation target part corresponding to the current label number to the observation target part corresponding to the label number next to the current label number in the observation order from the operation process storage unit 174, and the operation process acquired from the operation process storage unit 174 is displayed on the display 20.

[0262] Thus, for example, even a medical practitioner who is not familiar with ultrasonic images can accurately move the distal end portion 40 of the ultrasonic endoscope 12 from the current observation target site to the next observation target site without getting lost in the body of the subject.

[0263] For example, the display control unit 172 can display the operation procedure on the display 20 as character information explaining the operation procedure.

[0264] In this case, if Fig. 9 As shown in the right center portion of , the operation process may include the names of one or more organs that are marked as the operation. Fig. 9 In the example of the operation process of, "Please rotate clockwise along SV.", "SV" is the name of the organ that serves as the operation mark.

[0265] Thus, the medical practitioner can easily grasp the direction in which the distal end portion 40 of the ultrasonic endoscope 12 is to be moved, and can move the distal end portion 40 of the ultrasonic endoscope 12 in the correct direction.

[0266] The organ marked for operation is not particularly limited as long as it is an organ that becomes a target object when the distal end portion 40 of the ultrasonic endoscope 12 is moved and is depicted when the distal end portion 40 of the ultrasonic endoscope 12 is moved, in other words, it is an organ that exists in the direction from the observation object part corresponding to the current label number toward the observation object part corresponding to the next label number in the observation order from the current label number. Examples thereof include one or more organs selected from organs displayed in the current ultrasonic image, organs displayed within a certain range from the position of the distal end portion 40 of the ultrasonic endoscope 12 in the current ultrasonic image, and a plurality of specified organs.

[0267] Furthermore, the display control unit 172 can display, on the display 20 , an anatomical illustration in which the operation process is superimposed and displayed as image information indicating the movement path of the distal end portion 40 of the ultrasonic endoscope 12 .

[0268] In this case, the display control unit 172 may color the region of one or more organs serving as a mark of the operation on the anatomical illustration, and display the anatomical illustration with the region colored on the display 20 .

[0269] Furthermore, the display control unit 172 can color the area of ​​the observation object part corresponding to the next label number of the current label number in the observation order and the area of ​​one or more organs marked as the operation in different colors on the anatomical illustration, and display the anatomical illustration with these areas colored in different colors on the display 20.

[0270] In this way, by coloring the region of the organ that is present in the direction in which the distal end portion 40 of the ultrasonic endoscope 12 is moved, the medical practitioner can easily understand the direction in which the distal end portion 40 of the ultrasonic endoscope 12 is moved.

[0271] Furthermore, the display control unit 172 may simultaneously display the position and orientation of the distal end portion 40 of the ultrasonic endoscope 12 and the operation process on the display 20 .

[0272] like Fig. 9 As shown in the right center portion of , the display control unit 172 further displays the name of the observation target part corresponding to the tag number next to the current tag number in the observation order as character information on the display 20. Fig. 9 In the example of the operation process, "next observation object site: 4th pancreatic tail" is the name of the observation object site corresponding to the next label number. The label number is associated with the observation order of the observation object site, so the name of the observation object site corresponding to the label number next to the label number in the observation order can be obtained according to the label number.

[0273] Thereby, the practicing physician can easily understand which part is the next observation target part.

[0274] In addition, the display control unit 172 may add a check mark to the label number corresponding to the arrived observation target part each time the distal end portion 40 of the ultrasonic endoscope 12 reaches the observation target part corresponding to each label number, such as Fig.10 As shown in FIG. 1 , the label number to which the check mark is assigned is displayed as character information on the display 20. That is, the label number corresponding to the observation target site reached by the distal end portion 40 of the ultrasonic endoscope 12 is assigned a check mark. Fig.10In the example of , check marks are assigned to the label numbers (1) left lobe of the liver, (2) confluence of the aorta, celiac artery, and superior mesenteric artery, and (3) left side of the pancreatic body, but no check marks are assigned to the label numbers (4) pancreatic tail and thereafter. Therefore, it can be known that the front end portion 40 of the ultrasonic endoscope 12 has reached the observation object portion corresponding to the label number (3) pancreatic body.

[0275] Furthermore, each time the distal end portion 40 of the ultrasonic endoscope 12 reaches the observation target site corresponding to each label number, the display control unit 172 may color the region of the observation target site reached on the anatomical illustration, and display the anatomical illustration in which the region of the observation target site reached is colored on the display 20. That is, the region of the observation target site reached by the distal end portion 40 of the ultrasonic endoscope 12 is colored.

[0276] Thus, the practicing physician can easily understand the observation target part corresponding to the label number in the observation order that the front end portion 40 of the ultrasonic endoscope 12 has reached. That is, the practicing physician can confirm that the front end portion 40 of the ultrasonic endoscope 12 has reached the observation target part corresponding to the label number assigned with a check mark, or has reached the colored observation target part. Accordingly, the practicing physician can easily understand which observation target part corresponds to the label number next to the current label number in the observation order.

[0277] Furthermore, the display control unit 172 may emphasize the region of the observation target part corresponding to the label number next to the current label number in the observation order on the anatomical illustration, and display the anatomical illustration with the emphasized region of the observation target part on the display 20 .

[0278] For example, the display control unit 172 can color the area of ​​the observation object part corresponding to the next label number after the current label number in the observation order on the anatomical illustration with a color different from the color of the areas of the observation object part other than the observation object part corresponding to the next label number after the current label number in the observation order. For example, the display control unit 172 can color the area of ​​the observation object part corresponding to the next label number after the current label number in the observation order with a darker color or a lighter color than the areas of the observation object part other than the observation object part corresponding to the next label number after the current label number in the observation order, and display the anatomical illustration with the colored areas of the observation object part on the display 20.

[0279] Thus, on the anatomical illustration, the region of the observation target part corresponding to the label number next to the current label number in the observation order is highlighted compared to the regions of other observation target parts. Therefore, the practicing physician can easily move the front end portion 40 of the ultrasonic endoscope 12 from the observation target part corresponding to the current label number to the observation target part corresponding to the label number next to the current label number in the observation order.

[0280] The method of emphasizing the area of ​​the observation target part is not limited to the above method, and examples thereof include surrounding the area of ​​the observation target part to be emphasized with a thick frame, coloring only the area of ​​the observation target part to be emphasized, marking an arrow indicating the observation target part to be emphasized, etc.

[0281] Furthermore, the display control unit 172 can obtain the movement path of the front end portion 40 of the ultrasonic endoscope 12 when it ideally moves according to the observation order of the observation object part from the movement path registration unit 178, and arrange the movement path of the front end portion 40 of the ultrasonic endoscope 12 when it ideally moves according to the observation order of the observation object part and the movement path of the front end portion 40 of the ultrasonic endoscope 12 when it actually moves based on the actual operation of the practicing physician as image information representing these paths on the anatomical diagram and display them on the display 20.

[0282] Thus, the medical practitioner can move the distal end portion 40 of the ultrasonic endoscope 12 while confirming the ideal movement path and the movement path based on his / her own operation. Therefore, the medical practitioner can move the distal end portion 40 of the ultrasonic endoscope 12 in such a way that the actual path coincides with the ideal path, and as a result, the actual movement path can be brought close to the ideal movement path.

[0283] <<Arrangement Positions of Ultrasonic Image Recognition Unit 168 and Endoscope Image Recognition Unit 170>>

[0284] Next, the arrangement positions of the ultrasonic image recognition unit 168 and the endoscope image recognition unit 170 will be described.

[0285] In the present embodiment, the ultrasonic image recognition unit 168 is built into the ultrasonic observation device 14 , but the present invention is not limited thereto and may be built into the endoscope processor 16 , or may be provided outside the ultrasonic observation device 14 and the endoscope processor 16 .

[0286] When the ultrasonic image recognition unit 168 is built into the ultrasonic observation device 14 as in the present embodiment, Fig.11 As shown, the endoscopic image is transmitted from the endoscope processor 16 to the ultrasonic observation device 14 .

[0287] Furthermore, when the ultrasonic image recognition unit 168 is built into the endoscope processor 16, Fig.12 As shown, the ultrasonic image is transmitted from the ultrasonic observation device 14 to the endoscope processor 16 .

[0288] When the ultrasonic image recognition unit 168 is provided outside the ultrasonic observation device 14 and the endoscope processor 16, Fig.13 As shown, the endoscopic image is transmitted from the endoscope processor 16 to the ultrasonic observation device 14 , and the endoscopic image and the ultrasonic image are transmitted from the ultrasonic observation device 14 to the ultrasonic image recognition unit 168 .

[0289] In this case, the ultrasonic image may be transmitted from the ultrasonic observation device 14 to the endoscope processor 16, and the endoscopic image and the ultrasonic image may be transmitted from the endoscope processor 16 to the ultrasonic image recognition unit 168. Alternatively, the endoscopic image may be transmitted from the endoscope processor 16 to the ultrasonic image recognition unit 168 instead of being transmitted from the endoscope processor 16 to the ultrasonic observation device 14 and then being transmitted from the ultrasonic observation device 14 to the ultrasonic image recognition unit 168.

[0290] The display control unit 172 is arranged between the final image signal output to the display 20 and the display 20 .

[0291] When the ultrasonic image recognition unit 168 is built in the ultrasonic observation apparatus 14 , the display control unit 172 can be built in the ultrasonic observation apparatus 14 or provided between the ultrasonic observation apparatus 14 and the display 20 , for example.

[0292] Furthermore, when the ultrasonic image recognition unit 168 is built in the endoscope processor 16 , the display control unit 172 can be built in the endoscope processor 16 , or provided between the endoscope processor 16 and the display 20 , for example.

[0293] Furthermore, when the ultrasonic image recognition unit 168 is provided outside the ultrasonic observation apparatus 14 and the endoscope processor 16 , the display control unit 172 can be provided outside the ultrasonic observation apparatus 14 and the endoscope processor 16 , for example.

[0294] The display control unit 172 displays one or more images side by side on the screen of the display 20 from the endoscopic image (with / without display of the lesion area), the ultrasonic image (with / without display of the name of the organ) and the anatomical diagram (with / without display of the position and orientation of the front end portion 40 of the ultrasonic endoscope 12) in accordance with an order from a practicing physician.

[0295] The configuration position of the endoscope image recognition unit 170 can also be determined in the same manner as the configuration position of the ultrasonic image recognition unit 168. That is, in the case of the present embodiment, the endoscope image recognition unit 170 is built into the endoscope processor 16, but is not limited thereto, and may be built into the ultrasonic observation device 14, or may be provided outside the ultrasonic observation device 14 and the endoscope processor 16.

[0296] As described above, in the ultrasonic endoscope system 10 , the arrangement positions of the ultrasonic image recognition unit 168 and the endoscopic image recognition unit 170 are not fixed, and the ultrasonic image recognition unit 168 and the endoscopic image recognition unit 170 can be arranged at arbitrary arrangement positions.

[0297] In the device of the present invention, for example, the hardware structure of the processing unit (Processing Unit) that performs various processes, such as the endoscope image recognition unit 170 (lesion area detection unit 102, position information acquisition unit 104, selection unit 106 and lesion area detection control unit 108), the ultrasonic image recognition unit 168 (label number detection unit 112, position information acquisition unit 114, selection unit 116, label number detection control unit 118, organ name detection unit 120 and position and orientation detection unit 122), the display control unit 172, the warning generation unit 176 and the console (command acquisition unit) 100, may be dedicated hardware or various processors or computers that execute programs. Furthermore, the hardware structure of the operation process storage unit 174 and the movement path registration unit 178 may be dedicated hardware or a memory such as a semiconductor memory.

[0298] Various processors include general-purpose processors such as CPU (Central Processing Unit) that execute software (programs) and function as various processing units, FPGA (Field Programmable Gate Array) and other processors whose circuit structures can be changed after manufacturing, such as Programmable Logic Device (PLD), ASIC (Application Specific Integrated Circuit), and other processors with circuit structures specially designed to execute specific processing, such as dedicated electrical circuits.

[0299] One processing unit may be constituted by one of these various processors, or by a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of FPGAs and CPUs), etc. Furthermore, multiple processing units may be constituted by one of the various processors, or by integrating two or more of the multiple processing units and using one processor.

[0300] For example, in computers such as servers and clients, there is a method in which a processor is composed of a combination of one or more CPUs and software, and the processor functions as multiple processing units. In addition, there is a method in which a processor is used to implement the functions of the entire system including multiple processing units on a single IC (Integrated Circuit) chip, such as a system on chip (SoC).

[0301] Furthermore, more specifically, the hardware structure of these various processors is an electric circuit (Circuitry) in which circuit elements such as semiconductor elements are combined.

[0302] Furthermore, the method of the present invention can be implemented, for example, by a program for causing a computer to execute each step thereof, and a computer-readable recording medium recording the program can also be provided.

[0303] As mentioned above, although this invention was demonstrated in detail, this invention is not limited to the said embodiment, It is obvious that various improvements and changes are possible without departing from the scope of the gist of this invention.

[0304] Explanation of symbols

[0305] 10-ultrasonic endoscope system, 12-ultrasonic endoscope, 14-ultrasonic observation device, 16-endoscope processor, 18-light source device, 20-display, 21a-water supply tank, 21b-suction pump, 22-insertion part, 24-operation part, 26-universal cord, 28a-air and water supply button, 28b-suction button, 29-angle button, 30-treatment instrument insertion port, 32a-ultrasonic connector, 32b-endoscope connector, 32c-light source connector, 34a-air and water supply hose, 34b -Suction hose, 36-ultrasonic observation part, 38-endoscope observation part, 40-front end, 42-bending part, 43-soft part, 44-treatment instrument outlet, 45-treatment instrument channel, 46-ultrasonic vibrator unit, 48-ultrasonic vibrator, 50-ultrasonic vibrator array, 54-backing material layer, 56-coaxial cable, 60-FPC, 74-acoustic matching layer, 76-acoustic lens, 82-observation window, 84-objective lens, 86-solid imaging element, 88-illumination window, 90-cleaning nozzle, 92-wiring cable , 100-operating table, 102-lesion area detection unit, 102A~102K-1st to 12th detection units, 104, 114-position information acquisition unit, 106, 116-selection unit, 108-lesion area detection control unit, 112-label number detection unit, 112A~112L-1st to 12th detection units, 118-label number detection control unit, 120-organ name detection unit, 122-position and direction detection unit, 140-multiplexer, 142-receiving circuit, 144-transmitting circuit, 146- A / D converter, 148-ASIC, 150-movie memory, 151-storage controller, 152-CPU, 154-DSC, 158-pulse generating circuit, 160-phase matching unit, 162-B mode image generating unit, 164-PW mode image generating unit, 166-CF mode image generating unit, 168-ultrasonic image recognition unit, 170-endoscopic image recognition unit, 172-display control unit, 174-operation process storage unit, 176-warning generating unit, 178-movement path registration unit.

Claims

1. An ultrasonic endoscope system comprising: An ultrasonic endoscope has an ultrasonic transducer at the front end; an ultrasonic observation device for transmitting and receiving ultrasonic waves by the ultrasonic transducer, and generating a diagnostic ultrasonic image based on a reception signal of the ultrasonic waves transmitted from the body cavity of the subject toward the observation target site; an acquisition unit, which acquires the diagnostic ultrasonic image; an ultrasonic image recognition unit that outputs a label number corresponding to the position of the front end portion of the ultrasonic endoscope when the diagnostic ultrasonic image was taken, by inputting the acquired diagnostic ultrasonic image into a learned model, wherein the learned model outputs label numbers based on the order of observation target parts of the ultrasonic image taken in the body cavity of the subject corresponding to the position of the front end portion of the ultrasonic endoscope when the ultrasonic image was taken, i.e., the order of observation of the observation target parts, based on the ultrasonic image generated by the ultrasonic observation device; a display control unit that displays, on a display, a position of a distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit; and an operation process storage unit storing an operation process for moving the front end portion of the ultrasonic endoscope from an observation target portion corresponding to a tag number to an observation target portion corresponding to a tag number next to the tag number in the observation order, The display control unit obtains from the operation process storage unit an operation process for moving the front end portion of the ultrasonic endoscope from an observation object part corresponding to a current label number to an observation object part corresponding to a label number next to the current label number in the observation order, and displays the obtained operation process on the display.

2. The ultrasonic endoscope system according to claim 1, wherein: The display control unit displays the name of the observation target site corresponding to the recognized tag number on the display as character information.

3. The ultrasonic endoscope system according to claim 1, wherein: The display control unit displays, on the display, an anatomical illustration in which the position of the distal end portion of the ultrasonic endoscope corresponding to the recognized tag number is superimposed and displayed as image information.

4. The ultrasonic endoscope system according to claim 1, wherein: The ultrasonic image recognition unit outputs a label number corresponding to the position and orientation of the front end portion of the ultrasonic endoscope when the diagnostic ultrasonic image was taken by inputting the acquired diagnostic ultrasonic image into the learned model. The learned model outputs a label number based on the ultrasonic image generated by the ultrasonic observation device, which is the order of observation of the observation object parts in the body cavity of the subject where the ultrasonic image is taken, corresponding to the position and orientation of the front end portion of the ultrasonic endoscope when the ultrasonic image was taken. The display control unit displays, on the display, the position and orientation of the distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit.

5. The ultrasonic endoscope system according to claim 4, wherein: The display control unit displays the name of the observation target site corresponding to the recognized tag number on the display as character information.

6. The ultrasonic endoscope system according to claim 4, wherein: The display control unit displays, on the display, an anatomical illustration in which the position and orientation of the distal end portion of the ultrasonic endoscope corresponding to the recognized tag number are superimposed and displayed as image information.

7. The ultrasonic endoscope system according to claim 1, wherein: The display control unit further displays, on the display as character information, a name of an observation target site corresponding to a tag number next to the current tag number in the observation order.

8. The ultrasonic endoscope system according to claim 1, wherein: The display control unit displays the operation process as character information on the display.

9. The ultrasonic endoscope system according to claim 8, wherein: The operation process includes the names of one or more organs to be imaged when the distal end portion of the ultrasonic endoscope is moved.

10. The ultrasonic endoscope system according to claim 1, wherein: The display control unit displays, on the display, an anatomical diagram on which the operation process is superimposed and displayed as image information.

11. The ultrasonic endoscope system according to claim 10, wherein: The display control unit colors the regions of one or more organs depicted when the distal end portion of the ultrasonic endoscope is moved on the anatomical illustration, and displays the anatomical illustration with the regions colored on the display.

12. The ultrasonic endoscope system according to claim 11, wherein: The display control unit colors the area of ​​the observation object part corresponding to the next label number of the current label number in the observation order and the area of ​​one or more organs depicted when the front end portion of the ultrasonic endoscope is moved in different colors on the anatomical illustration, and displays the anatomical illustration with the areas colored in different colors on the display.

13. The ultrasonic endoscope system according to claim 1, wherein: The ultrasonic endoscope system further comprises: The warning generating unit generates a warning when the distal end portion of the ultrasonic endoscope moves from the observation target portion corresponding to the current tag number to an observation target portion other than the observation target portion corresponding to a tag number next to the current tag number in the observation order.

14. The ultrasonic endoscope system according to claim 13, wherein: The warning generating unit generates the warning as sound information, or generates both character information and sound information as the warning.

15. The ultrasonic endoscope system according to claim 1, wherein: The display control unit assigns a check mark to the label number corresponding to the arrived observation target part each time the distal end portion of the ultrasonic endoscope reaches the observation target part corresponding to each label number, and displays the label number assigned the check mark as character information on the display.

16. The ultrasonic endoscope system according to claim 1, wherein: Each time the front end portion of the ultrasonic endoscope reaches an observation target site corresponding to each label number, the display control unit colors the area of ​​the observation target site that has been reached on the anatomical illustration, and displays the colored anatomical illustration of the area of ​​the observation target site that has been reached on the display.

17. The ultrasonic endoscope system according to claim 1, wherein: The display control unit emphasizes a region of the observation target part corresponding to the label number next to the current label number in the observation order on the anatomical illustration, and displays the anatomical illustration with the emphasized region of the observation target part on the display.

18. The ultrasonic endoscope system according to claim 17, wherein: The display control unit colors, on the anatomical illustration, an area of ​​the observation object part corresponding to the label number next to the current label number in the observation order with a color different from that of areas of the observation object part other than the observation object part corresponding to the label number next to the current label number in the observation order, and displays the anatomical illustration with the colored area of ​​the observation object part on the display.

19. The ultrasonic endoscope system according to claim 1, wherein: The display control unit arranges the movement path of the ultrasonic endoscope when the distal end portion ideally moves according to the observation order of the observation target part and the movement path of the ultrasonic endoscope when the distal end portion actually moves according to the actual operation as image information on the anatomical diagram and displays them on the display.

20. The ultrasonic endoscope system according to claim 19, wherein: The ultrasonic endoscope system further comprises: The movement path registration unit registers in advance a movement path when the distal end portion of the ultrasonic endoscope is ideally moved according to the observation order of the observation target site.

21. The ultrasonic endoscope system according to claim 1 or 2, wherein: The ultrasonic image recognition unit is built in the ultrasonic observation device.

22. The ultrasonic endoscope system according to claim 1 or 2, wherein: The ultrasonic endoscope also has an illumination unit and an imaging unit at the front end. The ultrasonic endoscope system further includes an endoscope processor, which receives reflected light of the illumination light irradiated from the illumination unit through the imaging unit and generates a diagnostic endoscopic image based on an imaging signal of the reflected light. The ultrasonic image recognition unit is built into the endoscope processor.

23. The ultrasonic endoscope system according to claim 1 or 2, wherein: The ultrasonic endoscope also has an illumination unit and an imaging unit at the front end. The ultrasonic endoscope system further includes an endoscope processor, which receives reflected light of the illumination light irradiated from the illumination unit through the imaging unit and generates a diagnostic endoscopic image based on an imaging signal of the reflected light. The ultrasonic image recognition unit is provided outside the ultrasonic observation device and the endoscope processor.

24. A method for operating an ultrasonic endoscope system, comprising the following steps: The ultrasonic observation device transmits and receives ultrasonic waves by means of an ultrasonic transducer provided at a distal end of the ultrasonic endoscope, and generates a diagnostic ultrasonic image based on a received signal of the ultrasonic wave transmitted from a body cavity of a subject toward an observation target site; The step of an acquisition unit acquiring the diagnostic ultrasonic image; The ultrasonic image recognition unit outputs a label number corresponding to the position of the front end portion of the ultrasonic endoscope when the diagnostic ultrasonic image was taken by inputting the acquired diagnostic ultrasonic image into the learned model, wherein the learned model outputs label numbers based on the order of observation target parts in the body cavity of the subject where the ultrasonic image was taken, i.e., the observation order of the observation target parts, corresponding to the position of the front end portion of the ultrasonic endoscope when the ultrasonic image was taken, based on the ultrasonic image generated by the ultrasonic observation device; A step of causing a display control unit to display, on a display, a position of a distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit; and The operation process storage unit stores the steps of the operation process for moving the distal end portion of the ultrasonic endoscope from an observation target site corresponding to a tag number to an observation target site corresponding to a tag number next to the tag number in the observation order, The identified label number is used as the current label number, and the operation process for moving the front end portion of the ultrasonic endoscope from the observation object part corresponding to the current label number to the observation object part corresponding to the label number next to the current label number in the observation order is obtained from the operation process storage unit. The acquired operation process is displayed on the display.

25. The operating method of the ultrasonic endoscope system according to claim 24, wherein: The acquired diagnostic ultrasonic image is input into the learned model to output a label number corresponding to the position and orientation of the front end portion of the ultrasonic endoscope when the diagnostic ultrasonic image was taken. The learned model outputs label numbers corresponding to the position and orientation of the front end portion of the ultrasonic endoscope when the ultrasonic image was taken, i.e., the order of observation of the observation object parts, based on the ultrasonic image generated by the ultrasonic observation device, of the observation object parts in the body cavity of the subject where the ultrasonic image was taken. The position and orientation of the distal end portion of the ultrasonic endoscope corresponding to the tag number recognized by the ultrasonic image recognition unit are displayed on the display.

Images