Image processing device, endoscope system, image processing method, and program

Abstract

To provide an image processing device, an endoscope system, an image processing method, and a program capable of outputting an accurate recognition result for an observation object region shown in an ultrasonic image obtained by a B mode method.SOLUTION: An image processing device includes a processor. The processor acquires an ultrasonic image obtained by a B mode method showing an observation object region of a subject in ultrasonic diagnosis for the subject, and blood flow distribution information obtained by a Doppler method used together with the B mode method in the ultrasonic diagnosis. The processor outputs a recognition result acquired by performing recognition processing for causing a learned model to recognize the observation object region by inputting the ultrasonic image and the blood flow distribution information to the learned model.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an endoscope system, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an ultrasound diagnostic device including a transmission control unit, a detection unit, an evaluation unit, and a display image control unit. In the ultrasound diagnostic device described in Patent Document 1, the transmission control unit causes the ultrasound probe to transmit a first ultrasound wave toward biological tissue, and after the first ultrasound wave is transmitted, causes the ultrasound probe to transmit a second ultrasound wave. In the ultrasound diagnostic device described in Patent Document 1, the detection unit detects the presence or absence of movement of an object in the biological tissue due to the first ultrasound wave within a two-dimensional region based on an echo signal obtained by transmitting the second ultrasound wave. In the ultrasound diagnostic device described in Patent Document 1, the evaluation unit evaluates the tissue properties of a region of interest in the biological tissue based on the detection by the detection unit. In the ultrasound diagnostic device described in Patent Document 1, the display image control unit displays an image according to the evaluation by the evaluation unit. The process by the detection unit to detect the presence or absence of movement of an object is color Doppler processing or B-flow processing, and the evaluation unit performs evaluation according to the size of the region from which Doppler data is obtained by color Doppler processing or the region from which B-flow data is obtained by B-flow processing.

[0003] Patent Document 2 discloses an ultrasound diagnostic device comprising an image acquisition unit, a monitor, a debris identification unit, and a debris movement information providing unit. In the ultrasound diagnostic device described in Patent Document 2, the image acquisition unit acquires ultrasound images of a cyst in a subject by transmitting and receiving ultrasound to and from the subject. In the ultrasound diagnostic device described in Patent Document 2, the monitor displays the ultrasound images. In the ultrasound diagnostic device described in Patent Document 2, the debris identification unit identifies debris within the cyst from a first ultrasound image acquired by the image acquisition unit and a second ultrasound image acquired by the image acquisition unit after an external force different from that at the time of capturing the first ultrasound image has acted on the cyst or is still acting on the cyst, and from the same direction as that at the time of capturing the first ultrasound image. In the ultrasound diagnostic device described in Patent Document 2, the debris movement information providing unit provides information regarding the movement of debris within the cyst identified by the debris identification unit between the first ultrasound image and the second ultrasound image.

[0004] In the ultrasound diagnostic device described in Patent Document 2, the debris identification unit identifies debris while aligning the cyst identified from the first ultrasound image with the cyst identified from the second ultrasound image. In the ultrasound diagnostic device described in Patent Document 2, the debris identification unit includes an identity determination unit that determines whether the same cyst has been captured by performing image recognition on the first ultrasound image and the second ultrasound image. In the ultrasound diagnostic device described in Patent Document 2, the identity determination unit determines whether the same cyst has been captured using a trained determination model that receives as input the first ultrasound image and the second ultrasound image.

[0005] Patent Document 3 discloses an ultrasound diagnostic device including a physical quantity calculation unit, an elasticity image creation unit, and a detection unit. In the ultrasound diagnostic device described in Patent Document 3, the physical quantity calculation unit calculates physical quantities related to elasticity in each part of the subject. In the ultrasound diagnostic device described in Patent Document 3, the elasticity image creation unit creates an elasticity image of the subject based on the physical quantities calculated by the physical quantity calculation unit. In the ultrasound diagnostic device described in Patent Document 3, the elasticity image creation unit creates an elasticity image based on the physical quantities using statistical features of the physical quantities calculated excluding liquid portions in the subject. In the ultrasound diagnostic device described in Patent Document 3, the detection unit detects liquid portions based on echo signals corresponding to ultrasound transmitted to the subject. In the ultrasound diagnostic device described in Patent Document 3, the detection unit is a fluid information acquisition unit that acquires fluid information based on the echo signals. In the ultrasound diagnostic device described in Patent Document 3, the fluid information is any of color Doppler data, B-flow data, or harmonic data included in the echo signals acquired based on the echo signals. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-047474 [Patent Document 2] International Publication No. 2022 / 196088 [Patent Document 3] Japanese Patent Application Laid-Open No. 2012-061075 Summary of the Invention

[0007] One embodiment of the present disclosure provides an image processing device, an endoscope system, an image processing method, and a program that can output highly accurate recognition results for an observation area that appears in an ultrasound image obtained using a B-mode method. [Means for solving the problem]

[0008] A first aspect of the present disclosure is an image processing device that includes a processor, which acquires an ultrasound image obtained by B-mode ultrasound diagnosis of a subject and showing an observation region of the subject, and blood flow distribution information obtained by Doppler ultrasound, which is used in conjunction with B-mode ultrasound diagnosis, and outputs a recognition result obtained by inputting the ultrasound image and blood flow distribution information into a trained model, thereby performing a recognition process that causes the trained model to recognize the observation region.

[0009] A second aspect of the present disclosure is the image processing device according to the first aspect, in which the observation target region is a lesion, and the recognition result is information that can distinguish between a lesion and a non-lesion.

[0010] A third aspect of the present disclosure is the image processing device according to the second aspect, in which the lesion is a cyst.

[0011] A fourth aspect of the present disclosure is an image processing device according to the second or third aspect, in which the trained model is obtained by performing machine learning to recognize lesions, or machine learning to recognize lesions and blood vessels shown in ultrasound images.

[0012] A fifth aspect of the present disclosure is the image processing device according to the first or second aspect, in which the observation target region is a blood vessel, and the recognition result is information that can distinguish between a blood vessel and something other than a blood vessel.

[0013] A sixth aspect of the present disclosure is an image processing device according to the fifth aspect, in which the trained model is obtained by performing machine learning to recognize blood vessels, or machine learning to recognize lesions appearing in ultrasound images and blood vessels appearing in ultrasound images.

[0014] A seventh aspect of the present disclosure is the image processing device according to any one of the first to sixth aspects, in which the blood flow distribution information is a map that allows the intensity of blood flow to be identified.

[0015] An eighth aspect of the present disclosure is an image processing device according to any one of the first to seventh aspects, wherein outputting the recognition result includes displaying the recognition result on a first screen.

[0016] A ninth aspect of the present disclosure is an image processing device according to any one of the first to eighth aspects, in which an ultrasound image is displayed on a second screen in the foreground and recognition processing is performed in the background.

[0017] A tenth aspect of the present disclosure is an image processing device according to any one of the first to ninth aspects, wherein, when recognition processing is being executed, the processor outputs information indicating that recognition processing is being executed.

[0018] An eleventh aspect of the present disclosure is an endoscopic system comprising an image processing device according to any one of the first to tenth aspects, and an ultrasound probe that emits ultrasound waves and detects reflected waves of the ultrasound waves while inserted into the body of a subject, and in which an ultrasound image and blood flow distribution information are generated based on the reflected waves detected by the ultrasound probe.

[0019] A twelfth aspect of the present disclosure is an image processing method that includes acquiring an ultrasound image obtained by a B-mode method in ultrasound diagnosis of a subject and showing an observation target region of the subject, and blood flow distribution information obtained by a Doppler method that is used in conjunction with the B-mode method in ultrasound diagnosis, and outputting a recognition result obtained by inputting the ultrasound image and the blood flow distribution information into a trained model, thereby performing a recognition process that causes the trained model to recognize the observation target region.

[0020] A thirteenth aspect of the present disclosure is a program for causing a computer to execute a process including acquiring an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and showing an observation region of the subject, and blood flow distribution information obtained by a Doppler method used in conjunction with the B-mode method in ultrasound diagnosis, and outputting a recognition result obtained by inputting the ultrasound image and the blood flow distribution information into a trained model, thereby executing a recognition process that causes the trained model to recognize the observation region. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an aspect in which an endoscope system is used. [Figure 2] 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system. [Figure 3] FIG. 1 is a block diagram showing an example of the configuration of an ultrasonic endoscope. [Figure 4] FIG. 10 is a conceptual diagram illustrating an example of processing content of a generation unit. [Figure 5] FIG. 10 is a conceptual diagram showing an example of a mode in which a cyst recognition model is generated by training a model with a group of training images. [Figure 6] FIG. 4 is a conceptual diagram illustrating an example of processing contents of a recognition unit and a control unit. [Figure 7] 10 is a flowchart illustrating an example of the flow of a diagnosis support process. [Figure 8] 10 is a flowchart illustrating an example of the flow of background processing. [Figure 9] 10 is a flowchart showing an example of the flow of foreground processing. [Figure 10] FIG. 10 is a conceptual diagram showing an example of a mode in which a blood vessel recognition model is generated by training a model with a group of training images. [Figure 11] FIG. 10 is a conceptual diagram showing a modified example of the processing contents of the recognition unit and the control unit. [Figure 12] 10 is a flowchart showing a modified example of the flow of the diagnostic support process. [Figure 13]FIG. 10 is a conceptual diagram showing an example of how information indicating that cyst recognition processing is being executed is displayed on a screen. [Figure 14] A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, examples of embodiments of an image processing device, an endoscope system, an image processing method, and a program according to the present disclosure will be described with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". NN is an abbreviation for "Neural Network". CNN is an abbreviation for "Convolutional neural network". R-CNN is an abbreviation for "Region based Convolutional Neural Network". YOLO is an abbreviation for "You only Look Once". RNN is an abbreviation for "Recurrent Neural Network".FCN is an abbreviation for "Fully Convolutional Network." FIFO is an abbreviation for "First In First Out."

[0025] As an example, as shown in Fig. 1, an endoscopic system 10 includes an ultrasonic endoscope 12 and a display device 14. The ultrasonic endoscope 12 is a convex type ultrasonic endoscope and includes an ultrasonic endoscope main body 16 and a processing device 18. In this embodiment, the endoscopic system 10 is an example of an "endoscopic system" according to the present disclosure. Also, in this embodiment, the processing device 18 is an example of an "image processing device" according to the present disclosure.

[0026] In this embodiment, a convex-type ultrasound endoscope is given as an example of the ultrasound endoscope 12, but this is merely an example, and the present disclosure also applies to a radial-type ultrasound endoscope. Also, in this embodiment, an ultrasound endoscope 12 is given as an example, but this is merely an example, and the present disclosure also applies to an extracorporeal ultrasound diagnostic device.

[0027] The endoscope system 10 is used by a user 20. An example of the user 20 is a doctor. The ultrasound endoscope main body 16 is connected to a processing device 18, and various signals are exchanged between the processing device 18 and the ultrasound endoscope main body 16. That is, the processing device 18 controls the operation of the ultrasound endoscope main body 16 by outputting signals to the ultrasound endoscope main body 16, and performs various signal processing on signals input from the ultrasound endoscope main body 16.

[0028] The ultrasound endoscope 12 is an endoscope used in ultrasound diagnosis of a subject 22 (e.g., a patient). When a user 20 observes an internal organ (e.g., an internal organ such as the pancreas or gallbladder) that is present inside the subject 22 during ultrasound diagnosis, the user 20 inserts the ultrasound endoscope main body 16 into the subject 22 through the mouth or nose (the mouth in the example shown in FIG. 1 ). With the ultrasound endoscope main body 16 inserted inside the subject 22, the ultrasound endoscope main body 16 emits ultrasound waves toward the internal organ at a predetermined position inside the subject 22 (e.g., the stomach or duodenum) and detects reflected waves obtained when the emitted ultrasound waves are reflected by the internal organ. The processing device 18 generates an ultrasound image 24 based on the reflected waves detected by the ultrasound endoscope main body 16 and outputs the image to the display device 14 or the like.

[0029] In the example shown in Figure 1, an upper gastrointestinal endoscopy is being performed, but the present disclosure is not limited to this and can also be applied to lower gastrointestinal endoscopy or bronchial endoscopy, etc.

[0030] The display device 14 displays various information including images under the control of the processing device 18. Examples of the display device 14 include a liquid crystal display and an EL display. An ultrasound image 24 generated by the processing device 18 is displayed as a moving image on a screen 26 of the display device 14. In the example shown in FIG. 1 , a cyst 28 and a blood vessel 29 are shown in the ultrasound image 24 displayed on the screen 26. In this embodiment, the screen 26 is an example of a "first screen" and a "second screen" according to the present disclosure.

[0031] 1 shows an example in which the ultrasound image 24 is displayed on the screen 26 of the display device 14, but this is merely an example and the ultrasound image 24 may be displayed on a display device other than the display device 14 (for example, the display of a tablet terminal). Also, the ultrasound image 24 may be stored in a computer-readable non-transitory storage medium (for example, a flash memory, a HDD, and / or a magnetic tape).

[0032] An image corresponding to the type of ultrasound diagnostic method, which is a diagnostic method using ultrasound, is displayed on the display device 14. The type of ultrasound examination method is selected by the user 20 or selected according to various conditions. Examples of the types of ultrasound examination methods include the B-mode (Brightness mode) method and the Doppler method (e.g., color Doppler method).

[0033] The B-mode method is an ultrasound diagnostic method that generates and displays a B-mode image. A B-mode image is obtained by converting the intensity of the reflected waves obtained when ultrasound waves are emitted into an internal body part and reflected from the internal body part into brightness on a screen 26, and refers to a two-dimensional tomographic image of a cross section parallel to the direction in which the ultrasound waves travel.

[0034] The Doppler method is an ultrasound diagnostic method that generates and displays Doppler images. A Doppler image is an image that uses reflected waves obtained by ultrasound emitted toward an internal body part and reflected from the internal body part to show the hemodynamics within blood vessels using color information. An image showing the hemodynamics within blood vessels is, for example, an image expressed using reddish and blueish colors. Reddish colors indicate blood flow toward the side from which the ultrasound is emitted and blood flow away from the side from which the ultrasound is emitted. When a reflected wave with a higher frequency than the ultrasound that hits the blood flow is obtained, it is identified as blood flow toward the side from which the ultrasound is emitted, while when a reflected wave with a lower frequency than the ultrasound that hits the blood flow is identified as blood flow away from the side from which the ultrasound is emitted. Within blood vessels, there are areas with fast and slow blood flow, and in Doppler images, the blood flow speed is expressed as a gradation using color mapping.

[0035] Depending on the ultrasound diagnostic method selected by the user 20, the screen 26 displays either a B-mode image or a Doppler image as the ultrasound image 24, or displays both a B-mode image and a Doppler image in parallel. In the example shown in FIG. 1 , a B-mode image is displayed on the screen 26 as the ultrasound image 24. The ultrasound image 24 is a moving image including a plurality of frames generated at a predetermined frame rate (e.g., 30 frames / second). The frame rate of the Doppler method is lower than the frame rate of the B-mode method. An example of the frame rate of the Doppler method (i.e., the frame rate at which the Doppler image is generated) is 15 frames / second. While a moving image is illustrated here, this is merely an example, and the present disclosure also applies even if the ultrasound image 24 is a still image. Note that, in the following, to facilitate understanding of the present disclosure, the description will be given assuming that the ultrasound image 24 is a B-mode image.

[0036] As an example, as shown in FIG. 2 , the ultrasound endoscope main body 16 includes a control section 27 and an insertion section 30. The insertion section 30 is tubular. The insertion section 30 has a distal end section 32, a bending section 34, and a flexible section 36. The distal end section 32, the bending section 34, and the flexible section 36 are arranged in this order from the distal end to the proximal end of the insertion section 30. The flexible section 36 is made of a long, flexible material and connects the control section 27 and the bending section 34. The bending section 34 partially bends or rotates around the axis of the insertion section 30 when the control section 27 is operated. As a result, the insertion section 30 is advanced into the hollow organ while bending and rotating around the axis of the insertion section 30 in accordance with the shape of the hollow organ (for example, the shape of the duodenal tract).

[0037] An ultrasonic probe 38 and a treatment tool opening 40 are provided at the tip portion 32. The ultrasonic probe 38 is provided on the distal end side of the tip portion 32. The ultrasonic probe 38 is a convex ultrasonic probe, which, when inserted into the body of the subject 22, emits ultrasonic waves to an internal region and receives reflected waves obtained when the emitted ultrasonic waves are reflected by the internal region.

[0038] The treatment tool opening 40 is formed closer to the base end of the distal end portion 32 than the ultrasonic probe 38. The treatment tool opening 40 is an opening for allowing a treatment tool 42 to protrude from the distal end portion 32. A treatment tool insertion port 44 is formed in the operation section 27, and the treatment tool 42 is inserted into the insertion section 30 from the treatment tool insertion port 44. The treatment tool 42 passes through the insertion section 30 and protrudes from the treatment tool opening 40 to the outside of the ultrasonic endoscope body 16. The treatment tool opening 40 also functions as a suction port for sucking blood, internal waste, etc.

[0039] 2, a puncture needle is shown as the treatment tool 42. However, this is merely an example, and the treatment tool 42 may also be grasping forceps and / or a cannula, etc.

[0040] The tip portion 32 is provided with an illumination device 46 and a camera 48. The illumination device 46 emits light. Examples of the light emitted from the illumination device 46 include visible light (e.g., white light), invisible light (e.g., near-infrared light), and / or special light. Examples of the special light include light for BLI and / or light for LCI.

[0041] The camera 48 optically captures images of the inside of a hollow organ. A CMOS image sensor is used for the camera 48. The CMOS image sensor is merely an example, and other types of image sensors such as a CCD image sensor may also be used. The images captured by the camera 48 may be displayed on the display device 14, on a display device other than the display device 14 (for example, the display of a tablet terminal), stored in a storage medium (for example, a flash memory, a HDD, and / or a magnetic tape), or saved in an electronic medical record.

[0042] The ultrasonic endoscope 12 includes a processing device 18 and a universal cord 50. The universal cord 50 has a base end 50A and a tip end 50B. The base end 50A is connected to the operation unit 27. The tip end 50B is connected to the processing device 18. In other words, the ultrasonic endoscope main body 16 and the processing device 18 are connected via the universal cord 50.

[0043] The endoscope system 10 includes a reception device 52. The reception device 52 is connected to the processing device 18. The reception device 52 receives instructions from a user. Examples of the reception device 52 include an operation panel having a plurality of hard keys and / or a touch panel, a keyboard, a mouse, a trackball, a foot switch, a smart device, and / or a microphone.

[0044] The processing device 18 performs various signal processing operations and transmits and receives various signals to and from the ultrasonic endoscope main body 16, etc., in accordance with instructions received by the receiving device 52. For example, the processing device 18 causes the ultrasonic probe 38 to emit ultrasonic waves in accordance with instructions received by the receiving device 52, and generates and outputs an ultrasonic image 24 (see FIG. 1) based on the reflected waves received by the ultrasonic probe 38.

[0045] The display device 14 is also connected to the processing device 18. The processing device 18 controls the display device 14 in accordance with instructions received by the reception device 52. As a result, for example, an ultrasound image 24 generated by the processing device 18 is displayed on the screen 26 of the display device 14 (see FIG. 1).

[0046] 3, the processing device 18 includes a computer 54, an input / output interface 56, a transceiver circuit 58, and a communication module 60. The computer 54 is an example of a "computer" according to the present disclosure.

[0047] The computer 54 includes a processor 62, a memory 64, and a storage 66. The input / output interface 56, the processor 62, the memory 64, and the storage 66 are connected to a bus 68.

[0048] The processor 62 controls the entire processing device 18. For example, the processor 62 has a CPU and a GPU, and the GPU operates under the control of the CPU and is mainly responsible for executing image processing. The processor 62 may be one or more CPUs that integrate GPU functionality, or one or more CPUs that do not integrate GPU functionality. The processor 62 may also include a multi-core CPU or a TPU. In this embodiment, the processor 62 is an example of a "processor" according to the present disclosure.

[0049] The memory 64 is a memory in which information is temporarily stored, and is used as a work memory by the processor 62. An example of the memory 64 is a RAM. The storage 66 is a non-volatile storage device that stores various programs, various parameters, etc. An example of the storage 66 is a flash memory (for example, an EEPROM) and / or an SSD. Note that the flash memory and the SSD are merely examples, and the storage 66 may be another non-volatile storage device such as an HDD, or may be a combination of two or more types of non-volatile storage devices.

[0050] The input / output interface 56 is connected to the accepting device 52, and the processor 62 acquires instructions accepted by the accepting device 52 via the input / output interface 56 and executes processing according to the acquired instructions.

[0051] A transmission / reception circuit 58 is connected to the input / output interface 56. The transmission / reception circuit 58 generates an ultrasonic emission signal 70 having a pulse waveform in accordance with instructions from the processor 62 and outputs the signal to the ultrasonic probe 38. The ultrasonic probe 38 converts the ultrasonic emission signal 70 input from the transmission / reception circuit 58 into ultrasonic waves and emits the ultrasonic waves toward an organ 72 (e.g., the pancreas) serving as an internal body part of the subject 22. The ultrasonic probe 38 receives reflected waves obtained when the ultrasonic waves radiated toward the organ 72 are reflected by the organ 72, converts the reflected waves into reflected wave signals 74, which are electrical signals, and outputs the reflected waves to the transmission / reception circuit 58. The transmission / reception circuit 58 digitizes the reflected wave signals 74 input from the ultrasonic probe 38 and outputs the digitized reflected wave signals 74 to the processor 62 via the input / output interface 56. The processor 62 generates an ultrasonic image 24 (see FIG. 1 ) showing the state of the organ 72 based on the reflected wave signals 74 input from the transmission / reception circuit 58 via the input / output interface 56. In parallel with the generation of the ultrasound image 24, the processor 62 also generates a Doppler image based on the reflected wave signal 74.

[0052] Although not shown in Fig. 3, the lighting device 46 (see Fig. 2) is also connected to the input / output interface 56. The processor 62 controls the lighting device 46 via the input / output interface 56 to change the type of light emitted from the lighting device 46 and adjust the amount of light. Although not shown in Fig. 3, the camera 48 (see Fig. 2) is also connected to the input / output interface 56. The processor 62 controls the camera 48 via the input / output interface 56 and acquires, via the input / output interface 56, images of the inside of the body of the subject 22 captured by the camera 48.

[0053] A communication module 60 is connected to the input / output interface 56. The communication module 60 is an interface including a communication processor, an antenna, etc. The communication module 60 is connected to a network (not shown) such as a LAN or WAN, and controls communication with external devices (for example, a server, a personal computer, and / or a tablet terminal, etc.).

[0054] The display device 14 is connected to the input / output interface 56, and the processor 62 controls the display device 14 via the input / output interface 56 to cause the display device 14 to display various information.

[0055] The input / output interface 56 is connected to the accepting device 52, and the processor 62 acquires instructions accepted by the accepting device 52 via the input / output interface 56 and executes processing according to the acquired instructions.

[0056] The storage 66 stores a diagnostic assistance program 76 and a cyst recognition model 78. The processor 62 reads the diagnostic assistance program 76 from the storage 66 and executes the read diagnostic assistance program 76 on the memory 64 to perform diagnostic assistance processing. The diagnostic assistance processing is processing that executes recognition processing that recognizes an observation target region from the organ 72 using an AI method and assists the user 20 (see FIG. 1 ) in making a diagnosis based on the recognition results of the recognition processing. The diagnostic assistance processing is realized by the processor 62 operating as a generation unit 62A, a recognition unit 62B, and a control unit 62C in accordance with the diagnostic assistance program 76 executed on the memory 64.

[0057] In this embodiment, the diagnostic assistance program 76 is an example of a "program" according to the present disclosure. The cyst recognition model 78 is a trained model used in AI processing to recognize cysts 28 appearing in ultrasound images 24. In this embodiment, the cyst recognition model 78 is an example of a "trained model" according to the present disclosure.

[0058] As an example, as shown in Fig. 4, the generator 62A acquires a reflected wave signal 74 from the transmission / reception circuit 58 and generates an ultrasound image 24 by the B-mode method based on the acquired reflected wave signal 74. That is, the generator 62A generates a B-mode image based on the reflected wave signal 74 as the ultrasound image 24. The B-mode image generated by the generator 62A as the ultrasound image 24 is an image that shows a cross section of an organ 72 (see Fig. 3) in two dimensions. In the example shown in Fig. 4, the generator 62A generates an ultrasound image 24 that shows a cyst 28 and a blood vessel 29, but of course, a B-mode image that does not show the cyst 28 and / or the blood vessel 29 may also be generated as the ultrasound image 24.

[0059] In this embodiment, the ultrasound image 24 generated by the generation unit 62A is an example of an "ultrasound image" according to the present disclosure. Also, in this embodiment, the cyst 28 shown in the ultrasound image 24 is an example of an "observation region of the subject," a "lesion," and a "cyst" according to the present disclosure. Also, in this embodiment, the blood vessel 29 shown in the ultrasound image 24 is an example of a "blood vessel" according to the present disclosure.

[0060] In this embodiment, the B-mode method and the Doppler method are used in combination in the ultrasound diagnosis of the subject 22. That is, the generation unit 62A generates a Doppler image 88 in parallel with the ultrasound image 24. Specifically, the generation unit 62A acquires the reflected wave signal 74 used in generating the ultrasound image 24 from the transmission / reception circuit 58, and generates the Doppler image 88 by the Doppler method based on the acquired reflected wave signal 74. The frame rate of the ultrasound image 24 is higher than the frame rate of the Doppler image 88, and the ultrasound image 24 and the Doppler image 88 are linked in units of a predetermined number of frames of the ultrasound image 24 (for example, in units of 15 frames of the ultrasound image 24). An example of the frame rate of the ultrasound image 24 is 30 frames per second. An example of the frame rate of the Doppler image 88 is 15 frames per second. Note that this is merely an example, and the frame rate of the ultrasound image 24 may be the same as the frame rate of the Doppler image 88.

[0061] The Doppler image 88 is an image obtained by overlaying a color map 90, which represents characteristics (here, as an example, the distribution of blood flow) within the organ 72 using colors (i.e., chromatic colors) on the ultrasound image 24. The color map 90 is information indicating blood flow dynamics determined using the Doppler effect. In the color map 90, the intensity of blood flow is represented by color. In this embodiment, the concept of blood flow intensity also includes the strength, speed, and / or momentum of blood flow. The color map 90 has a blood flow region 90A indicating an area where blood flow is present and a blood-free region 90B indicating an area where blood flow is absent. The blood flow intensity can be determined from the color assigned to the blood flow region 90A. Therefore, the image obtained by overlaying the color map 90 on the ultrasound image 24, i.e., the Doppler image 88, can also be considered a map from which the intensity of blood flow can be determined. In this embodiment, the Doppler image 88 is an example of "blood flow distribution information" and "map" according to the present disclosure.

[0062] As an example, as shown in FIG. 5, a cyst recognition model 78 is a trained model generated by performing machine learning on a model 92. The cyst recognition model 78 is obtained by performing machine learning to recognize a cyst 28 (see FIG. 1) captured in an ultrasound image 24 (see FIG. 1). A training image group 94 is used as training data for the machine learning performed on the model 92. The training image group 94 is a group of images in which a plurality of different B-mode images 94A are linked one-to-one with a plurality of different Doppler images 94B corresponding to the B-mode images 94A. That is, in the training image group 94, the B-mode image 94A is associated with the Doppler image 94B used in combination with the B-mode image 94A. An example of the Doppler image 94B used in combination with the B-mode image 94A is an image obtained by superimposing a color map showing the blood flow dynamics of a region captured in the B-mode image 94A (in other words, a map that allows the intensity of blood flow to be identified by color) on the B-mode image 94A. Examples of B-mode images 94A and Doppler images 94B linked to each other include a sample B-mode image and a sample Doppler image obtained by actual ultrasound diagnosis, and / or a sample B-mode image and a sample Doppler image generated by so-called generation AI.

[0063] An example of the model 92 is a mathematical model using a neural network (NN). Examples of the type of NN include YOLO, R-CNN, and FCN. The NN used in the model 92 may be, for example, a combination of YOLO, R-CNN, or FCN with an RNN. An RNN is suitable for machine learning of multiple images obtained in a time series. Note that the types of NN listed here are merely examples, and other types of NNs that enable object detection by machine learning images may also be used.

[0064] The multiple B-mode images 94A show at least the cyst 84 out of the cyst 84 and the blood vessel 86. An annotation 98 is added to the B-mode image 94A. The annotation 98 is information that can identify the position where the cyst 84 is shown in the B-mode image 94A (for example, information including multiple coordinates that can identify the position of a rectangular frame circumscribing the cyst 84).

[0065] Here, for convenience of explanation, information capable of identifying the position where the cyst 84 is shown in the B-mode image 94A is shown as an example of the annotation 98, but this is merely an example. For example, the annotation 98 may include various information regarding the lesion shown in the B-mode image 94A, such as information capable of identifying the type of lesion shown in the B-mode image 94A, the importance of the lesion, and / or the size of the lesion. Examples of lesions include types of lesions other than cysts (e.g., tumors).

[0066] For ease of explanation, the processing using cyst recognition model 78 will be described below as processing that is actively performed mainly by cyst recognition model 78. That is, for ease of explanation, cyst recognition model 78 will be described as a function that processes input information and outputs the processing results. Also, for ease of explanation, part of the processing for training model 92 will be described below as processing that is actively performed mainly by model 92. That is, for ease of explanation, model 92 will be described as a function that processes input information and outputs the processing results.

[0067] The model 92 receives as example data a B-mode image 94A and a Doppler image 94B that are linked to each other and are included in the training image group 82. In response to this, the model 92 recognizes the position where the cyst 84 appears in the B-mode image 94A based on the input B-mode image 94A and Doppler image 94B (in other words, predicts the position where the cyst 84 appears in the B-mode image 94A). The model 92 then outputs a recognition result (in other words, a prediction result). The recognition result includes information that can identify the position recognized by the model 92 as the position where the cyst 84 appears in the B-mode image 94A. An example of the information that can identify the position recognized by the model 92 is information that includes a plurality of coordinates that can identify the position of a bounding box surrounding an area recognized as the position where the cyst 84 exists (i.e., the position of the bounding box in the B-mode image 94A).

[0068] The model 92 is adjusted according to the error between the annotations 98 assigned as correct data to the B-mode images 94A input to the model 92 and the recognition results output from the model 92. That is, the model 92 is optimized by adjusting a plurality of optimization variables (for example, a plurality of connection weights and a plurality of offset values) in the model 92 so as to minimize the error, thereby generating the cyst recognition model 78. That is, the data structure of the cyst recognition model 78 is obtained by having the model 92 learn a plurality of different B-mode images 94A assigned with the annotations 98 and a plurality of Doppler images 94B corresponding to these images.

[0069] 6 , the recognition unit 62B acquires from the generation unit 62A the ultrasound image 24 and the Doppler image 88 that correspond to each other and that were generated by the generation unit 62A, and executes a cyst recognition process 100 on the acquired ultrasound image 24 and the Doppler image 88. The cyst recognition process 100 is a process in which the ultrasound image 24 and the Doppler image 88 that correspond to each other and that were acquired from the generation unit 62A are input to the cyst recognition model 78, thereby causing the cyst recognition model 78 to recognize a cyst 28. In this embodiment, the cyst recognition process 100 is an example of a "recognition process" according to the present disclosure.

[0070] By executing the cyst recognition process 100, a cyst recognition result 102 is generated by the cyst recognition model 78. The cyst recognition result 102 includes information that can identify the position of the cyst 28 shown in the ultrasound image 24.

[0071] The control unit 62C acquires the cyst recognition result 102 generated by the cyst recognition model 78. The control unit 62C reflects the cyst recognition result 102 on the ultrasound image 24 used in the cyst recognition process 100. In this case, for example, the cyst recognition result 102 is reflected on the ultrasound image 24 as a bounding box BB1. The control unit 62C superimposes the bounding box BB1 on the location in the ultrasound image 24 where the cyst 28 appears. In the example shown in FIG. 6, the bounding box BB1 is superimposed on the ultrasound image 24 as a circumscribing box for the cyst 28 appearing in the ultrasound image 24.

[0072] The control unit 62C outputs the ultrasound image 24 in which a bounding box BB1 is superimposed on the portion of the ultrasound image 24 in which the cyst 28 is shown. In the example shown in Fig. 6, the control unit 62C displays the ultrasound image 24 on which the bounding box BB1 is superimposed on the screen 26. That is, the bounding box BB1 superimposed on the ultrasound image 24 is displayed on the screen 26 as information that enables the cyst 28 to be distinguished from other parts of the body than the cyst 28.

[0073] Here, an example of displaying the ultrasound image 24 superimposed with the bounding box BB1 on the screen 26 has been given as an example of outputting the ultrasound image 24 superimposed with the bounding box BB1, but the present disclosure is not limited to this. For example, the ultrasound image 24 superimposed with the bounding box BB1 may be stored in a storage area (storage 66, a server, a personal computer, a tablet terminal, an electronic medical record, etc.), or may be recorded on a medium (e.g., paper) by a printer (not shown). Furthermore, the position of the bounding box BB1 within the ultrasound image 24 (i.e., the position of the cyst 28) may be announced by voice via a speaker (not shown).

[0074] In this embodiment, the bounding box BB1 superimposed on the ultrasound image 24 is an example of a "recognition result" according to the present disclosure. Also, in this embodiment, the bounding box BB1 superimposed on the ultrasound image 24 displayed on the screen 26 is an example of "information capable of distinguishing between a lesion and a non-lesion" according to the present disclosure.

[0075] Next, the operation of the endoscope system 10 will be described with reference to FIG.

[0076] 7 shows an example of the flow of diagnostic support processing performed by the processor 62 of the processing device 18. The flow of diagnostic support processing shown in FIG. 7 is an example of the "image processing method" according to the present disclosure.

[0077] 7, first, in step ST10, the generation unit 62A controls the ultrasound probe 38 via the transmission / reception circuitry 58 to cause the ultrasound probe 38 to emit ultrasound waves toward the organ 72 (here, the pancreas, as an example). After the processing of step ST10 is executed, the diagnostic support processing proceeds to step ST12.

[0078] In step ST12, the generation unit 62A determines whether or not a reflected wave based on the ultrasound emitted as a result of the processing of step ST10 has been detected by the ultrasound probe 38. If, in step ST12, the ultrasound probe 38 has not detected a reflected wave based on the ultrasound emitted as a result of the processing of step ST10, the determination is negative, and the diagnostic support processing proceeds to step ST28. If, in step ST12, the ultrasound probe 38 has detected a reflected wave based on the ultrasound emitted as a result of the processing of step ST10, the determination is positive, and the diagnostic support processing proceeds to step ST14.

[0079] In step ST14, the generation unit 62A generates the ultrasound image 24 and the Doppler image 88 based on the reflected waves detected by the ultrasound probe 38. After the processing of step ST14 is executed, the diagnostic support processing proceeds to step ST16.

[0080] In step ST16, the recognition unit 62B inputs the ultrasound image 24 and Doppler image 88 generated in step ST14 into the cyst recognition model 78. After the processing of step ST16 is executed, the diagnosis support processing proceeds to step ST18.

[0081] In step ST18, the recognition unit 62B determines whether or not the ultrasound image 24 and the Doppler image 88 have been input into the cyst recognition model 78, resulting in the cyst recognition model 78 recognizing that a cyst 28 is captured in the ultrasound image 24. If the ultrasound image 24 is not recognized as capturing a cyst 28 in step ST18, the determination is negative, and the diagnostic support processing proceeds to step ST24. If the ultrasound image 24 is recognized as capturing a cyst 28 in step ST18, the determination is positive, and the diagnostic support processing proceeds to step ST20.

[0082] In step ST20, the recognition unit 62B superimposes the cyst recognition result 102 as a bounding box BB1 on the ultrasound image 24 generated in step ST14. After the processing of step ST20 is executed, the diagnosis support processing proceeds to step ST22.

[0083] In step ST22, the control unit 62C displays the ultrasound image 24 with the bounding box BB1 superimposed on the screen 26. That is, when the determination in step ST18 is negative, the processing of step ST26 is executed, and the ultrasound image 24 generated in step ST14 is displayed on the screen 26. Furthermore, when the processing of step ST20 is executed and the diagnostic support processing proceeds to step ST22, the processing of step ST22 is executed, and the ultrasound image 24 with the bounding box BB1 superimposed on the screen 26 is displayed. After the processing of step ST22 is executed, the diagnostic support processing proceeds to step ST24.

[0084] In step ST24, the control unit 62C determines whether or not the condition for terminating the diagnostic support processing has been satisfied. One example of the condition for terminating the diagnostic support processing is that an instruction to terminate the diagnostic support processing has been accepted by the acceptance device 52. If the condition for terminating the diagnostic support processing has not been satisfied in step ST24, the determination is denied, and the diagnostic support processing proceeds to step ST10. If the condition for terminating the diagnostic support processing has been satisfied in step ST24, the diagnostic support processing is terminated.

[0085] As described above, in the endoscope system 10, the ultrasound image 24 obtained by the B-mode method and the Doppler image 88 obtained by the Doppler method used in combination with the B-mode method are input to the cyst recognition model 78, causing the cyst recognition model 78 to recognize the cyst 28 appearing in the ultrasound image 24. The cyst recognition result 102 generated by the cyst recognition model 78 is superimposed as a bounding box BB1 on the location in the ultrasound image 24 where the cyst 28 appears. Then, the ultrasound image 24 with the bounding box BB1 superimposed on the location in the ultrasound image 24 where the cyst 28 appears is displayed on the screen 26.

[0086] This allows the user 20 to visually recognize an image obtained by the B-mode method, i.e., a highly accurate recognition result (cyst recognition result 102 in the example shown in FIG. 6 ) for the cyst 82 shown in the ultrasound image 24, compared to when the trained model is made to recognize the cyst 82 shown in the ultrasound image 24 without using the Doppler image 88. This means that the user 20 can visually distinguish between a portion of the ultrasound image 24 in which the cyst 28 is shown and a portion other than the cyst 28 through the bounding box BB1. This also means that when the ultrasound image 24 shows a mixture of the cyst 28 and the blood vessel 29, the user 20 can visually distinguish between the cyst 28 and the blood vessel 29 (in other words, it is possible to prevent the blood vessel 29 from being mistakenly recognized as the cyst 28 and vice versa).

[0087] In the above embodiment, an example in which the diagnostic support processing is executed by sequential processing has been described, but the diagnostic support processing may also be realized by parallel processing. For example, as shown in Figures 8 and 9, background processing and foreground processing may be executed in parallel to realize processing similar to the diagnostic support processing described in the above embodiment. The background processing refers to processing that occurs in the background, and the foreground processing refers to processing that occurs in the foreground. For example, in the background processing, a cyst recognition processing 100 is executed, and in the foreground processing, an ultrasound image 24 is displayed on the screen 26.

[0088] An example of the flow of background processing will now be described with reference to FIG.

[0089] The background processing shown in FIG. 8 differs from the diagnostic support processing shown in FIG. 7 in that it includes processing in steps ST100 and ST102 instead of processing in steps ST22 and ST24.

[0090] In step ST100 shown in FIG. 8, the control unit 62C stores the ultrasound image 24 in the memory 64. The ultrasound image 24 is stored in the memory 64 using the FIFO method. The memory 64 also serves as a buffer memory capable of storing a plurality of ultrasound images 24 in chronological order. When the determination in step ST18 is negative, the processing of step ST100 is executed, whereby the ultrasound image 24 generated in step ST14 is stored in the memory 64. Furthermore, when the diagnostic support processing proceeds to step ST100 after the processing of step ST22 is executed, the processing of step ST100 is executed, whereby the ultrasound image 24 on which the bounding box BB1 is superimposed is stored in the memory 64. After the processing of step ST100 is executed, the diagnostic support processing proceeds to step ST102.

[0091] In step ST102, the control unit 62C determines whether or not the conditions for ending the background processing are satisfied. One example of the conditions for ending the background processing is that an instruction to end the background processing is accepted by the acceptance device 52. If the conditions for ending the background processing are not satisfied in step ST102, the determination is negative, and the background processing proceeds to step ST10. If the conditions for ending the background processing are satisfied in step ST102, the background processing is terminated.

[0092] Next, an example of the flow of foreground processing will be described with reference to FIG.

[0093] 9, the control unit 62C determines whether or not a new ultrasound image 24 is stored in the memory 64. If a new ultrasound image 24 is not stored in the memory 64 in step ST200, the determination is negative, and the foreground processing proceeds to step ST202.

[0094] In step ST202, the control unit 62C acquires an ultrasound image 24 stored in the memory 64 (for example, the ultrasound image 24 that was stored most recently among the multiple ultrasound images 24 stored in the memory 64 in a FIFO manner), and displays the acquired ultrasound image 24 on the screen 26. After the processing of step ST202 is executed, the foreground processing proceeds to step ST204.

[0095] In step ST204, control unit 62C determines whether or not the conditions for terminating the foreground processing are satisfied. One example of the conditions for terminating the foreground processing is that an instruction to terminate the foreground processing is accepted by acceptance device 52. If the conditions for terminating the foreground processing are not satisfied in step ST204, the determination is negative, and the foreground processing proceeds to step ST200. If the conditions for terminating the foreground processing are satisfied in step ST204, the foreground is terminated.

[0096] 8 and 9, the cyst recognition process 100 is executed in the background, and the ultrasound image 24 is displayed in the foreground on the screen 26. This allows the user 20 to view the ultrasound image 24 without being aware that the cyst recognition process 100 is being executed.

[0097] In the above embodiment, an example in which the cyst recognition model 78 recognizes the cyst 28 has been given, but this is merely an example. For example, as shown in Figs. 10 to 12, the blood vessel recognition model 104 may be made to recognize the blood vessel 29.

[0098] To realize this embodiment, a blood vessel recognition model 104 shown in Fig. 10 is used instead of the cyst recognition model 78. In the example shown in Fig. 10, the blood vessel recognition model 104 is an example of a "trained model" according to the present disclosure.

[0099] The blood vessel recognition model 104 is obtained by performing machine learning to recognize blood vessels 29 (see FIG. 1) shown in an ultrasound image 24 (see FIG. 1). As an example, as shown in FIG. 10, the blood vessel recognition model 104 is generated by performing machine learning on a model 106. The model 106 has the same data structure as the model 92.

[0100] In the machine learning performed on the model 106, a training image group 108 is used as training data. The training image group 108 is an image group in which a plurality of different B-mode images 108A are linked one-to-one with a plurality of different Doppler images 108B that correspond to these B-mode images 108A. That is, in the training image group 108, the B-mode images 108A are associated with the Doppler images 108B that are used in combination with the B-mode images 108A. An example of the Doppler images 108B that are used in combination with the B-mode images 108A is an image obtained by superimposing a color map showing the blood flow dynamics of a region captured in the B-mode image 108A (in other words, a map that allows the intensity of blood flow to be identified by color) on the B-mode image 108A. Examples of B-mode image 108A and Doppler image 108B linked to each other include a sample B-mode image and a sample Doppler image obtained by actual ultrasound diagnosis, and / or a sample B-mode image and a sample Doppler image generated by so-called generation AI.

[0101] The multiple B-mode images 108A show at least the blood vessel 86 out of the cyst 84 and the blood vessel 86. Annotation 110 is added to the B-mode images 108A. The annotation 110 is information that can identify the position where the blood vessel 86 is shown in the B-mode images 108A (for example, information including multiple coordinates that can identify the position of a rectangular frame circumscribing the blood vessel 86).

[0102] Here, for convenience of explanation, information capable of identifying the position where the blood vessel 86 is shown in the B-mode image 108A is illustrated as an example of the annotation 110, but this is merely an example. For example, the annotation 110 may include various information related to the blood vessel 86 shown in the B-mode image 108A, such as information capable of identifying the type of blood vessel 86 shown in the B-mode image 108A, the importance of the blood vessel 86, and / or the size of the blood vessel 86.

[0103] For ease of explanation, the processing using the blood vessel recognition model 104 will be described below as processing that is actively performed mainly by the blood vessel recognition model 104. That is, for ease of explanation, the blood vessel recognition model 104 will be described as a function that processes input information and outputs processing results. Also, for ease of explanation, part of the processing for training the model 106 will be described below as processing that is actively performed mainly by the model 106. That is, for ease of explanation, the model 106 will be described as a function that processes input information and outputs processing results.

[0104] The model 106 receives as input a B-mode image 108A and a Doppler image 108B included in the training image set 108. In response to this, the model 106 recognizes the position where the blood vessel 86 appears in the B-mode image 108A from the input B-mode image 108A and Doppler image 108B (in other words, predicts the position where the blood vessel 86 appears in the B-mode image 108A). The model 106 then outputs a recognition result (in other words, a prediction result). The recognition result includes information that can identify the position recognized by the model 106 as the position where the blood vessel 86 appears in the B-mode image 108A. An example of the information that can identify the position recognized by the model 106 is information that includes a plurality of coordinates that can identify the position of a bounding box surrounding an area recognized as the position where the blood vessel 86 exists (i.e., the position of the bounding box in the B-mode image 108A).

[0105] The model 106 is adjusted according to the error between the annotation 110 assigned as correct data to the B-mode image 108A input to the model 106 and the recognition result output from the model 106. That is, the model 106 is optimized by adjusting a plurality of optimization variables (e.g., a plurality of connection weights and a plurality of offset values) in the model 106 so as to minimize the error, thereby generating the blood vessel recognition model 104. That is, the data structure of the blood vessel recognition model 104 is obtained by having the model 106 learn a plurality of different B-mode images 108A assigned with the annotation 110 and a plurality of Doppler images 108B corresponding to these images.

[0106] 11 , the recognition unit 62B acquires from the generation unit 62A an ultrasound image 24 and a Doppler image 88 corresponding to each other and generated by the generation unit 62A, and executes a blood vessel recognition process 112 on the acquired ultrasound image 24 and Doppler image 88. The blood vessel recognition process 112 is a process of inputting the ultrasound image 24 and the Doppler image 88 corresponding to each other and acquired from the generation unit 62A to the blood vessel recognition model 104, thereby causing the blood vessel recognition model 104 to recognize a blood vessel 29. In the example shown in FIG. 11 , the blood vessel recognition process 112 is an example of a "recognition process" according to the present disclosure. Furthermore, in the example shown in FIG. 11 , the blood vessel 29 is an example of an "observation target region" and a "blood vessel" according to the present disclosure.

[0107] By executing the blood vessel recognition process 112, the blood vessel recognition model 104 generates a blood vessel recognition result 114. The blood vessel recognition result 114 includes information that can identify the position of the blood vessel 29 shown in the ultrasound image 24.

[0108] The control unit 62C acquires the blood vessel recognition result 114 generated by the blood vessel recognition model 104. The control unit 62C reflects the blood vessel recognition result 114 on the ultrasound image 24 used in the blood vessel recognition process 112. In this case, for example, the blood vessel recognition result 114 is reflected on the ultrasound image 24 as a bounding box BB2. The control unit 62C superimposes the bounding box BB2 on a portion of the ultrasound image 24 where a blood vessel 29 is shown. In the example shown in FIG. 11 , the bounding box BB2 is superimposed on the ultrasound image 24 as a circumscribing frame for the blood vessel 29 shown in the ultrasound image 24.

[0109] The control unit 62C outputs the ultrasound image 24 in which a bounding box BB2 is superimposed on a portion of the ultrasound image 24 in which the blood vessels 29 are shown. In the example shown in Fig. 11, the ultrasound image 24 on which the bounding box BB2 is superimposed is displayed on the screen 26 by the control unit 62C. That is, the bounding box BB2 superimposed on the ultrasound image 24 is displayed on the screen 26 as information that enables the blood vessels 29 to be distinguished from other parts of the body than the blood vessels 29.

[0110] Here, an example of displaying the ultrasound image 24 superimposed with the bounding box BB2 on the screen 26 has been given as an example of outputting the ultrasound image 24 superimposed with the bounding box BB2, but the present disclosure is not limited to this. For example, the ultrasound image 24 superimposed with the bounding box BB2 may be stored in a storage area (storage 66, a server, a personal computer, a tablet terminal, an electronic medical record, etc.), or may be recorded on a medium (e.g., paper) by a printer (not shown). Furthermore, the position of the bounding box BB2 within the ultrasound image 24 (i.e., the position of the blood vessel 29) may be announced by voice via a speaker (not shown).

[0111] In this embodiment, the bounding box BB2 superimposed on the ultrasound image 24 is an example of a "recognition result" according to the present disclosure. Also, in this embodiment, the bounding box BB2 superimposed on the ultrasound image 24 displayed on the screen 26 is an example of "information capable of distinguishing between blood vessels and non-blood vessels" according to the present disclosure.

[0112] 12 shows a modified example of the flow of the diagnostic support processing performed by the processor 62 of the processing device 18, that is, an example of the flow of processing for recognizing blood vessels 29 and suppressing cysts 28 from being erroneously recognized as blood vessels 29. The flow of the diagnostic support processing shown in FIG. 12 is an example of the "image processing method" according to the present disclosure.

[0113] The flowchart shown in FIG. 12 differs from the flowchart shown in FIG. 7 in that it includes processes of steps ST300 to ST306 instead of the processes of steps ST16 to ST22.

[0114] 12, the recognition unit 62B inputs the ultrasound image 24 and the Doppler image 88 generated in step ST14 to the blood vessel recognition model 104. After the processing of step ST300 is executed, the diagnostic support processing proceeds to step ST302.

[0115] In step ST302, the recognition unit 62B determines whether or not the ultrasound image 24 and the Doppler image 88 have been input to the blood vessel recognition model 104, resulting in the blood vessel recognition model 104 recognizing that a blood vessel 29 is shown in the ultrasound image 24. If the blood vessel 29 is not recognized as being shown in the ultrasound image 24 in step ST302, the determination is negative, and the diagnostic support processing proceeds to step ST24. If the blood vessel 29 is recognized as being shown in the ultrasound image 24 in step ST302, the determination is positive, and the diagnostic support processing proceeds to step ST304.

[0116] In step ST304, the recognition unit 62B superimposes the blood vessel recognition result 114 as a bounding box BB2 on the ultrasound image 24 generated in step ST14. After the processing of step ST304 is executed, the diagnosis support processing proceeds to step ST306.

[0117] In step ST306, the control unit 62C displays the ultrasound image 24 with the bounding box BB2 superimposed on the screen 26. That is, when the determination in step ST302 is negative, the processing of step ST306 is executed, and the ultrasound image 24 generated in step ST14 is displayed on the screen 26. Furthermore, when the diagnostic support processing proceeds to step ST306 after the processing of step ST304 is executed, the processing of step ST306 is executed, and the ultrasound image 24 with the bounding box BB2 superimposed on the screen 26 is displayed. After the processing of step ST306 is executed, the diagnostic support processing proceeds to step ST24.

[0118] 10 to 12, in the endoscope system 10, the ultrasound image 24 obtained by the B-mode method and the Doppler image 88 obtained by the Doppler method used in combination with the B-mode method are input to the blood vessel recognition model 104, causing the blood vessel recognition model 104 to recognize the blood vessels 29 appearing in the ultrasound image 24. The blood vessel recognition result 114 generated by the blood vessel recognition model 104 is superimposed as a bounding box BB2 on the location in the ultrasound image 24 where the blood vessels 29 appear. Then, the ultrasound image 24 with the bounding box BB2 superimposed on the location in the ultrasound image 24 where the blood vessels 29 appear is displayed on the screen 26.

[0119] This allows the user 20 to visually recognize an image obtained by the B-mode method, i.e., a highly accurate recognition result (blood vessel recognition result 114 in the example shown in FIG. 11 ) of the blood vessels 29 shown in the ultrasound image 24, compared to when the trained model is made to recognize the blood vessels 29 shown in the ultrasound image 24 without using the Doppler image 88. This means that the user 20 can visually distinguish between areas in which the blood vessels 29 appear and areas other than the blood vessels 29 in the ultrasound image 24 through the bounding box BB2. This also means that when a mixture of cysts 28 and blood vessels 29 appears in the ultrasound image 24, it is possible to contribute to allowing the user 20 to visually distinguish between the cysts 28 and the blood vessels 29 (in other words, it is possible to prevent the blood vessels 29 from being mistakenly recognized as cysts 28 and vice versa).

[0120] Although an example has been given in which the diagnostic support processing shown in FIG. 12 is executed instead of the diagnostic support processing shown in FIG. 7, this is merely one example. For example, the diagnostic support processing shown in FIG. 7 and the diagnostic support processing shown in FIG. 12 may be executed in parallel. In this case, the bounding boxes BB1 and BB2 may be displayed on the screen 26 in a visually distinguishable manner (e.g., color, brightness, thickness, and / or line type). Furthermore, the bounding boxes BB1 and BB2 may be displayed in a manner that allows the respective meanings of the bounding boxes BB1 and BB2 to be identified. For example, the bounding boxes BB1 and BB2 may be displayed in a manner that allows the distinguishing fact that the bounding box BB1 is a bounding box that allows the position of the cyst 28 to be identified, and the bounding box BB2 is a bounding box that allows the position of the blood vessel 29 to be identified. In addition, information (e.g., an identifier) ​​that visually identifies that bounding box BB1 is information that can identify the position of cyst 28 and bounding box BB2 is information that can identify the position of blood vessel 29 may be displayed.

[0121] The example shown in FIG. 5 illustrates an example of a configuration in which machine learning is performed to recognize cysts 28 shown in ultrasound images 24, and the example shown in FIG. 11 illustrates an example of a configuration in which machine learning is performed to recognize blood vessels 29 shown in ultrasound images 24. However, these are merely examples. For example, machine learning may be performed to recognize cysts 28 and blood vessels 29 shown in ultrasound images 24. In this case, machine learning is performed to recognize cysts 28 and blood vessels 29 in a distinguishable manner. To achieve this, annotations that can identify the position of cysts 28 and annotations that can identify the position of blood vessels 29 may be used as ground truth data. When ultrasound images 24 and Doppler images 88 are input to a trained model obtained by performing machine learning to recognize cysts 28 and blood vessels 29 in a distinguishable manner, the trained model recognizes cysts 28 and blood vessels 29 in a distinguishable manner.

[0122] In this way, if the trained model is able to distinguish between cyst 28 and blood vessel 29, a bounding box BB1 can be superimposed on the location of cyst 28 in ultrasound image 24 in a manner similar to the example shown in Figure 6, and a bounding box BB2 can be superimposed on the location of blood vessel 29 in ultrasound image 24 in a manner similar to the example shown in Figure 11.

[0123] In the above embodiment, a trained model for object recognition using an AI bounding box method is exemplified, but the present disclosure can also be implemented by using a trained model for object recognition using an AI segmentation method instead of or together with the trained model for object recognition using an AI bounding box method.

[0124] In the above embodiment, an example was given in which the ultrasound image 24 is displayed on the screen 26. However, for example, as shown in FIG. 13 , when the cyst recognition processing 100 is being executed, information 116 indicating that the cyst recognition processing 100 is being executed may be displayed on the screen 26, output as audio, or stored in a storage area (storage 66, a server, a personal computer, a tablet terminal, an electronic medical record, or the like). In the example shown in FIG. 13 , as an example of the information 116, text indicating that the cyst recognition processing 100 is being executed is displayed on the screen 26. This allows the user 20 to recognize that the cyst recognition processing 100 is being executed. Note that when the blood vessel recognition processing 112 is being executed, text indicating that the blood vessel recognition processing 112 is being executed may be displayed on the screen 26.

[0125] Furthermore, in the examples shown in Figures 8 and 9, when background processing is being executed, information indicating that background processing (i.e., cyst recognition processing 100 and / or blood vessel recognition processing 112 in the background) is being executed in the foreground processing may be displayed on the screen 26, output as audio, or stored in a storage area.

[0126] In the above embodiment, the generation of the ultrasound image 24 by the B-mode method and the generation of the Doppler image 88 by the Doppler method are performed in parallel, and the ultrasound image 24 and the Doppler image 88 generated at the same time are linked, but this is merely an example. For example, the generation of the ultrasound image 24 by the B-mode method and the generation of the Doppler image 88 by the Doppler method may be performed alternately, and the ultrasound image 24 and the Doppler image 88 generated one frame at a time may be linked.

[0127] In the above embodiment, an example is shown in which an ultrasound image 24 and a Doppler image 88 are generated based on reflected waves detected by the ultrasound probe 38 of the ultrasound endoscope 12, but this is merely one example, and even if an ultrasound image (i.e., an image equivalent to the ultrasound image 24) and a Doppler image (i.e., an image equivalent to the Doppler image 88) are generated based on reflected waves detected by an extracorporeal ultrasound probe, the diagnostic support process can be performed in the same manner as in the above embodiment.

[0128] In the above embodiment, an example in which the diagnostic support processing is performed by the computer 54 has been described, but the present disclosure is not limited to this, and at least a part of the processing included in the diagnostic support processing may be performed by a device provided outside the computer 54. An example of this case will be described below with reference to FIG.

[0129] 14 is a conceptual diagram showing an example of the configuration of an endoscope system 200. The endoscope system 200 is an example of an "endoscope system" according to the present disclosure. The endoscope system 200 differs from the endoscope system 10 described in the above embodiment in that it includes an external device 202.

[0130] The external device 202 is communicatively coupled to the computer 54 via a network 204 (eg, a WAN and / or a LAN, etc.).

[0131] An example of the external device 202 is at least one server that directly or indirectly transmits and receives data to and from the computer 54 via the network 204. The external device 202 receives a processing execution instruction provided via the network 204 from the processor 62 of the computer 54. The external device 202 then executes processing in accordance with the received processing execution instruction and transmits the processing result to the computer 54 via the network 204. In the computer 54, the processor 62 receives the processing result transmitted from the external device 202 via the network 204 and executes processing using the received processing result.

[0132] An example of the processing execution instruction is an instruction to cause the external device 202 to execute at least a part of the diagnosis support processing. A first example of at least a part of the diagnosis support processing (i.e., processing to be executed by the external device 202) is the above-described cyst recognition processing 100 and / or blood vessel recognition processing 112. In this case, the external device 202 executes the cyst recognition processing 100 and / or blood vessel recognition processing 112 in accordance with the processing execution instruction provided from the processor 62 via the network 204, and transmits the recognition result (e.g., the cyst recognition result 102 and / or the blood vessel recognition result 114) to the computer 54 via the network 204. In the computer 54, the processor 62 receives the recognition result and executes processing similar to that of the above-described embodiment using the received recognition result.

[0133] A second example of at least a portion of the diagnosis support processing (i.e., processing to be executed by the external device 202) is a portion of the processing by the control unit 62C. One example of a portion of the processing by the control unit 62C is processing to superimpose a bounding box BB1 or BB2 on the ultrasound image 24. In this case, the external device 202 executes processing to superimpose a bounding box BB1 or BB2 on the ultrasound image 24 in accordance with a processing execution instruction provided from the processor 62 via the network 204, and transmits the processing result (e.g., the ultrasound image 24 on which the bounding box BB1 or BB2 has been superimposed) to the computer 54 via the network 204. In the computer 54, the processor 62 receives the processing result and executes processing similar to that of the above embodiment using the received processing result.

[0134] For example, the external device 202 is realized by cloud computing. Note that cloud computing is merely one example, and the external device 202 may be realized by network computing such as fog computing, edge computing, or grid computing. Instead of a server, at least one personal computer or the like may be used as the external device 202. Alternatively, the external device 202 may be a computing device with a communication function and equipped with multiple types of AI functions.

[0135] In the above embodiment, an example in which the diagnostic assistance program 76 is stored in the storage 66 has been described, but the present disclosure is not limited to this. For example, the diagnostic assistance program 76 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or a USB memory. The diagnostic assistance program 76 stored in the non-transitory storage medium is installed in the computer 54 of the endoscope system 10. The processor 62 executes diagnostic assistance processing in accordance with the diagnostic assistance program 76.

[0136] In addition, the diagnostic assistance program 76 may be stored in a storage device such as another computer or server connected to the endoscopic system 10 via a network, and the diagnostic assistance program 76 may be downloaded and installed on the computer 54 in response to a request from the endoscopic system 10.

[0137] It is not necessary to store all of the diagnostic assistance program 76 in a storage device such as another computer or server device connected to the endoscopic system 10, or to store all of the diagnostic assistance program 76 in the storage 66; only a portion of the diagnostic assistance program 76 may be stored.

[0138] The hardware resources that execute the diagnostic support processing can include various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes the diagnostic support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes the diagnostic support processing by using the memory.

[0139] The hardware resource that executes the diagnostic support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the diagnostic support processing may be a single processor.

[0140] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the diagnostic support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute the diagnostic support processing, on a single IC chip, as typified by SoCs. In this way, the diagnostic support processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0141] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above diagnostic support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0142] The above-described description and illustrations are a detailed explanation of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or elements may be replaced with other parts from the above-described description and illustrations, as long as they do not deviate from the gist of the present disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the parts related to the present disclosure, the above-described description and illustrations omit explanations of common general technical knowledge that do not require particular explanation to enable the implementation of the present disclosure.

[0143] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0144] The following additional notes are provided regarding the above-described embodiments.

[0145] (Appendix 1) Creating example data in which a sample ultrasound image (e.g., B-mode image 94A or 108A) that is a sample of an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and that shows an observation target region of the subject is associated with sample blood flow distribution information (e.g., Doppler image 94B or 108B) that is a sample of blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasound diagnosis; and The method includes generating training data (e.g., a group of learning images 94 or 108) by associating an annotation (e.g., annotation 98 or 110) that can identify a location in the sample ultrasound image where the observation target region is captured with the example data. How to generate training data.

[0146] (Appendix 2) Creating training data (e.g., a group of learning images 94) in which example data is associated with a sample ultrasound image (e.g., B-mode image 94A or 108A) that is a sample of an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and that shows an observation target region of the subject, and example data is associated with sample blood flow distribution information (e.g., Doppler image 94B or 108B) that is a sample of blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasound diagnosis, and an annotation (e.g., annotation 98 or 110) that can identify the location in the sample ultrasound image where the observation target region is shown; and and generating a trained model (e.g., cyst recognition model 78 or blood vessel recognition model 104) by optimizing the model (e.g., model 92 or 106) by performing machine learning using the training data on the model. How to generate a trained model. [Explanation of symbols]

[0147] 10,200 Endoscopy Systems 12 Endoscopic Ultrasound 14 Display device 16 Ultrasound endoscope body 18 Processing equipment 20 users 22 Subject 24 Ultrasound images 26 screens 27 Control section 28,84 Cyst 29,86 blood vessels 30 Insertion section 32 Tip 34 Curved section 36 Soft part 38 Ultrasound Probe 40 Opening for treatment instruments 42 Treatment tools 44 Insertion port for treatment tool 46 Lighting equipment 48 Camera 50 Universal Code 50A proximal end 50B Tip 52 Reception device 54 Computer 56 Input / Output Interface 58 Transmitting and receiving circuit 60 Communication Module 62 processors 62A generation section 62B Recognition part 62C Control unit 64 memory 66 Storage 68 Bus 70 Ultrasonic Emission Signal 72 Organs 74 Reflected Wave Signal 76 Diagnostic Support Program 78 Cyst Recognition Model 92,106 models 94,108 training images 94A, 108A B-mode images 98,110 annotations 88, 94B, 108B Doppler images 90 color maps 90A Blood flow area 90B Blood flow free area 100 Cyst Recognition Processing 102 Cyst Recognition Results 104 Blood Vessel Recognition Model 112 Blood vessel recognition processing 114 Blood vessel recognition results 116 Information 202 External device 204 Network BB1,BB2 bounding boxes

Claims

1. a processor; The processor: Acquiring an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and showing an observation target region of the subject, and blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasound diagnosis; The ultrasound image and the blood flow distribution information are input to a trained model, and a recognition process is executed to cause the trained model to recognize the observation target region, and the obtained recognition result is output. Image processing device.

2. the observation target area is a lesion, The recognition result is information that can distinguish between the lesion and other than the lesion. The image processing device according to claim 1 .

3. The lesion is a cyst The image processing device according to claim 2 .

4. The trained model is obtained by performing machine learning to recognize the lesion, or machine learning to recognize the lesion and the blood vessels shown in the ultrasound image. The image processing device according to claim 2 .

5. the observation target region is a blood vessel, The recognition result is information that can distinguish between the blood vessels and non-blood vessels. The image processing device according to claim 1 .

6. The trained model is obtained by performing machine learning to recognize the blood vessels, or machine learning to recognize lesions appearing in the ultrasound image and blood vessels appearing in the ultrasound image. The image processing device according to claim 5 .

7. The blood flow distribution information is a map that can identify the strength of the blood flow. The image processing device according to claim 1 .

8. Outputting the recognition result includes displaying the recognition result on a first screen. The image processing device according to claim 1 .

9. In the foreground, the ultrasound image is displayed on a second screen; The recognition process runs in the background. The image processing device according to claim 1 .

10. If the recognition process is being performed, The processor outputs information indicating that the recognition process is in progress. The image processing device according to claim 1 .

11. The image processing device according to any one of claims 1 to 10; an ultrasonic probe that is inserted into the body of the subject and emits ultrasonic waves and detects reflected waves of the ultrasonic waves; The ultrasonic image and the blood flow distribution information are generated based on the reflected waves detected by the ultrasonic probe. Endoscopy system.

12. Acquiring an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject, the ultrasound image showing an observation target region of the subject, and blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasound diagnosis; and and outputting a recognition result obtained by inputting the ultrasound image and the blood flow distribution information into a trained model and executing a recognition process for causing the trained model to recognize the observation target region. Image processing methods.

13. Acquiring an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject, the ultrasound image showing an observation target region of the subject, and blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasound diagnosis; and A program for causing a computer to execute a process including inputting the ultrasound image and the blood flow distribution information into a trained model, thereby executing a recognition process that causes the trained model to recognize the observation target area, and outputting the recognition result obtained by the recognition process.

Images