Image processing device, endoscope system, image processing method, and program

Abstract

To provide an image processing device, an endoscope system, an image processing method, and a program capable of outputting a correct recognition result and inhibiting an erroneous recognition result from being output when there occurs erroneous recognition of an observation object region by recognition processing using a learned model for an ultrasonic image acquired by a B mode method.SOLUTION: An image processing device includes a processor. The processor acquires a first recognition result obtained by performing recognition processing for causing a learned model to recognize an observation object region by inputting an ultrasonic image obtained by a B mode method showing an observation object region of a subject to the learned model in ultrasonic diagnosis for the subject. The processor outputs a second recognition result in which the first recognition result is changed, on the basis of blood flow distribution information obtained by a Doppler method used together with the B mode method in the ultrasonic diagnosis.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an endoscope system, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses an ultrasound diagnostic device including an acquisition unit and a display control unit. In the ultrasound diagnostic device described in Patent Document 1, the acquisition unit transmits ultrasound waves to a scan region within a subject and acquires a B-mode image in which the signal intensity of the reflected waves from within the subject is expressed as brightness, and a Doppler image of a region of interest included in the scan region. In the ultrasound diagnostic device described in Patent Document 1, the display control unit displays the Doppler image on the display unit by superimposing it on the B-mode image acquired by the acquisition unit. In the ultrasound diagnostic device described in Patent Document 1, the display control unit changes the display state of the Doppler image displayed on the display unit using feature amounts obtained based on at least one of the time-series B-mode images and Doppler images acquired by the acquisition unit.

[0003] Patent Document 2 discloses an ultrasound diagnostic device including a transmission control unit, a detection unit, an evaluation unit, and a display image control unit. In the ultrasound diagnostic device described in Patent Document 2, the transmission control unit causes the ultrasound probe to transmit a first ultrasound wave toward biological tissue, and after the first ultrasound wave is transmitted, causes the ultrasound probe to transmit a second ultrasound wave. In the ultrasound diagnostic device described in Patent Document 2, the detection unit detects, within a two-dimensional region, whether an object in the biological tissue has moved due to the first ultrasound wave, based on an echo signal obtained by transmitting the second ultrasound wave. In the ultrasound diagnostic device described in Patent Document 2, the evaluation unit evaluates the tissue properties of a region of interest in the biological tissue based on the detection by the detection unit. In the ultrasound diagnostic device described in Patent Document 2, the display image control unit displays an image according to the evaluation by the evaluation unit. The process performed by the detection unit to detect whether an object has moved is color Doppler processing or B-flow processing, and the evaluation unit performs evaluation according to the size of the region from which Doppler data was obtained by color Doppler processing or the region from which B-flow data was obtained by B-flow processing.

[0004] Patent Document 3 discloses an ultrasound diagnostic device including a gate setting unit, a Doppler processing unit, a display unit, and an image enlargement unit. In the ultrasound diagnostic device described in Patent Document 3, the gate setting unit sets a Doppler gate within a vascular region by performing image analysis on a B-mode image in which at least a vascular region is captured. In the ultrasound diagnostic device described in Patent Document 3, the Doppler processing unit generates a Doppler waveform image based on the Doppler data in the Doppler gate. In the ultrasound diagnostic device described in Patent Document 3, the display unit displays a B-mode image and a Doppler waveform image. In the ultrasound diagnostic device described in Patent Document 3, when both the B-mode image and the Doppler waveform image are frozen by the user, the image enlargement unit causes the display unit to display an enlarged B-mode image that enlarges the vascular region including the Doppler gate. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-159696 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-047474 [Patent Document 3] International Publication No. 2020 / 110500 Summary of the Invention

[0006] One embodiment of the present disclosure provides an image processing device, an endoscopic system, an image processing method, and a program that can output a correct recognition result and suppress the output of an erroneous recognition result when an erroneous recognition of an observation target area occurs during recognition processing using a trained model for an ultrasound image obtained by a B-mode method. [Means for solving the problem]

[0007] A first aspect of the present disclosure is an image processing device that includes a processor, and that acquires a first recognition result obtained by inputting an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and showing an observation target area of ​​the subject into a trained model, thereby executing a recognition process that causes the trained model to recognize the observation target area, and outputs a second recognition result in which the first recognition result has been modified based on blood flow distribution information obtained by a Doppler method that is used in conjunction with the B-mode method in ultrasound diagnosis.

[0008] A second aspect of the present disclosure is an image processing device according to the first aspect, in which the observation target area is a lesion, and the second recognition result is information obtained by changing the first recognition result so that the lesion can be distinguished from non-lesions.

[0009] A third aspect of the present disclosure is the image processing device according to the second aspect, in which the lesion is a cyst.

[0010] A fourth aspect of the present disclosure is an image processing device according to the second or third aspect, in which the blood flow distribution information is a map capable of identifying the intensity of blood flow, and the second recognition result is information in which the first recognition result has been changed based on locations in the map where the intensity is equal to or greater than a reference intensity.

[0011] A fifth aspect of the present disclosure is an image processing device according to any one of the second to fourth aspects, in which the trained model is obtained by performing machine learning to recognize lesions.

[0012] A sixth aspect of the present disclosure is the image processing device according to any one of the second to fifth aspects, in which the second recognition result is information that can distinguish between a lesion and a non-lesion.

[0013] A seventh aspect of the present disclosure is an image processing device according to any one of the second to sixth aspects, wherein outputting the second recognition result includes displaying the second recognition result on a first screen as visible information that can distinguish between lesions and non-lesions.

[0014] An eighth aspect of the present disclosure is an image processing device according to the first aspect, in which the observation target area is a blood vessel, the blood vessel is identified from blood flow distribution information, and the second recognition result is information obtained by changing the first recognition result so that blood vessels can be distinguished from non-blood vessels.

[0015] A ninth aspect of the present disclosure is the image processing device according to the eighth aspect, in which the components other than blood vessels include a lesion, and the lesion is a cyst.

[0016] A tenth aspect of the present disclosure is an image processing device according to the eighth or ninth aspect, in which the blood flow distribution information is a map capable of identifying the intensity of blood flow, and the second recognition result is information in which the first recognition result has been changed based on points in the map where the intensity is less than a reference intensity.

[0017] An eleventh aspect of the present disclosure is an image processing device according to any one of the eighth to tenth aspects, in which the trained model is obtained by performing machine learning to recognize blood vessels.

[0018] A twelfth aspect of the present disclosure is the image processing device according to any one of the eighth to eleventh aspects, in which the second recognition result is information that can distinguish between blood vessels and non-blood vessels.

[0019] A thirteenth aspect of the present disclosure is an image processing device according to any one of the eighth to twelfth aspects, wherein outputting the second recognition result includes displaying the second recognition result on a first screen as visible information that can distinguish between blood vessels and non-blood vessels.

[0020] A fourteenth aspect of the present disclosure is an image processing device according to any one of the first to thirteenth aspects, in which an ultrasound image is displayed on a second screen in the foreground, recognition processing is performed in the background, and the first recognition result is changed to a second recognition result.

[0021] A fifteenth aspect of the present disclosure is an image processing device according to any one of the first to fourteenth aspects, in which, when processing to change the first recognition result to the second recognition result based on blood flow distribution information is being executed, the processor outputs information that can identify that processing to change the first recognition result to the second recognition result based on blood flow distribution information is being executed.

[0022] A sixteenth aspect of the present disclosure is an endoscopic system comprising an image processing device according to any one of the first to fifteenth aspects, and an ultrasound probe that emits ultrasound waves and detects reflected waves of the ultrasound waves while inserted into the body of a subject, and in which an ultrasound image and blood flow distribution information are generated based on the reflected waves detected by the ultrasound probe.

[0023] A seventeenth aspect of the present disclosure is an image processing method including: acquiring a first recognition result obtained by inputting an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and showing an observation target area of ​​the subject into a trained model, thereby performing a recognition process to cause the trained model to recognize the observation target area; and outputting a second recognition result in which the first recognition result has been modified based on blood flow distribution information obtained by a Doppler method used in conjunction with the B-mode method in the ultrasound diagnosis.

[0024] An 18th aspect of the present disclosure is a program for causing a computer to execute processing including: acquiring a first recognition result obtained by inputting an ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and showing an observation target area of ​​the subject into a trained model, thereby executing a recognition process for causing the trained model to recognize the observation target area; and outputting a second recognition result in which the first recognition result has been modified based on blood flow distribution information obtained by a Doppler method used in conjunction with the B-mode method in ultrasound diagnosis. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an aspect in which an endoscope system is used. [Figure 2] 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system. [Figure 3] FIG. 1 is a block diagram showing an example of the configuration of an ultrasonic endoscope. [Figure 4] FIG. 1 is a conceptual diagram showing an example of a mode in which a cyst recognition model is generated by having a model learn a group of B-mode images. [Figure 5] FIG. 10 is a conceptual diagram illustrating an example of processing content of a generation unit. [Figure 6] FIG. 10 is a conceptual diagram showing an example of processing by a processor when a cyst is mistakenly recognized (here, as an example, when a blood vessel is mistakenly recognized as a cyst). [Figure 7] 10 is a flowchart illustrating an example of the flow of a diagnostic support process. [Figure 8] 10 is a flowchart illustrating an example of the flow of background processing. [Figure 9] 10 is a flowchart showing an example of the flow of foreground processing. [Figure 10] FIG. 1 is a conceptual diagram showing an example of a mode in which a blood vessel recognition model is generated by having a model learn a group of B-mode images. [Figure 11] FIG. 10 is a conceptual diagram showing an example of processing by a processor when a blood vessel is erroneously recognized (here, as an example, when a cyst is erroneously recognized as a blood vessel). [Figure 12] 10 is a flowchart showing a modified example of the flow of the diagnostic support process. [Figure 13] FIG. 10 is a conceptual diagram showing an example of how information indicating that the erroneous recognition prevention process is being executed is displayed on a screen. [Figure 14] A conceptual diagram showing an example of a series of processes in which a processor included in a computer issues a processing execution request to an external device via a network, the external device executes processing in response to the processing execution request, and the processor included in the computer receives the processing result from the external device. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, examples of embodiments of an image processing device, an endoscope system, an image processing method, and a program according to the present disclosure will be described with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". NN is an abbreviation for "Neural Network". CNN is an abbreviation for "Convolutional neural network". R-CNN is an abbreviation for "Region based Convolutional Neural Network". YOLO is an abbreviation for "You only Look Once". RNN is an abbreviation for "Recurrent Neural Network".FCN is an abbreviation for "Fully Convolutional Network." FIFO is an abbreviation for "First In First Out."

[0029] As an example, as shown in Fig. 1, an endoscopic system 10 includes an ultrasonic endoscope 12 and a display device 14. The ultrasonic endoscope 12 is a convex type ultrasonic endoscope and includes an ultrasonic endoscope main body 16 and a processing device 18. In this embodiment, the endoscopic system 10 is an example of an "endoscopic system" according to the present disclosure. Also, in this embodiment, the processing device 18 is an example of an "image processing device" according to the present disclosure.

[0030] In this embodiment, a convex-type ultrasound endoscope is given as an example of the ultrasound endoscope 12, but this is merely an example, and the present disclosure also applies to a radial-type ultrasound endoscope. Also, in this embodiment, an ultrasound endoscope 12 is given as an example, but this is merely an example, and the present disclosure also applies to an extracorporeal ultrasound diagnostic device.

[0031] The endoscope system 10 is used by a user 20. An example of the user 20 is a doctor. The ultrasound endoscope main body 16 is connected to a processing device 18, and various signals are exchanged between the processing device 18 and the ultrasound endoscope main body 16. That is, the processing device 18 controls the operation of the ultrasound endoscope main body 16 by outputting signals to the ultrasound endoscope main body 16, and performs various signal processing on signals input from the ultrasound endoscope main body 16.

[0032] The ultrasound endoscope 12 is an endoscope used in ultrasound diagnosis of a subject 22 (e.g., a patient). When a user 20 observes an internal part (e.g., the pancreas) that is a part present inside the subject 22 during ultrasound diagnosis, the user 20 inserts the ultrasound endoscope main body 16 into the subject 22 through the mouth or nose (the mouth in the example shown in FIG. 1 ). With the ultrasound endoscope main body 16 inserted inside the subject 22, the ultrasound endoscope main body 16 emits ultrasound waves toward the internal part at a predetermined position inside the subject 22 (e.g., the stomach or duodenum), and detects reflected waves obtained when the emitted ultrasound waves are reflected by the internal part. The processing device 18 generates an ultrasound image 24 based on the reflected waves detected by the ultrasound endoscope main body 16 and outputs the image to the display device 14 or the like.

[0033] In the example shown in Figure 1, an upper gastrointestinal endoscopy is being performed, but the present disclosure is not limited to this and can also be applied to lower gastrointestinal endoscopy or bronchial endoscopy, etc.

[0034] The display device 14 displays various information including images under the control of the processing device 18. Examples of the display device 14 include a liquid crystal display and an EL display. An ultrasound image 24 generated by the processing device 18 is displayed as a moving image on a screen 26 of the display device 14. In the example shown in FIG. 1 , a cyst 28 and a blood vessel 29 are shown in the ultrasound image 24 displayed on the screen 26. In this embodiment, the screen 26 is an example of a "first screen" and a "second screen" according to the present disclosure.

[0035] 1 shows an example in which the ultrasound image 24 is displayed on the screen 26 of the display device 14, but this is merely an example and the ultrasound image 24 may be displayed on a display device other than the display device 14 (for example, the display of a tablet terminal). Also, the ultrasound image 24 may be stored in a computer-readable non-transitory storage medium (for example, a flash memory, a HDD, and / or a magnetic tape).

[0036] An image corresponding to the type of ultrasound diagnostic method, which is a diagnostic method using ultrasound, is displayed on the display device 14. The type of ultrasound examination method is selected by the user 20 or selected according to various conditions. Examples of the types of ultrasound examination methods include the B-mode (Brightness mode) method and the Doppler method (e.g., color Doppler method).

[0037] The B-mode method is an ultrasound diagnostic method that generates and displays a B-mode image. A B-mode image is obtained by converting the intensity of the reflected waves obtained when ultrasound waves are emitted into an internal body part and reflected from the internal body part into brightness on a screen 26, and refers to a two-dimensional tomographic image of a cross section parallel to the direction in which the ultrasound waves travel.

[0038] The Doppler method is an ultrasound diagnostic method that generates and displays Doppler images. A Doppler image is an image that uses reflected waves obtained by ultrasound emitted toward an internal body part and reflected from the internal body part to show the hemodynamics within blood vessels using color information. An image showing the hemodynamics within blood vessels is, for example, an image expressed using reddish and blueish colors. Reddish colors indicate blood flow toward the side from which the ultrasound is emitted and blood flow away from the side from which the ultrasound is emitted. When a reflected wave with a higher frequency than the ultrasound that hits the blood flow is obtained, it is identified as blood flow toward the side from which the ultrasound is emitted, while when a reflected wave with a lower frequency than the ultrasound that hits the blood flow is identified as blood flow away from the side from which the ultrasound is emitted. Within blood vessels, there are areas with fast and slow blood flow, and in Doppler images, the blood flow speed is expressed as a gradation using color mapping.

[0039] Depending on the ultrasound diagnostic method selected by the user 20, the screen 26 displays either a B-mode image or a Doppler image as the ultrasound image 24, or displays both a B-mode image and a Doppler image in parallel. In the example shown in FIG. 1 , a B-mode image is displayed on the screen 26 as the ultrasound image 24. The ultrasound image 24 is a moving image including a plurality of frames generated at a predetermined frame rate (e.g., 30 frames / second). The frame rate of the Doppler method is lower than the frame rate of the B-mode method. An example of the frame rate of the Doppler method (i.e., the frame rate at which the Doppler image is generated) is 15 frames / second. While a moving image is illustrated here, this is merely an example, and the present disclosure also applies even if the ultrasound image 24 is a still image. Note that, in the following, to facilitate understanding of the present disclosure, the description will be given assuming that the ultrasound image 24 is a B-mode image.

[0040] As shown in FIG. 2 as an example, the ultrasound endoscope main body 16 includes a control section 28 and an insertion section 30. The insertion section 30 is tubular. The insertion section 30 has a distal end section 32, a bending section 34, and a flexible section 36. The distal end section 32, the bending section 34, and the flexible section 36 are arranged in this order from the distal end to the proximal end of the insertion section 30. The flexible section 36 is made of a long, flexible material and connects the control section 28 and the bending section 34. The bending section 34 partially bends and rotates around the axis of the insertion section 30 when the control section 28 is operated. As a result, the insertion section 30 is advanced into the hollow organ while bending and rotating around the axis of the insertion section 30 in accordance with the shape of the hollow organ (e.g., the shape of the duodenal tract)

[0041] An ultrasonic probe 38 and a treatment tool opening 40 are provided at the tip portion 32. The ultrasonic probe 38 is provided on the distal end side of the tip portion 32. The ultrasonic probe 38 is a convex ultrasonic probe, which, when inserted into the body of the subject 22, emits ultrasonic waves to an internal region and receives reflected waves obtained when the emitted ultrasonic waves are reflected by the internal region.

[0042] The treatment tool opening 40 is formed closer to the base end of the distal end portion 32 than the ultrasonic probe 38. The treatment tool opening 40 is an opening for allowing a treatment tool 42 to protrude from the distal end portion 32. A treatment tool insertion port 44 is formed in the operation section 28, and the treatment tool 42 is inserted into the insertion section 30 from the treatment tool insertion port 44. The treatment tool 42 passes through the insertion section 30 and protrudes from the treatment tool opening 40 to the outside of the ultrasonic endoscope body 16. The treatment tool opening 40 also functions as a suction port for sucking blood, internal waste, etc.

[0043] 2, a puncture needle is shown as the treatment tool 42. However, this is merely an example, and the treatment tool 42 may also be grasping forceps and / or a cannula, etc.

[0044] The tip portion 32 is provided with an illumination device 46 and a camera 48. The illumination device 46 emits light. Examples of the light emitted from the illumination device 46 include visible light (e.g., white light), invisible light (e.g., near-infrared light), and / or special light. Examples of the special light include light for BLI and / or light for LCI.

[0045] The camera 48 optically captures images of the inside of a hollow organ. A CMOS image sensor is used for the camera 48. The CMOS image sensor is merely an example, and other types of image sensors such as a CCD image sensor may also be used. The images captured by the camera 48 may be displayed on the display device 14, on a display device other than the display device 14 (for example, the display of a tablet terminal), stored in a storage medium (for example, a flash memory, a HDD, and / or a magnetic tape), or saved in an electronic medical record.

[0046] The ultrasonic endoscope 12 is equipped with a processing device 18 and a universal cord 50. The universal cord 50 has a base end 50A and a tip end 50B. The base end 50A is connected to the operation unit 28. The tip end 50B is connected to the processing device 18. In other words, the ultrasonic endoscope main body 16 and the processing device 18 are connected via the universal cord 50.

[0047] The endoscope system 10 includes a reception device 52. The reception device 52 is connected to the processing device 18. The reception device 52 receives instructions from a user. Examples of the reception device 52 include an operation panel having a plurality of hard keys and / or a touch panel, a keyboard, a mouse, a trackball, a foot switch, a smart device, and / or a microphone.

[0048] The processing device 18 performs various signal processing operations and transmits and receives various signals to and from the ultrasonic endoscope main body 16, etc., in accordance with instructions received by the receiving device 52. For example, the processing device 18 causes the ultrasonic probe 38 to emit ultrasonic waves in accordance with instructions received by the receiving device 52, and generates and outputs an ultrasonic image 24 (see FIG. 1) based on the reflected waves received by the ultrasonic probe 38.

[0049] The display device 14 is also connected to the processing device 18. The processing device 18 controls the display device 14 in accordance with instructions received by the reception device 52. As a result, for example, an ultrasound image 24 generated by the processing device 18 is displayed on the screen 26 of the display device 14 (see FIG. 1).

[0050] 3, the processing device 18 includes a computer 54, an input / output interface 56, a transceiver circuit 58, and a communication module 60. The computer 54 is an example of a "computer" according to the present disclosure.

[0051] The computer 54 includes a processor 62, a memory 64, and a storage 66. The input / output interface 56, the processor 62, the memory 64, and the storage 66 are connected to a bus 68.

[0052] The processor 62 controls the entire processing device 18. For example, the processor 62 has a CPU and a GPU, and the GPU operates under the control of the CPU and is mainly responsible for executing image processing. The processor 62 may be one or more CPUs that integrate GPU functionality, or one or more CPUs that do not integrate GPU functionality. The processor 62 may also include a multi-core CPU or a TPU. In this embodiment, the processor 62 is an example of a "processor" according to the present disclosure.

[0053] The memory 64 is a memory in which information is temporarily stored, and is used as a work memory by the processor 62. An example of the memory 64 is a RAM. The storage 66 is a non-volatile storage device that stores various programs, various parameters, etc. An example of the storage 66 is a flash memory (for example, an EEPROM) and / or an SSD. Note that the flash memory and the SSD are merely examples, and the storage 66 may be another non-volatile storage device such as an HDD, or may be a combination of two or more types of non-volatile storage devices.

[0054] The input / output interface 56 is connected to the accepting device 52, and the processor 62 acquires instructions accepted by the accepting device 52 via the input / output interface 56 and executes processing according to the acquired instructions.

[0055] A transmission / reception circuit 58 is connected to the input / output interface 56. The transmission / reception circuit 58 generates an ultrasonic emission signal 70 having a pulse waveform in accordance with instructions from the processor 62 and outputs the signal to the ultrasonic probe 38. The ultrasonic probe 38 converts the ultrasonic emission signal 70 input from the transmission / reception circuit 58 into ultrasonic waves and emits the ultrasonic waves toward an organ 72 (e.g., the pancreas) serving as an internal body part of the subject 22. The ultrasonic probe 38 receives reflected waves obtained when the ultrasonic waves radiated toward the organ 72 are reflected by the organ 72, converts the reflected waves into reflected wave signals 74, which are electrical signals, and outputs the reflected waves to the transmission / reception circuit 58. The transmission / reception circuit 58 digitizes the reflected wave signals 74 input from the ultrasonic probe 38 and outputs the digitized reflected wave signals 74 to the processor 62 via the input / output interface 56. The processor 62 generates an ultrasonic image 24 (see FIG. 1 ) showing the state of the organ 72 based on the reflected wave signals 74 input from the transmission / reception circuit 58 via the input / output interface 56. In parallel with the generation of the ultrasound image 24, the processor 62 also generates a Doppler image based on the reflected wave signal 74.

[0056] Although not shown in Fig. 3, the lighting device 46 (see Fig. 2) is also connected to the input / output interface 56. The processor 62 controls the lighting device 46 via the input / output interface 56 to change the type of light emitted from the lighting device 46 and adjust the amount of light. Although not shown in Fig. 3, the camera 48 (see Fig. 2) is also connected to the input / output interface 56. The processor 62 controls the camera 48 via the input / output interface 56 and acquires, via the input / output interface 56, images of the inside of the body of the subject 22 captured by the camera 48.

[0057] A communication module 60 is connected to the input / output interface 56. The communication module 60 is an interface including a communication processor, an antenna, etc. The communication module 60 is connected to a network (not shown) such as a LAN or WAN, and controls communication with external devices (for example, a server, a personal computer, and / or a tablet terminal, etc.).

[0058] The display device 14 is connected to the input / output interface 56, and the processor 62 controls the display device 14 via the input / output interface 56 to cause the display device 14 to display various information.

[0059] The input / output interface 56 is connected to the accepting device 52, and the processor 62 acquires instructions accepted by the accepting device 52 via the input / output interface 56 and executes processing according to the acquired instructions.

[0060] The storage 66 stores a diagnostic assistance program 76 and a cyst recognition model 78. The processor 62 reads the diagnostic assistance program 76 from the storage 66 and executes the read diagnostic assistance program 76 on the memory 64 to perform diagnostic assistance processing. The diagnostic assistance processing is processing that executes recognition processing that recognizes an observation target region from the organ 72 using an AI method and assists the user 20 (see FIG. 1 ) in making a diagnosis based on the recognition results of the recognition processing. The diagnostic assistance processing is realized by the processor 62 operating as a generation unit 62A, a recognition unit 62B, and a control unit 62C in accordance with the diagnostic assistance program 76 executed on the memory 64.

[0061] In this embodiment, the diagnostic assistance program 76 is an example of a "program" according to the present disclosure. The cyst recognition model 78 is a trained model used in AI processing to recognize cysts 28 appearing in ultrasound images 24. In this embodiment, the cyst recognition model 78 is an example of a "trained model" according to the present disclosure.

[0062] As an example, as shown in FIG. 4, a cyst recognition model 78 is a trained model generated by performing machine learning on a model 80. The cyst recognition model 78 is obtained by performing machine learning to recognize a cyst 28 (see FIG. 1) captured in an ultrasound image 24 (see FIG. 1). A B-mode image group 82 is used as training data for the machine learning performed on the model 80. The B-mode image group 82 is made up of a plurality of B-mode images 82A that are different from one another. Examples of the B-mode image 82A include a sample B-mode image obtained by an actual ultrasound diagnosis and / or a sample B-mode image generated by so-called generation AI.

[0063] An example of the model 80 is a mathematical model using a neural network (NN). Examples of the type of NN include YOLO, R-CNN, and FCN. The NN used in the model 80 may be, for example, a combination of YOLO, R-CNN, or FCN with an RNN. An RNN is suitable for machine learning of multiple images obtained in a time series. Note that the types of NN listed here are merely examples, and other types of NNs that enable object detection by machine learning images may also be used.

[0064] A cyst 84 is captured in the multiple B-mode images 82A. Annotation 86 is attached to the B-mode images 82A. The annotation 86 is information that can identify the position where the cyst 84 is captured in the B-mode images 82A (for example, information including multiple coordinates that can identify the position of a rectangular frame circumscribing the cyst 84).

[0065] Here, for convenience of explanation, information capable of identifying the position where the cyst 84 is shown in the B-mode image 82A is exemplified as an example of the annotation 86, but this is merely an example. For example, the annotation 86 may include various information regarding the lesion shown in the B-mode image 82A, such as information capable of identifying the type of lesion shown in the B-mode image 82A, the importance of the lesion, and / or the size of the lesion. Examples of the lesion include lesions of types other than cysts (e.g., tumors).

[0066] For ease of explanation, the processing using cyst recognition model 78 will be described below as processing that is actively performed mainly by cyst recognition model 78. That is, for ease of explanation, cyst recognition model 78 will be described as a function that processes input information and outputs the processing results. Also, for ease of explanation, part of the processing for training model 80 will be described below as processing that is actively performed mainly by model 80. That is, for ease of explanation, model 80 will be described as a function that processes input information and outputs the processing results.

[0067] Each B-mode image 82A included in the B-mode image group 82 is input to the model 80. In response to this, the model 80 recognizes the position where the cyst 84 is captured from the input B-mode image 82A (in other words, predicts the position where the cyst 84 is captured). The model 80 then outputs the recognition result (in other words, the prediction result). The recognition result includes information that can identify the position recognized by the model 80 as the position where the cyst 84 is captured in the B-mode image 82A. An example of the information that can identify the position recognized by the model 80 is information that includes a plurality of coordinates that can identify the position of a bounding box surrounding the area recognized as the position where the cyst 84 is present (i.e., the position of the bounding box in the B-mode image 82A).

[0068] The model 80 is adjusted according to the error between the annotations 86 added to the B-mode images 82A input to the model 80 and the recognition results output from the model 80. That is, the model 80 is optimized by adjusting a plurality of optimization variables (for example, a plurality of connection weights and a plurality of offset values) in the model 80 so as to minimize the error, thereby generating the cyst recognition model 78. That is, the data structure of the cyst recognition model 78 is obtained by having the model 80 learn a plurality of different B-mode images 82A to which the annotations 86 are added.

[0069] As an example, as shown in Fig. 5, the generator 62A acquires a reflected wave signal 74 from the transmission / reception circuit 58 and generates an ultrasound image 24 by the B-mode method based on the acquired reflected wave signal 74. That is, the generator 62A generates a B-mode image based on the reflected wave signal 74 as the ultrasound image 24. The B-mode image generated by the generator 62A as the ultrasound image 24 is an image that shows a cross section of an organ 72 (see Fig. 3) in two dimensions. In the example shown in Fig. 5, the generator 62A generates an ultrasound image 24 that shows a cyst 28 and a blood vessel 29, but of course, a B-mode image that does not show the cyst 28 and / or the blood vessel 29 may also be generated as the ultrasound image 24.

[0070] In this embodiment, the ultrasound image 24 generated by the generation unit 62A is an example of an "ultrasound image" according to the present disclosure. Also, in this embodiment, the cyst 28 shown in the ultrasound image 24 is an example of an "observation region of the subject," a "lesion," and a "cyst" according to the present disclosure. Also, in this embodiment, the blood vessel 29 shown in the ultrasound image 24 is an example of a "blood vessel" according to the present disclosure.

[0071] In this embodiment, the B-mode method and the Doppler method are used in combination in the ultrasound diagnosis of the subject 22. That is, the generation unit 62A generates a Doppler image 88 in parallel with the ultrasound image 24. Specifically, the generation unit 62A acquires the reflected wave signal 74 used in generating the ultrasound image 24 from the transmission / reception circuit 58, and generates the Doppler image 88 by the Doppler method based on the acquired reflected wave signal 74. The frame rate of the ultrasound image 24 is higher than the frame rate of the Doppler image 88, and the ultrasound image 24 and the Doppler image 88 are linked in units of a predetermined number of frames of the ultrasound image 24 (for example, in units of 15 frames of the ultrasound image 24). An example of the frame rate of the ultrasound image 24 is 30 frames per second. An example of the frame rate of the Doppler image 88 is 15 frames per second. Note that this is merely an example, and the frame rate of the ultrasound image 24 may be the same as the frame rate of the Doppler image 88.

[0072] The Doppler image 88 is an image obtained by overlaying a color map 90, which represents characteristics (here, as an example, the distribution of blood flow) within the organ 72 using colors (i.e., chromatic colors) on the B-mode image 24. The color map 90 is information indicating blood flow dynamics determined using the Doppler effect. In the color map 90, the intensity of blood flow is represented by color. In this embodiment, the concept of blood flow intensity also includes the strength, speed, and / or momentum of blood flow. The color map 90 has a blood flow region 90A indicating an area where blood flow is present and a blood-free region 90B indicating an area where blood flow is absent. The blood flow intensity can be determined from the color assigned to the blood flow region 90A. Therefore, the image obtained by overlaying the color map 90 on the B-mode image 24, i.e., the Doppler image 88, can also be considered a map from which the intensity of blood flow can be determined. In this embodiment, the Doppler image 88 is an example of the "blood flow distribution information" and "map" according to the present disclosure.

[0073] 6, as an example, the recognition unit 62B acquires the ultrasound image 24 generated by the generation unit 62A from the generation unit 62A, and executes a cyst recognition process 91 on the acquired ultrasound image 24. The cyst recognition process 91 is a process in which the ultrasound image 24 acquired from the generation unit 62A is input to the cyst recognition model 78, thereby causing the cyst recognition model 78 to recognize a cyst 28. In this embodiment, the cyst recognition process 91 is an example of a "recognition process" according to the present disclosure.

[0074] By executing the cyst recognition process 91, a cyst recognition result 92 is generated by the cyst recognition model 78. The cyst recognition result 92 includes information that can identify the position of the cyst 28 shown in the ultrasound image 24. Here, the cyst recognition result 92 may include a result in which a blood vessel 29 has been mistakenly recognized as a cyst 28.

[0075] The recognition unit 62B acquires the cyst recognition result 92 generated by the cyst recognition model 78. The recognition unit 62B reflects the cyst recognition result 92 on the ultrasound image 24 used in the cyst recognition process 91. In this case, for example, the cyst recognition result 92 is reflected on the ultrasound image 24 as a bounding box BB1. The bounding box BB1 is classified into a first bounding box BB1a and a second bounding box BB1b. The first bounding box BB1a is superimposed by the recognition unit 62B on a portion of the ultrasound image 24 where the cyst 28 is shown. In the example shown in FIG. 6, the first bounding box BB1a is superimposed on the ultrasound image 24 as a circumscribing box for the cyst 28 shown in the ultrasound image 24. The second bounding box BB1b is superimposed by the recognition unit 62B on a portion of the ultrasound image 24 where the blood vessel 29 is shown. The second bounding box BB1b is superimposed on the ultrasound image 24 as a circumscribing frame for the blood vessel 29 shown in the ultrasound image 24.

[0076] 6, blood vessel 29 is erroneously recognized as cyst 28, and the result is superimposed as second bounding box BB1b on ultrasound image 24. If ultrasound image 24 in which first bounding box BB1a and second bounding box BB1b are superimposed is displayed on screen 26, there is a risk that user 20 will also erroneously recognize blood vessel 29 as cyst 28.

[0077] Therefore, in this embodiment, the control unit 62C erases the second bounding box BB1b on the ultrasound image 24. To achieve this, first, the control unit 62C acquires from the recognition unit 62B the ultrasound image 24 on which the first bounding box BB1a and the second bounding box BB1b are superimposed. The control unit 62C also acquires from the generation unit 62A a Doppler image 88 (i.e., the Doppler image 88 linked to the ultrasound image 24) generated in parallel with the ultrasound image 24 used in the cyst recognition process 91, and compares the acquired Doppler image 88 with the ultrasound image 24 on which the first bounding box BB1a and the second bounding box BB1b are superimposed. The control unit 62C then identifies a location in the Doppler image 88 where the blood flow intensity is equal to or greater than a reference intensity TH (hereinafter referred to as a "high-intensity location"). The reference intensity TH may be a fixed value or a variable value that is changed according to instructions and / or various conditions received by the reception device 52, etc. An example of the reference intensity TH when it is a fixed value is a predetermined intensity that is generally known as the lower limit of the intensity of the blood flow in the blood vessels 29 present in the organ 72 (here, as an example, the pancreas).

[0078] The control unit 62C identifies a location corresponding to the high-intensity location in the ultrasound image 24 on which the first bounding box BB1a and the second bounding box BB1b are superimposed (hereinafter referred to as the "high-intensity corresponding location"). If the second bounding box BB1b is superimposed at a position surrounding the identified high-intensity corresponding location, the control unit 62C erases the second bounding box BB1b. In other words, if the blood flow intensity at a location in the Doppler image 88 corresponding to the location on which the second bounding box BB1b is superimposed is equal to or greater than the reference intensity TH, the control unit 62C erases the second bounding box BB1b. As a result, the ultrasound image 24 is left in a state in which only the first bounding box BB1a of the first bounding box BB1a and the second bounding box BB1b is superimposed. This means that the cyst recognition result 92 reflected on the ultrasound image 24 has been changed based on the Doppler image 88. In other words, the first bounding box BB1a remaining on the ultrasound image 24 after the second bounding box BB1b has been erased from the ultrasound image 24 can be said to be information in which the cyst recognition result 92 has been changed based on the high-intensity areas.

[0079] The fact that the second bounding box BB1b has been erased from the ultrasound image 24 and the first bounding box BB1a remains means that it is now possible to distinguish between cysts 28 and other objects in the ultrasound image 24. Furthermore, the fact that the second bounding box BB1b has been erased from the ultrasound image 24 and the first bounding box BB1a remains means that the cyst 28 is captured in the area surrounded by the first bounding box BB1a in the ultrasound image 24, and that there is a possibility that blood vessels 29 are captured in the area other than the area surrounded by the first bounding box BB1a in the ultrasound image 24. This means that the cyst recognition result 92 has been changed based on the Doppler image 88 so that it is possible to distinguish between cysts 28 and blood vessels 29. In other words, the first bounding box BB1a remaining on the ultrasound image 24 due to the second bounding box BB1b being erased from the ultrasound image 24 can be said to be information obtained by changing the cyst recognition result 92 so that the cyst 28 and the blood vessel 29 can be distinguished based on the Doppler image 88.

[0080] The control unit 62C outputs the ultrasound image 24 obtained by erasing the second bounding box BB1b from the ultrasound image 24 on which the first bounding box BB1a and the second bounding box BB1b are superimposed. In the example shown in Fig. 6, the control unit 62C displays the ultrasound image 24 obtained by erasing the second bounding box BB1b from the ultrasound image 24 on which the first bounding box BB1a and the second bounding box BB1b are superimposed on the screen 26. That is, the first bounding box BB1a superimposed on the ultrasound image 24 is displayed on the screen 26 as visible information that enables a distinction between the cyst 28 and an area other than the cyst 28.

[0081] Here, an example of displaying the ultrasound image 24 superimposed with the first bounding box BB1a on the screen 26 has been given as an example of outputting the ultrasound image 24 superimposed with the first bounding box BB1a, but the present disclosure is not limited to this. For example, the ultrasound image 24 superimposed with the first bounding box BB1a may be stored in a storage area (storage 66, a server, a personal computer, a tablet terminal, an electronic medical record, etc.) or may be recorded on a medium (e.g., paper) by a printer (not shown). Furthermore, the position of the first bounding box BB1a within the ultrasound image 24 (i.e., the position of the cyst 28) may be announced by voice via a speaker (not shown).

[0082] In this embodiment, the first bounding box BB1a and the second bounding box BB1b superimposed on the ultrasound image 24 are examples of the "first recognition result" according to the present disclosure. Also, in this embodiment, the first bounding box BB1a superimposed on the ultrasound image 24 displayed on the screen 26 is an example of the "second recognition result" and "visible information" according to the present disclosure.

[0083] Next, the operation of the endoscope system 10 will be described with reference to FIG.

[0084] 7 shows an example of the flow of diagnostic support processing performed by the processor 62 of the processing device 18. The flow of diagnostic support processing shown in FIG. 7 is an example of the "image processing method" according to the present disclosure.

[0085] 7, first, in step ST10, the generation unit 62A controls the ultrasound probe 38 via the transmission / reception circuitry 58 to cause the ultrasound probe 38 to emit ultrasound waves toward the organ 72 (here, the pancreas, as an example). After the processing of step ST10 is executed, the diagnostic support processing proceeds to step ST12.

[0086] In step ST12, the generation unit 62A determines whether or not a reflected wave based on the ultrasound emitted as a result of the processing of step ST10 has been detected by the ultrasound probe 38. If, in step ST12, the ultrasound probe 38 has not detected a reflected wave based on the ultrasound emitted as a result of the processing of step ST10, the determination is negative, and the diagnostic support processing proceeds to step ST28. If, in step ST12, the ultrasound probe 38 has detected a reflected wave based on the ultrasound emitted as a result of the processing of step ST10, the determination is positive, and the diagnostic support processing proceeds to step ST14.

[0087] In step ST14, the generation unit 62A generates the ultrasound image 24 and the Doppler image 88 based on the reflected waves detected by the ultrasound probe 38. After the processing of step ST14 is executed, the diagnostic support processing proceeds to step ST16.

[0088] In step ST16, the recognition unit 62B inputs the ultrasound image 24 generated in step ST14 into the cyst recognition model 78. After the processing of step ST16 is executed, the diagnosis support processing proceeds to step ST18.

[0089] In step ST18, the recognition unit 62B determines whether or not the ultrasound image 24 has been input into the cyst recognition model 78, resulting in the cyst recognition model 78 recognizing that a cyst 28 is captured in the ultrasound image 24. If the ultrasound image 24 is not recognized as capturing a cyst 28 in step ST18, the determination is negative, and the diagnostic support processing proceeds to step ST26. If the ultrasound image 24 is recognized as capturing a cyst 28 in step ST18, the determination is positive, and the diagnostic support processing proceeds to step ST20.

[0090] In step ST20, the recognition unit 62B superimposes the cyst recognition result 92 as a bounding box BB1 on the ultrasound image 24 generated in step ST14. After the processing of step ST20 is executed, the diagnosis support processing proceeds to step ST22.

[0091] In step ST22, the control unit 62C compares the Doppler image 88 generated in step ST14 with the ultrasound image 24 on which the bounding box BB1 is superimposed, thereby determining whether or not a second bounding box BB1b is superimposed at a position surrounding the high-intensity corresponding portion in the ultrasound image 24. If, in step ST22, the second bounding box BB1b is not superimposed at a position surrounding the high-intensity corresponding portion in the ultrasound image 24, the determination is negative, and the diagnostic support processing proceeds to step ST26. If, in step ST22, the second bounding box BB1b is superimposed at a position surrounding the high-intensity corresponding portion in the ultrasound image 24, the determination is positive, and the diagnostic support processing proceeds to step ST24.

[0092] In step ST24, the control unit 62C erases the second bounding box BB1b from the ultrasound image 24 on which the bounding box BB1 is superimposed. After the processing of step ST24 is executed, the diagnostic support processing proceeds to step ST26.

[0093] In step ST26, the control unit 62C displays the ultrasound image 24 on the screen 26. That is, when the determination in step ST18 is negative, the processing of step ST26 is executed, and the ultrasound image 24 generated in step ST14 is displayed on the screen 26. Also, when the determination in step ST22 is negative, the processing of step ST26 is executed, and the ultrasound image 24 on which the first bounding box BB1a is superimposed is displayed on the screen 26. Furthermore, when the diagnostic support processing proceeds to step ST26 after the processing of step ST24 is executed, the processing of step ST26 is executed, and the ultrasound image 24 on which the first bounding box BB1a is superimposed is displayed on the screen 26. After the processing of step ST24 is executed, the diagnostic support processing proceeds to step ST28.

[0094] In step ST28, the control unit 62C determines whether or not the condition for terminating the diagnostic support processing has been satisfied. One example of the condition for terminating the diagnostic support processing is that an instruction to terminate the diagnostic support processing has been accepted by the acceptance device 52. If the condition for terminating the diagnostic support processing has not been satisfied in step ST28, the determination is denied, and the diagnostic support processing proceeds to step ST10. If the condition for terminating the diagnostic support processing has been satisfied in step ST28, the diagnostic support processing is terminated.

[0095] As described above, in the endoscope system 10, when the cyst recognition process 91 is executed to recognize a cyst 28 and a blood vessel 29 is erroneously recognized as the cyst 28, the cyst recognition result 92 is superimposed on the ultrasound image 24 as a first bounding box BB1a and a second bounding box BB1b. The first bounding box BB1a is superimposed so as to surround the portion of the ultrasound image 24 in which the cyst 28 is shown, and the second bounding box BB1b is superimposed so as to surround the portion of the ultrasound image 24 in which the blood vessel 29 is shown.

[0096] However, if the ultrasound image 24 in this state is displayed on the screen 26 as is, there is a risk that the user 20 will mistakenly recognize the blood vessel 29 as a cyst 28.

[0097] Therefore, in the endoscope system 10, the Doppler image 88 linked to the ultrasound image 24 is compared with the ultrasound image 24 on which the first bounding box BB1a and the second bounding box BB1b are superimposed, thereby identifying the high-intensity corresponding portion. The high-intensity corresponding portion refers to a portion corresponding to the high-intensity portion in the ultrasound image 24 on which the first bounding box BB1a and the second bounding box BB1b are superimposed. The high-intensity portion refers to a portion in the Doppler image 88 where the blood flow intensity is equal to or greater than a reference intensity TH. In the endoscope system 10, if the second bounding box BB1b is superimposed at a position surrounding the high-intensity corresponding portion, the second bounding box BB1b is erased. Then, the ultrasound image 24 on which the first bounding box BB1a is superimposed is displayed on the screen 26.

[0098] As a result, when a blood vessel 29 is erroneously recognized as a cyst 28 by the cyst recognition processing 91, a correct recognition result can be displayed and the display of the erroneous recognition result can be suppressed. That is, it is possible to allow the user 20 to visually recognize the first bounding box BB1a superimposed on the ultrasound image 24, without perceiving the second bounding box BB1b. This means that the user 20 can visually recognize the area in which the cyst 28 appears and the area other than the cyst 28 in the ultrasound image 24 through the first bounding box BB1a. This also means that when the ultrasound image 24 shows a mixture of cysts 28 and blood vessels 29, the user 20 can visually recognize that the cyst 28 and the blood vessel 29 are visually distinguishable from each other.

[0099] In the above embodiment, an example in which the diagnostic support processing is executed sequentially has been described. However, the diagnostic support processing may be implemented by parallel processing. For example, as shown in Figures 8 and 9, background processing and foreground processing may be executed in parallel to implement processing similar to the diagnostic support processing described in the above embodiment. The background processing refers to processing that occurs in the background, and the foreground processing refers to processing that occurs in the foreground. For example, in the background processing, a cyst recognition processing 91 is executed, and a cyst recognition result 92 is changed based on a Doppler image 88. In the foreground processing, an ultrasound image 24 is displayed on the screen 26.

[0100] An example of the flow of background processing will now be described with reference to FIG.

[0101] The background processing shown in FIG. 8 differs from the diagnostic support processing shown in FIG. 7 in that it includes processing in steps ST100 and ST102 instead of processing in steps ST26 and ST28.

[0102] In step ST100 shown in FIG. 8, the control unit 62C stores the ultrasound image 24 in the memory 64. The ultrasound image 24 is stored in the memory 64 using the FIFO method. The memory 64 also serves as a buffer memory capable of storing a plurality of ultrasound images 24 in chronological order. When the determination in step ST18 is negative, the process of step ST100 is executed, whereby the ultrasound image 24 generated in step ST14 is stored in the memory 64. When the determination in step ST22 is negative, the process of step ST100 is executed, whereby the ultrasound image 24 on which the first bounding box BB1a is superimposed is stored in the memory 64. Furthermore, when the diagnostic support processing proceeds to step ST100 after the processing of step ST24 is executed, the process of step ST100 is executed, whereby the ultrasound image 24 on which the first bounding box BB1a is superimposed is stored in the memory 64. After the processing of step ST100 is executed, the diagnostic support processing proceeds to step ST102.

[0103] In step ST102, the control unit 62C determines whether or not the conditions for ending the background processing are satisfied. One example of the conditions for ending the background processing is that an instruction to end the background processing is accepted by the acceptance device 52. If the conditions for ending the background processing are not satisfied in step ST102, the determination is negative, and the background processing proceeds to step ST10. If the conditions for ending the background processing are satisfied in step ST102, the background processing is terminated.

[0104] Next, an example of the flow of foreground processing will be described with reference to FIG.

[0105] 9, the control unit 62C determines whether or not a new ultrasound image 24 is stored in the memory 64. If a new ultrasound image 24 is not stored in the memory 64 in step ST200, the determination is negative, and the foreground processing proceeds to step ST202.

[0106] In step ST202, the control unit 62C acquires an ultrasound image 24 stored in the memory 64 (for example, the ultrasound image 24 that was stored most recently among the multiple ultrasound images 24 stored in the memory 64 in a FIFO manner), and displays the acquired ultrasound image 24 on the screen 26. After the processing of step ST202 is executed, the foreground processing proceeds to step ST204.

[0107] In step ST204, control unit 62C determines whether or not the conditions for terminating the foreground processing are satisfied. One example of the conditions for terminating the foreground processing is that an instruction to terminate the foreground processing is accepted by acceptance device 52. If the conditions for terminating the foreground processing are not satisfied in step ST204, the determination is negative, and the foreground processing proceeds to step ST200. If the conditions for terminating the foreground processing are satisfied in step ST204, the foreground is terminated.

[0108] 8 and 9, cyst recognition processing 91 is executed in the background, cyst recognition result 92 is changed based on Doppler image 88, and ultrasound image 24 is displayed in the foreground on screen 26. In this manner, user 20 can view ultrasound image 24 without being aware that cyst recognition processing 91 is being executed or that cyst recognition result 92 is being changed based on Doppler image 88.

[0109] In the above embodiment, an example in which cyst 28 is recognized and blood vessel 29 is prevented from being erroneously recognized as cyst 28 has been described, but this is merely one example. For example, as shown in FIGS. 10 to 12, blood vessel 29 may be recognized and cyst 28 may be prevented from being erroneously recognized as blood vessel 29. To realize this example, a blood vessel recognition model 94 shown in FIG. 10 is used instead of cyst recognition model 78. In the example shown in FIG. 10, blood vessel recognition model 94 is an example of a "trained model" according to the present disclosure.

[0110] The blood vessel recognition model 94 is obtained by performing machine learning to recognize blood vessels 29 (see FIG. 1) shown in an ultrasound image 24 (see FIG. 1). As an example, as shown in FIG. 10, the blood vessel recognition model 94 is generated by performing machine learning on a model 96. The model 96 has the same data structure as the model 80.

[0111] A B-mode image group 98 is used as training data for machine learning performed on the model 96. The B-mode image group 98 is made up of a plurality of B-mode images 98A that are different from one another.

[0112] A plurality of B-mode images 98A show blood vessels 100. Annotations 102 are attached to the B-mode images 98A. The annotations 102 are information that can identify the position where the blood vessels 100 appear in the B-mode images 98A (for example, information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing the blood vessels 100).

[0113] Here, for convenience of explanation, information capable of identifying the position where the blood vessel 100 appears in the B-mode image 98A is exemplified as an example of the annotation 102, but this is merely an example. For example, the annotation 102 may include various information related to the blood vessel 100 appearing in the B-mode image 98A, such as information capable of identifying the type of blood vessel 100 appearing in the B-mode image 98A, the importance of the blood vessel 100, and / or the size of the blood vessel 100.

[0114] For ease of explanation, the processing using the blood vessel recognition model 94 will be described below as processing that is actively performed mainly by the blood vessel recognition model 94. That is, for ease of explanation, the blood vessel recognition model 94 will be described as a function that processes input information and outputs the processing results. Also, for ease of explanation, part of the processing for training the model 96 will be described below as processing that is actively performed mainly by the model 96. That is, for ease of explanation, the model 96 will be described as a function that processes input information and outputs the processing results.

[0115] Each B-mode image 98A included in the B-mode image group 98 is input to the model 96. In response to this, the model 96 recognizes the position where the blood vessel 100 is captured from the input B-mode image 98A (in other words, predicts the position where the blood vessel 100 is captured). The model 96 then outputs the recognition result (in other words, the prediction result). The recognition result includes information that can identify the position recognized by the model 96 as the position where the blood vessel 100 is captured in the B-mode image 98A. An example of the information that can identify the position recognized by the model 96 is information that includes a plurality of coordinates that can identify the position of a bounding box surrounding an area recognized as the position where the blood vessel 100 exists (i.e., the position of the bounding box in the B-mode image 98A).

[0116] The model 96 is adjusted according to the error between the annotations 102 added to the B-mode images 98A input to the model 96 and the recognition results output from the model 96. That is, the model 96 is optimized by adjusting a plurality of optimization variables (e.g., a plurality of connection weights and a plurality of offset values) in the model 96 so as to minimize the error, thereby generating the blood vessel recognition model 94. That is, the data structure of the blood vessel recognition model 94 is obtained by having the model 96 learn a plurality of different B-mode images 98A to which the annotations 102 are added.

[0117] 11, the recognition unit 62B acquires the ultrasound image 24 generated by the generation unit 62A from the generation unit 62A, and executes blood vessel recognition processing 103 on the acquired ultrasound image 24. The blood vessel recognition processing 103 is processing in which the ultrasound image 24 acquired from the generation unit 62A is input to the blood vessel recognition model 94, thereby causing the blood vessel recognition model 94 to recognize blood vessels 29. In the example shown in FIG. 11, the blood vessel recognition processing 103 is an example of a "recognition processing" according to the present disclosure. Furthermore, in the example shown in FIG. 11, the blood vessels 29 are an example of an "observation target region" and a "blood vessel" according to the present disclosure.

[0118] By executing the blood vessel recognition process 103, a blood vessel recognition result 104 is generated by the blood vessel recognition model 94. The blood vessel recognition result 104 includes information that can identify the position of the blood vessel 29 shown in the ultrasound image 24. Here, the blood vessel recognition result 104 may include a result in which a cyst 28 is mistakenly recognized as a blood vessel 29.

[0119] The recognition unit 62B acquires a blood vessel recognition result 104 generated by the blood vessel recognition model 94. The recognition unit 62B reflects the blood vessel recognition result 104 on the ultrasound image 24 used in the blood vessel recognition process 103. In this case, for example, the blood vessel recognition result 104 is reflected on the ultrasound image 24 as a bounding box BB2. The bounding box BB2 is classified into a third bounding box BB2a and a fourth bounding box BB2b. The third bounding box BB2a is superimposed by the recognition unit 62B on a portion of the ultrasound image 24 where the cyst 28 is shown. In the example shown in FIG. 11 , the third bounding box BB2a is superimposed on the ultrasound image 24 as a circumscribing box for the cyst 28 shown in the ultrasound image 24. The fourth bounding box BB2b is superimposed by the recognition unit 62B on a portion of the ultrasound image 24 where the blood vessel 29 is shown. In the example shown in FIG. 11, the fourth bounding box BB2b is superimposed on the ultrasound image 24 as a circumscribing box for the blood vessel 29 shown in the ultrasound image 24.

[0120] 11, the cyst 28 is erroneously recognized as a blood vessel 29, and the result is superimposed as a third bounding box BB2a on the ultrasound image 24. If the ultrasound image 24 in which the third bounding box BB2a and the fourth bounding box BB2b are superimposed is displayed on the screen 26, there is a risk that the user 20 will erroneously recognize the cyst 28 as a blood vessel 29.

[0121] Therefore, in this embodiment, the control unit 62C erases the third bounding box BB2a from the ultrasound image 24. To achieve this, first, the control unit 62C acquires from the recognition unit 62B the ultrasound image 24 on which the third bounding box BB2a and the fourth bounding box BB2b are superimposed. The control unit 62C also acquires from the generation unit 62A a Doppler image 88 (i.e., the Doppler image 88 linked to the ultrasound image 24) generated in parallel with the ultrasound image 24 used in the blood vessel recognition process 103, and compares the acquired Doppler image 88 with the ultrasound image 24 on which the third bounding box BB2a and the fourth bounding box BB2b are superimposed. The control unit 62C then identifies areas in the Doppler image 88 where the blood flow intensity is less than the reference intensity TH (hereinafter referred to as "low-intensity areas").

[0122] The control unit 62C identifies a location corresponding to the low-intensity location in the ultrasound image 24 on which the third bounding box BB2a and the fourth bounding box BB2b are superimposed (hereinafter referred to as the "low-intensity corresponding location"). If the third bounding box BB2a is superimposed at a position surrounding the identified low-intensity corresponding location, the control unit 62C erases the third bounding box BB2a. In other words, if the blood flow intensity at a location in the Doppler image 88 corresponding to the location on which the third bounding box BB2a is superimposed is less than the reference intensity TH, the control unit 62C erases the third bounding box BB2a. As a result, the ultrasound image 24 is left in a state in which only the third bounding box BB2b of the third bounding box BB2a and the fourth bounding box BB2b is superimposed. This means that the blood vessel recognition result 104 reflected on the ultrasound image 24 has been changed based on the Doppler image 88. In other words, the fourth bounding box BB2b remaining on the ultrasound image 24 after the third bounding box BB2a has been erased from the ultrasound image 24 can be said to be information in which the blood vessel recognition result 104 has been changed based on the low-intensity area.

[0123] The fact that the third bounding box BB2a has been erased from the ultrasound image 24 and the fourth bounding box BB2b remains means that it is now possible to distinguish between blood vessels 29 and things other than blood vessels 29 in the ultrasound image 24. Furthermore, the fact that the third bounding box BB2a has been erased from the ultrasound image 24 and the fourth bounding box BB2b remains means that blood vessels 29 are captured in the area surrounded by the fourth bounding box BB2b in the ultrasound image 24, and that there is a possibility that cysts 28 are captured in an area other than the area surrounded by the fourth bounding box BB2b in the ultrasound image 24. This means that the blood vessel recognition result 104 has been changed based on the Doppler image 88 so that cysts 28 and blood vessels 29 can be distinguished. In other words, the fourth bounding box BB2b remaining on the ultrasound image 24 after the third bounding box BB2a has been erased from the ultrasound image 24 can be said to be information obtained by changing the blood vessel recognition result 104 so that the cyst 28 and the blood vessel 29 can be distinguished based on the Doppler image 88.

[0124] The control unit 62C outputs an ultrasound image 24 obtained by erasing the third bounding box BB2a from the ultrasound image 24 on which the third bounding box BB2a and the fourth bounding box BB2b are superimposed. In the example shown in FIG. 11 , the control unit 62C displays on the screen 26 the ultrasound image 24 obtained by erasing the third bounding box BB2a from the ultrasound image 24 on which the third bounding box BB2a and the fourth bounding box BB2b are superimposed. That is, the fourth bounding box BB2b superimposed on the ultrasound image 24 is displayed on the screen 26 as visible information that enables distinction between blood vessels 29 and objects other than the blood vessels 29. An example of "objects other than the blood vessels 29" here is an area that includes a cyst 28 among the areas captured in the ultrasound image 24.

[0125] Here, an example of displaying the ultrasound image 24 superimposed with the fourth bounding box BB2a on the screen 26 has been given as an example of outputting the ultrasound image 24 superimposed with the fourth bounding box BB2b, but the present disclosure is not limited to this. For example, the ultrasound image 24 superimposed with the fourth bounding box BB2b may be stored in a storage area (storage 66, a server, a personal computer, a tablet terminal, an electronic medical record, etc.), or may be recorded on a medium (e.g., paper) by a printer (not shown). Furthermore, the position of the fourth bounding box BB2b within the ultrasound image 24 (i.e., the position of the blood vessel 29) may be announced by voice via a speaker (not shown).

[0126] In this embodiment, the third bounding box BB2a and the fourth bounding box BB2b superimposed on the ultrasound image 24 are examples of the "first recognition result" according to the present disclosure. Also, in this embodiment, the fourth bounding box BB2b superimposed on the ultrasound image 24 displayed on the screen 26 is an example of the "second recognition result" and "visible information" according to the present disclosure.

[0127] 12 shows a modified example of the flow of the diagnostic support processing performed by the processor 62 of the processing device 18, that is, an example of the flow of processing for recognizing blood vessels 29 and suppressing cysts 28 from being erroneously recognized as blood vessels 29. The flow of the diagnostic support processing shown in FIG. 12 is an example of the "image processing method" according to the present disclosure.

[0128] The flowchart shown in FIG. 12 differs from the flowchart shown in FIG. 7 in that it includes processes of steps ST300 to ST310 instead of the processes of steps ST16 to ST26.

[0129] 12, the recognition unit 62B inputs the ultrasound image 24 generated in step ST14 to the blood vessel recognition model 94. After the processing of step ST300 is executed, the diagnosis support processing proceeds to step ST302.

[0130] In step ST302, the recognition unit 62B determines whether or not the ultrasound image 24 has been input to the blood vessel recognition model 94, resulting in the blood vessel recognition model 94 recognizing that a blood vessel 29 is shown in the ultrasound image 24. In step ST302, if the blood vessel 29 is not recognized as being shown in the ultrasound image 24, the determination is negative, and the diagnostic support processing proceeds to step ST306. In step ST302, if the blood vessel 29 is recognized as being shown in the ultrasound image 24, the determination is positive, and the diagnostic support processing proceeds to step ST304.

[0131] In step ST304, the recognition unit 62B superimposes the blood vessel recognition result 104 as a bounding box BB2 on the ultrasound image 24 generated in step ST14. After the processing of step ST304 is executed, the diagnosis support processing proceeds to step ST306.

[0132] In step ST306, the control unit 62C compares the Doppler image 88 generated in step ST14 with the ultrasound image 24 on which the bounding box BB2 is superimposed, thereby determining whether or not a third bounding box BB2a is superimposed at a position surrounding the low intensity corresponding portion in the ultrasound image 24. In step ST306, if the third bounding box BB2a is not superimposed at a position surrounding the low intensity corresponding portion in the ultrasound image 24, the determination is negative, and the diagnostic support processing proceeds to step ST310. In step ST306, if the third bounding box BB2a is superimposed at a position surrounding the low intensity corresponding portion in the ultrasound image 24, the determination is positive, and the diagnostic support processing proceeds to step ST308.

[0133] In step ST308, the control unit 62C erases the third bounding box BB2a from the ultrasound image 24 on which the bounding box BB2 is superimposed. After the process of step ST304 is executed, the diagnostic support process proceeds to step ST310.

[0134] In step ST310, the control unit 62C displays the ultrasound image 24 on the screen 26. That is, when the determination in step ST302 is negative, the processing of step ST310 is executed, and the ultrasound image 24 generated in step ST14 is displayed on the screen 26. Also, when the determination in step ST306 is negative, the processing of step ST310 is executed, and the ultrasound image 24 on which the fourth bounding box BB2b is superimposed is displayed on the screen 26. Furthermore, when the diagnostic support processing proceeds to step ST310 after the processing of step ST308 is executed, the processing of step ST310 is executed, and the ultrasound image 24 on which the fourth bounding box BB2b is superimposed is displayed on the screen 26. After the processing of step ST310 is executed, the diagnostic support processing proceeds to step ST28.

[0135] 10 to 12, when blood vessel 29 is recognized by executing blood vessel recognition processing 103 and cyst 28 is erroneously recognized as blood vessel 29, blood vessel recognition result 104 is superimposed on ultrasound image 24 as third bounding box BB2a and fourth bounding box BB2b. Third bounding box BB2a is superimposed so as to surround the portion of ultrasound image 24 in which cyst 28 is captured, and fourth bounding box BB2b is superimposed so as to surround the portion of ultrasound image 24 in which blood vessel 29 is captured.

[0136] However, if the ultrasound image 24 in this state is displayed on the screen 26 as is, there is a risk that the user 20 will mistakenly recognize the cyst 28 as a blood vessel 29.

[0137] Therefore, in the endoscope system 10, the Doppler image 88 linked to the ultrasound image 24 is compared with the ultrasound image 24 on which the third bounding box BB2a and the fourth bounding box BB2b are superimposed, thereby identifying the low-intensity corresponding portion. The low-intensity corresponding portion refers to a portion corresponding to the low-intensity portion in the ultrasound image 24 on which the third bounding box BB2a and the fourth bounding box BB2b are superimposed. The low-intensity portion refers to a portion in the Doppler image 88 where the intensity of the blood flow is less than the reference intensity TH. In the endoscope system 10, if the third bounding box BB2a is superimposed at a position surrounding the low-intensity corresponding portion, the third bounding box BB2a is erased. Then, the ultrasound image 24 on which the fourth bounding box BB2b is superimposed is displayed on the screen 26.

[0138] As a result, when the blood vessel recognition process 103 erroneously recognizes the cyst 28 as a blood vessel 29, a correct recognition result can be displayed and the display of the erroneous recognition result can be suppressed. That is, the user 20 can visually recognize the fourth bounding box BB2b superimposed on the ultrasound image 24, without perceiving the third bounding box BB2a. In other words, the user 20 can visually recognize the location of the blood vessel 29 in the ultrasound image 24 through the fourth bounding box BB2b. This also means that when the ultrasound image 24 shows a mixture of the cyst 28 and the blood vessel 29, the user 20 can visually recognize that the cyst 28 and the blood vessel 29 are visually distinguishable from each other.

[0139] Although an example in which the diagnostic support processing shown in FIG. 12 is executed instead of the diagnostic support processing shown in FIG. 7 has been described, this is merely an example. For example, the diagnostic support processing shown in FIG. 7 and the diagnostic support processing shown in FIG. 12 may be executed in parallel. In this case, the first bounding box BB1a and the fourth bounding box BB2b may be displayed on the screen 26 in a visually distinguishable manner (e.g., color, brightness, thickness, and / or line type). Furthermore, the first bounding box BB1a and the fourth bounding box BB2b may be displayed in a manner that allows the meaning of each of the first bounding box BB1a and the fourth bounding box BB2b to be identified. For example, the first bounding box BB1a and the fourth bounding box BB2b may be displayed in a manner that allows the user to distinguish that the first bounding box BB1a is a bounding box that allows the location of a cyst 28 to be identified, and that the fourth bounding box BB2b is a bounding box that allows the location of a blood vessel 29 to be identified. In addition, information (e.g., an identifier) ​​that visually identifies that the first bounding box BB1a is information that can identify the position of the cyst 28 and the fourth bounding box BB2b is information that can identify the position of the blood vessel 29 may be displayed.

[0140] In the above embodiment, an example in which the second bounding box BB1b is erased has been described, but this is merely an example. For example, the second bounding box BB1b may be displayed on the screen 26 at a display level that is visually imperceptible. Alternatively, the first bounding box BB1a and the second bounding box BB1b may be displayed on the screen 26 in a visually distinguishable manner so that the first bounding box BB1a is information that can identify the position of the cyst 28 and the second bounding box BB1b is information that can identify the position of the blood vessel 29. For example, the first bounding box BB1a and the second bounding box BB1b may be different in color and / or brightness, or an identifier that can identify the first bounding box BB1a as the bounding box BB1 indicating the position of the cyst 28 may be assigned to the first bounding box BB1a, and an identifier that can identify the second bounding box BB1b as the bounding box BB1 indicating the position of the blood vessel 29 may be assigned to the second bounding box BB1b. The same can be said about the relationship between the third bounding box BB2a and the fourth bounding box BB2b shown in FIG.

[0141] In the above embodiment, the Doppler image 88 is used as the image to be compared with the ultrasound image 24, but the present disclosure is not limited to this, and the color map 90 may be used instead of the Doppler image 88. This is because the control unit 62C can determine the presence or absence of blood flow and the strength of the blood flow from the color map 90, and therefore, similarly to the above embodiment, the control unit 62C can identify areas where "intensity ≧ reference intensity TH" holds, i.e., high-intensity areas, and areas where "intensity < reference intensity TH" holds, i.e., low-intensity areas.

[0142] In the above embodiment, a trained model for object recognition using an AI bounding box method is exemplified, but the present disclosure can also be implemented by using a trained model for object recognition using an AI segmentation method instead of or together with the trained model for object recognition using an AI bounding box method.

[0143] In the above embodiment, an example is shown in which an ultrasound image 24 and a Doppler image 88 are generated based on reflected waves detected by the ultrasound probe 38 of the ultrasound endoscope 12. However, this is merely one example, and even if an ultrasound image and a Doppler image are generated based on reflected waves detected by an extracorporeal ultrasound probe, the diagnostic support process can be performed in the same manner as in the above embodiment.

[0144] In the above embodiment, an example was given in which the ultrasound image 24 was displayed on the screen 26. However, for example, as shown in FIG. 13 , when the cyst recognition process 91 is being executed, information 106 indicating that the process of suppressing erroneous recognition using the Doppler image 88 is being executed may be displayed on the screen 26, output as audio, or stored in a storage area (such as the storage 66, a server, a personal computer, a tablet terminal, or an electronic medical record). In the example shown in FIG. 13 , as an example of the information 106, text indicating that the process of suppressing erroneous recognition using the Doppler image 88 (the cyst recognition process 91 in the example shown in FIG. 13 ) is being executed is displayed on the screen 26. This allows the user 20 to recognize that the process of suppressing erroneous recognition using the Doppler image 88 is being executed. Note that when the blood vessel recognition process 103 is being executed, text indicating that the blood vessel recognition process 103 is being executed may be displayed on the screen 26.

[0145] 8 and 9, when background processing is being executed, information indicating that background processing is being executed may be displayed on screen 26, output as audio, or stored in a storage area in the foreground processing. Note that "background processing is being executed" means that correct recognition results are being displayed in the background and the display of erroneous recognition results is being suppressed (in other words, cyst recognition processing 91 and / or blood vessel recognition processing 103 is being executed in the background).

[0146] In the above embodiment, the generation of the ultrasound image 24 by the B-mode method and the generation of the Doppler image 88 by the Doppler method are performed in parallel, and the ultrasound image 24 and the Doppler image 88 generated at the same time are linked, but this is merely an example. For example, the generation of the ultrasound image 24 by the B-mode method and the generation of the Doppler image 88 by the Doppler method may be performed alternately, and the ultrasound image 24 and the Doppler image 88 generated one frame at a time may be linked.

[0147] In the above embodiment, an example in which the diagnostic support processing is performed by the computer 54 has been described, but the present disclosure is not limited to this, and at least a part of the processing included in the diagnostic support processing may be performed by a device provided outside the computer 54. An example of this case will be described below with reference to FIG.

[0148] 14 is a conceptual diagram showing an example of the configuration of an endoscope system 200. The endoscope system 200 is an example of an "endoscope system" according to the present disclosure. The endoscope system 200 differs from the endoscope system 10 described in the above embodiment in that it includes an external device 202.

[0149] The external device 202 is communicatively coupled to the computer 54 via a network 204 (eg, a WAN and / or a LAN, etc.).

[0150] An example of the external device 202 is at least one server that directly or indirectly transmits and receives data to and from the computer 54 via the network 204. The external device 202 receives a processing execution instruction provided via the network 204 from the processor 62 of the computer 54. The external device 202 then executes processing in accordance with the received processing execution instruction and transmits the processing result to the computer 54 via the network 204. In the computer 54, the processor 62 receives the processing result transmitted from the external device 202 via the network 204 and executes processing using the received processing result.

[0151] An example of the processing execution instruction is an instruction to cause the external device 202 to execute at least a part of the diagnosis support processing. A first example of at least a part of the diagnosis support processing (i.e., processing to be executed by the external device 202) is the above-described cyst recognition processing 91 and / or blood vessel recognition processing 103. In this case, the external device 202 executes the cyst recognition processing 91 and / or blood vessel recognition processing 103 in accordance with the processing execution instruction provided from the processor 62 via the network 204, and transmits the recognition result (e.g., the cyst recognition result 92 and / or the blood vessel recognition result 104) to the computer 54 via the network 204. In the computer 54, the processor 62 receives the recognition result and executes processing similar to that in the above-described embodiment using the received recognition result.

[0152] A second example of at least a portion of the diagnostic support processing (i.e., processing to be executed by the external device 202) is a portion of the processing by the control unit 62C. One example of a portion of the processing by the control unit 62C is processing to compare the ultrasound image 24 with the Doppler image 88 and erase unnecessary bounding boxes based on the comparison results. In this case, the external device 202 executes processing to compare the ultrasound image 24 with the Doppler image 88 and erase unnecessary bounding boxes based on the comparison results in accordance with a processing execution instruction provided from the processor 62 via the network 204, and transmits the processing results (e.g., the ultrasound image 24 from which the unnecessary bounding boxes have been erased) to the computer 54 via the network 204. In the computer 54, the processor 62 receives the processing results and executes processing similar to that of the above embodiment using the received processing results.

[0153] For example, the external device 202 is realized by cloud computing. Note that cloud computing is merely one example, and the external device 202 may be realized by network computing such as fog computing, edge computing, or grid computing. Instead of a server, at least one personal computer or the like may be used as the external device 202. Alternatively, the external device 202 may be a computing device with a communication function and equipped with multiple types of AI functions.

[0154] In the above embodiment, an example in which the diagnostic assistance program 76 is stored in the storage 66 has been described, but the present disclosure is not limited to this. For example, the diagnostic assistance program 76 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or a USB memory. The diagnostic assistance program 76 stored in the non-transitory storage medium is installed in the computer 54 of the endoscope system 10. The processor 62 executes diagnostic assistance processing in accordance with the diagnostic assistance program 76.

[0155] In addition, the diagnostic assistance program 76 may be stored in a storage device such as another computer or server connected to the endoscopic system 10 via a network, and the diagnostic assistance program 76 may be downloaded and installed on the computer 54 in response to a request from the endoscopic system 10.

[0156] It is not necessary to store all of the diagnostic assistance program 76 in a storage device such as another computer or server device connected to the endoscopic system 10, or to store all of the diagnostic assistance program 76 in the storage 66; only a portion of the diagnostic assistance program 76 may be stored.

[0157] The hardware resources that execute the diagnostic support processing can include various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes the diagnostic support processing by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to execute specific processing. Each processor has built-in or connected memory, and executes the diagnostic support processing by using the memory.

[0158] The hardware resource that executes the diagnostic support processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the diagnostic support processing may be a single processor.

[0159] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the diagnostic support processing. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute the diagnostic support processing, on a single IC chip, as typified by SoCs. In this way, the diagnostic support processing is realized using one or more of the above-mentioned various processors as hardware resources.

[0160] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above diagnostic support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0161] The above-described description and illustrations are a detailed explanation of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or elements may be replaced with other parts from the above-described description and illustrations, as long as they do not deviate from the gist of the present disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the parts related to the present disclosure, the above-described description and illustrations omit explanations of common general technical knowledge that do not require particular explanation to enable the implementation of the present disclosure.

[0162] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0163] 10,200 Endoscopy Systems 12 Endoscopic Ultrasound 14 Display device 16 Ultrasound endoscope body 18 Processing equipment 20 users 22 Subject 24 Ultrasound images 26 screens 28,84 Cyst 29,100 blood vessels 30 Insertion section 32 Tip 34 Curved section 36 Soft part 38 Ultrasound Probe 40 Opening for treatment instruments 42 Treatment tools 44 Insertion port for treatment tool 46 Lighting equipment 48 Camera 50 Universal Code 50A proximal end 50B Tip 52 Reception device 54 Computer 56 Input / Output Interface 58 Transmitting and receiving circuit 60 Communication Module 62 processors 62A generation section 62B Recognition part 62C Control unit 64 memory 66 Storage 68 Bus 70 Ultrasonic Emission Signal 72 Organs 74 Reflected Wave Signal 76 Diagnostic Support Program 78 Cyst Recognition Model 80,96 model 82,98 B-mode images 82A, 98A B-mode images 86,102 annotations 88 Doppler images 90 color maps 90A Blood flow area 90B Blood flow free area 91 Cyst Recognition Processing 92 Cyst Recognition Results 94 Blood Vessel Recognition Model 103 Blood vessel recognition processing 104 Blood vessel recognition results 106 Information 202 External device 204 Network BB1,BB2 bounding boxes BB1a First bounding box BB1b Second bounding box BB2a Third bounding box BB2b 4th bounding box TH reference strength

Claims

1. a processor; The processor: An ultrasound image obtained by a B-mode method in an ultrasound diagnosis of a subject and showing an observation target region of the subject is input to a trained model, and a recognition process is performed to cause the trained model to recognize the observation target region, thereby obtaining a first recognition result; A second recognition result is output in which the first recognition result is changed based on blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasonic diagnosis. Image processing device.

2. the observation target area is a lesion, The second recognition result is information obtained by modifying the first recognition result so that the lesion can be distinguished from other than the lesion. The image processing device according to claim 1 .

3. The lesion is a cyst The image processing device according to claim 2 .

4. the blood flow distribution information is a map that can identify the strength of the blood flow, The second recognition result is information obtained by changing the first recognition result based on a location in the map where the intensity is equal to or greater than a reference intensity. The image processing device according to claim 2 .

5. The trained model is obtained by performing machine learning to recognize the lesion. The image processing device according to claim 2 .

6. The second recognition result is information that can distinguish between the lesion and other than the lesion. The image processing device according to claim 2 .

7. Outputting the second recognition result includes displaying the second recognition result on a first screen as visible information that enables the lesion and other than the lesion to be distinguished. The image processing device according to claim 2 .

8. the observation target region is a blood vessel, the blood vessel is identified from the blood flow distribution information; The second recognition result is information obtained by changing the first recognition result so that the blood vessels can be distinguished from other than the blood vessels. The image processing device according to claim 1 .

9. Lesions are included other than the blood vessels, The lesion is a cyst The image processing device according to claim 8 .

10. the blood flow distribution information is a map that can identify the strength of the blood flow, The second recognition result is information obtained by changing the first recognition result based on a portion in the map where the intensity is less than a reference intensity. The image processing device according to claim 8 .

11. The trained model is obtained by performing machine learning to recognize the blood vessels. The image processing device according to claim 8 .

12. The second recognition result is information that can distinguish between the blood vessel and a non-blood vessel. The image processing device according to claim 8 .

13. Outputting the second recognition result includes displaying the second recognition result on a first screen as visible information that allows the blood vessels to be distinguished from other than the blood vessels. The image processing device according to claim 8 .

14. In the foreground, the ultrasound image is displayed on a second screen; The recognition process is executed in the background, and the first recognition result is changed to the second recognition result. The image processing device according to claim 1 .

15. When a process of changing the first recognition result to the second recognition result based on the blood flow distribution information is being executed, The processor outputs information that can identify that a process of changing the first recognition result to the second recognition result based on the blood flow distribution information is being executed. The image processing device according to claim 1 .

16. An image processing device according to any one of claims 1 to 15; an ultrasonic probe that is inserted into the body of the subject and emits ultrasonic waves and detects reflected waves of the ultrasonic waves; The ultrasonic image and the blood flow distribution information are generated based on the reflected waves detected by the ultrasonic probe. Endoscopy system.

17. Acquiring a first recognition result obtained by inputting an ultrasound image obtained by a B-mode method in ultrasound diagnosis of a subject and showing an observation target area of ​​the subject into a trained model, and executing a recognition process to cause the trained model to recognize the observation target area; and and outputting a second recognition result obtained by modifying the first recognition result based on blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasound diagnosis. Image processing methods.

18. Acquiring a first recognition result obtained by inputting an ultrasound image obtained by a B-mode method in ultrasound diagnosis of a subject and showing an observation target area of ​​the subject into a trained model, and executing a recognition process to cause the trained model to recognize the observation target area; and and outputting a second recognition result obtained by modifying the first recognition result based on blood flow distribution information obtained by a Doppler method used in combination with the B-mode method in the ultrasound diagnosis.

Images