Learning device, trained model, medical diagnosis device, endoscopic ultrasonography device, learning method, and program

Abstract

This learning device is provided with a first processor. The first processor acquires a plurality of medical images to each of which an annotation identifying a lesion is added, and allows a model to learn the acquired plurality of medical images. The medical images are generated on the basis of at least one convex-type ultrasonic image, and are images in a mode that mimics at least a part of a radial-type ultrasonic image.

Description

Learning device, trained model, medical diagnostic device, ultrasonic endoscope device, learning method, and program

[0001] The technology disclosed herein relates to a learning device, a trained model, a medical diagnostic device, an ultrasound endoscope device, a learning method, and a program.

[0002] Japanese Patent Laid-Open Publication No. 09-084793 discloses an ultrasonic image processing device. In the ultrasonic image processing device described in Japanese Patent Laid-Open Publication No. 09-084793, an ultrasonic probe control unit drives an ultrasonic probe to perform a three-dimensional scan of a subject by combining radial scanning and linear scanning. Also, in the ultrasonic image processing device described in Japanese Patent Laid-Open Publication No. 09-084793, an ultrasonic observation device sequentially creates multiple consecutive ultrasonic tomographic images from echo signals of the subject obtained by the three-dimensional scan of the ultrasonic probe via the ultrasonic probe control unit, and sequentially outputs the images to a tomographic image monitor. The tomographic image monitor then sequentially displays the ultrasonic tomographic images.

[0003] International Publication No. 2022 / 071326 discloses an information processing device. The information processing device described in International Publication No. 2022 / 071326 includes an image acquisition unit and a first position information output unit. The image acquisition unit acquires catheter images obtained by a radial scanning image acquisition catheter. The first position information output unit inputs the acquired catheter images into a medical device trained model that outputs first position information regarding the position of a medical instrument included in the catheter images when the catheter images are input, and outputs the first position information.

[0004] Japanese Patent Laid-Open Publication No. 2000-316864 discloses an ultrasonic diagnostic device that includes an ultrasonic observation device that transmits and receives ultrasonic waves and obtains real-time echo images (ultrasonic tomographic images), and an image processing device that performs various image processing based on the echo data obtained by the ultrasonic observation device.

[0005] One embodiment of the technology of the present disclosure provides a learning device, a trained model, a medical diagnostic device, an ultrasound endoscope device, a learning method, and a program that can contribute to identifying lesions shown in radial ultrasound images using a trained model, without using a trained model obtained by training a model only on radial ultrasound images.

[0006] A first aspect of the technology of the present disclosure is a learning device that includes a first processor, which acquires a plurality of medical images annotated to identify lesions, and trains a model using the acquired medical images, and the medical images are generated based on at least one convex ultrasound image and are images that mimic at least a portion of a radial ultrasound image.

[0007] A second aspect of the technology of the present disclosure is a learning device according to the first aspect, in which the plurality of medical images includes a circular image generated by combining a plurality of convex ultrasound images.

[0008] A third aspect of the technique of the present disclosure is the learning device according to the second aspect, in which the scale of the circular image is adjusted based on the scale of the radial ultrasound image.

[0009] A fourth aspect of the technology of the present disclosure is a learning device according to the second or third aspect, in which a circular image is pre-stored in a first memory, a first processor acquires the circular image from the first memory, and trains a model on the acquired circular image.

[0010] A fifth aspect of the technology of the present disclosure is a learning device according to any one of the first to fourth aspects, in which the plurality of medical images include rotated images obtained by rotating convex ultrasound images.

[0011] A sixth aspect of the technique of the present disclosure is the learning device according to the fifth aspect, in which the scale of the rotated image is adjusted based on the scale of the radial ultrasound image.

[0012] A seventh aspect of the technology of the present disclosure is a learning device according to the fifth or sixth aspect, in which the rotated image is pre-stored in the second memory, the first processor acquires the rotated image from the second memory, and trains the acquired rotated image into a model.

[0013] An eighth aspect of the technology of the present disclosure is a learning device according to any one of the first to seventh aspects, in which the plurality of medical images include scale-adjusted images obtained by adjusting the scale of a convex ultrasound image based on the scale of a radial ultrasound image.

[0014] A ninth aspect of the technology of the present disclosure is a learning device according to the eighth aspect, in which the scaled image is pre-stored in a third memory, and the first processor acquires the scaled image from the third memory and trains a model using the acquired scaled image.

[0015] A tenth aspect of the technology of the present disclosure is a learning device according to the first aspect, in which a first processor randomly selects one generation method from a plurality of generation methods for generating a medical image based on at least one convex ultrasound image, acquires the medical image by generating it according to the selected generation method, and trains a model using the acquired medical image.

[0016] An eleventh aspect of the technology of the present disclosure is a learning device according to the tenth aspect, wherein the plurality of generation methods include a first generation method, a second generation method, and a third generation method, wherein the first generation method includes generating a circular image as a medical image by synthesizing a plurality of convex ultrasound images, the second generation method includes generating a rotated image as a medical image by rotating the convex ultrasound image, and the third generation method includes generating a scale-adjusted image as a medical image by adjusting the scale of the convex ultrasound image based on the scale of the radial ultrasound image.

[0017] A twelfth aspect of the technique of the present disclosure is a learning device according to the eleventh aspect, in which the first generation method includes adjusting the scale of the circular image based on the scale of the radial ultrasound image.

[0018] A thirteenth aspect of the technology of the present disclosure is a learning device according to the eleventh or twelfth aspect, in which the second generation method includes adjusting the scale of the rotated image based on the scale of the radial ultrasound image.

[0019] A fourteenth aspect of the technology of the present disclosure is a learning device according to any one of the first to thirteenth aspects, in which a first processor acquires at least one first ultrasound image obtained by a first radial type ultrasound endoscope and trains a model on the acquired first ultrasound image.

[0020] A fifteenth aspect of the technology of the present disclosure is a learning device according to any one of the first to fourteenth aspects, in which a first processor acquires a virtual ultrasound image that is generated based on volume data representing a subject and that mimics at least a portion of a radial ultrasound image, and trains a model on the acquired virtual ultrasound image.

[0021] A sixteenth aspect of the technology of the present disclosure is a trained model obtained by a learning device according to any one of the first to fifteenth aspects training a model using multiple medical images.

[0022] A seventeenth aspect of the technology of the present disclosure is a trained model having a data structure used in a process of identifying a lesion from a second ultrasound image obtained by a second radial ultrasound endoscope, wherein the data structure is obtained by training the model on a plurality of medical images annotated to identify the lesion, and the medical image is generated based on at least one convex ultrasound image and is an image of a form that imitates at least a portion of the radial ultrasound image.

[0023] An 18th aspect of the technology of the present disclosure is a medical diagnostic device comprising a trained model according to the 16th or 17th aspect and a second processor, wherein the second processor acquires a third ultrasound image obtained by a third radial type ultrasound endoscope and detects an area corresponding to a lesion from the acquired third ultrasound image in accordance with the trained model.

[0024] A 19th aspect of the technology of the present disclosure is an ultrasound endoscopic device comprising a trained model according to the 16th or 17th aspect, a fourth radial ultrasound endoscope, and a third processor, wherein the third processor acquires a fourth ultrasound image obtained by the fourth radial ultrasound endoscope and detects an area corresponding to a lesion from the acquired fourth ultrasound image in accordance with the trained model.

[0025] A twentieth aspect of the technology of the present disclosure is a learning method that includes acquiring a plurality of medical images annotated with annotations that identify lesions, and training a model from the acquired plurality of medical images, wherein the medical images are generated based on at least one convex ultrasound image and are images that mimic at least a portion of a radial ultrasound image.

[0026] A 21st aspect of the technology of the present disclosure is a program for causing a computer to execute processing, the processing including acquiring a plurality of medical images annotated to identify a lesion, and training a model from the acquired plurality of medical images, wherein the medical images are generated based on at least one convex ultrasound image and are images that mimic at least a portion of a radial ultrasound image.

[0027] 1 is a conceptual diagram showing an example of an aspect in which an endoscopic system is used. FIG. 2 is a conceptual diagram showing an example of the overall configuration of an endoscopic system. FIG. 3 is a block diagram showing an example of the configuration of an ultrasonic endoscopic device. FIG. 4 is a block diagram showing an example of the configuration of a learning device. FIG. 5 is a conceptual diagram showing an example of the processing content of an acquisition unit and a learning execution unit of a learning device. FIG. 6 is a conceptual diagram showing an example of how to create a circular image, a scaled image, a rotated image, a radial ultrasound image, and a virtual image used for learning a neural network. FIG. 7 is a conceptual diagram showing an example of the processing content of a generation unit, a detection unit, and a control unit of a processing device. FIG. 8 is a flowchart showing an example of the flow of a learning execution process. FIG. 9 is a flowchart showing an example of the flow of a lesion detection process. FIG. 10 is a conceptual diagram showing an example of the processing content of an acquisition unit according to a first modified example. FIG. 11 is a flowchart showing an example of the flow of a learning execution process according to a first modified example.

[0028] Hereinafter, examples of embodiments of a learning device, a trained model, a medical diagnostic device, an ultrasound endoscope device, a learning method, and a program according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0029] First, the terms used in the following description will be explained.

[0030] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". CT is an abbreviation for "Computed Tomography". MRI is an abbreviation for "Magnetic Resonance Imaging". PC is an abbreviation for "Personal Computer". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network". AI is an abbreviation for "Artificial Intelligence". BLI is an abbreviation for "Blue Light Imaging". LCI is an abbreviation for "Linked Color Imaging". NN is an abbreviation for "Neural Network". CNN is an abbreviation for "Convolutional neural network".R-CNN is an abbreviation for "Region-based Convolutional Neural Network." YOLO is an abbreviation for "You only Look Once." RNN is an abbreviation for "Recurrent Neural Network." FCN is an abbreviation for "Fully Convolutional Network."

[0031] In this embodiment, "match" refers to a match that includes not only a perfect match but also a match that includes an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not violate the spirit of the technology of the present disclosure. In this embodiment, "identical" refers to a match that includes not only a perfect match but also an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not violate the spirit of the technology of the present disclosure. In this embodiment, a radial ultrasound image refers to an ultrasound image obtained by an endoscopic ultrasound examination using a radial scanning method. In this embodiment, a convex ultrasound image refers to an ultrasound image obtained by an endoscopic ultrasound examination using a convex scanning method. In addition, the following description is based on the assumption that the scale of a radial ultrasound image is smaller than that of a convex ultrasound image.

[0032] 1 , an endoscopic system 10 includes an ultrasound endoscopic device 12 and a display device 14. The ultrasound endoscopic device 12 includes a radial ultrasound endoscope 16 (hereinafter referred to as the "ultrasound endoscope 16") and a processing device 18. The ultrasound endoscopic device 12 is an example of a "medical diagnostic device" and an "ultrasound endoscopic device" according to the technology of the present disclosure. The ultrasound endoscope 16 is an example of a "first radial ultrasound endoscope," a "second radial ultrasound endoscope," a "third radial ultrasound endoscope," and a "fourth radial ultrasound endoscope" according to the technology of the present disclosure.

[0033] The ultrasonic endoscope 16 is a radial scanning ultrasonic endoscope. The ultrasonic endoscopic device 12 is used by a doctor 20 or the like. The processing device 18 is connected to the ultrasonic endoscope 16 and exchanges various signals with the ultrasonic endoscope 16. That is, the processing device 18 controls the operation of the ultrasonic endoscope 16 by outputting signals to the ultrasonic endoscope 16, and performs various signal processing on signals input from the ultrasonic endoscope 16.

[0034] The ultrasound endoscopic device 12 is a device for performing medical treatment (e.g., diagnosis and / or treatment) on a target area (e.g., an organ such as the pancreas) inside the body of a subject 22, and generates and outputs an ultrasound image showing the observation target area including the target area.

[0035] For example, when observing an observation target region inside the body of the subject 22, the doctor 20 inserts the ultrasound endoscope 16 into the body of the subject 22 through the mouth or nose (the mouth in the example shown in FIG. 1 ) and emits ultrasound waves at a position such as the stomach or duodenum. Because the ultrasound endoscope 16 is a radial scanning ultrasound endoscope, it emits ultrasound waves concentrically and detects the reflected waves obtained when the emitted ultrasound waves are reflected off the observation target region.

[0036] 1 shows an example in which an upper gastrointestinal endoscopy is being performed, but the technology of the present disclosure is not limited to this and can also be applied to lower gastrointestinal endoscopy, bronchial endoscopy, etc. In other words, the technology of the present disclosure can be applied to any radial scanning ultrasound examination.

[0037] The processing device 18 generates a radial ultrasound image 24 based on the reflected waves detected by the ultrasound endoscope 16 under a specific image mode and outputs the image to the display device 14, etc. The specific image mode is B-mode (Brightness mode). However, B-mode is merely an example, and other modes such as A-mode (Amplitude mode) and M-mode (Motion mode) may also be used.

[0038] The radial type ultrasound image 24 is an ultrasound image having a circular outer shape. The radial type ultrasound image 24 is a moving image including a plurality of frames generated at a specific frame rate (e.g., several tens of frames per second). Although a moving image is illustrated here, this is merely an example, and the technology of the present disclosure is also applicable even if the radial type ultrasound image 24 is a still image. The radial type ultrasound image 24 is an example of a "second ultrasound image," a "third ultrasound image," and a "fourth ultrasound image" according to the technology of the present disclosure.

[0039] For ease of explanation, the radial ultrasound image 24 obtained by the ultrasound endoscopic device 12 in a radial ultrasound endoscopic examination of the subject 22 will be referred to as a "radial ultrasound image" without a reference symbol unless it is necessary to distinguish between the radial ultrasound image 24 and other radial ultrasound images. Here, the other radial ultrasound images refer to, for example, radial ultrasound images obtained in a specific image mode (here, B-mode, as an example) in one or more radial ultrasound endoscopic examinations of one or more subjects other than the subject 22 (for example, one or more radial ultrasound endoscopic examinations performed prior to the radial ultrasound endoscopic examination shown in FIG. 1 ). The other radial ultrasound images depict an observation area corresponding to the observation area depicted in the radial ultrasound image 24.

[0040] The display device 14 displays various information including images under the control of the processing device 18. Examples of the display device 14 include a liquid crystal display and an EL display. A radial ultrasound image 24 generated by the processing device 18 is displayed as a moving image on a screen 26 of the display device 14. By observing the radial ultrasound image 24 displayed on the screen 26, the doctor 20 can determine whether or not a lesion is present in the observation area, and, if a lesion is found, can identify the position of the lesion within the observation area through the radial ultrasound image 24.

[0041] 1 shows an example in which the radial ultrasound image 24 is displayed on the screen 26 of the display device 14, but this is merely an example, and the radial ultrasound image 24 may be displayed on a display device other than the display device 14 (for example, the display of a tablet terminal). Also, the radial ultrasound image 24 may be stored in a computer-readable non-transitory storage medium (for example, a flash memory, a HDD, and / or a magnetic tape).

[0042] As shown in FIG. 2 , the ultrasonic endoscope 16 includes a control section 28 and an insertion section 30. The insertion section 30 is tubular. The insertion section 30 includes a distal end section 32, a bending section 34, and a flexible section 36. The distal end section 32, the bending section 34, and the flexible section 36 are arranged in this order from the distal end to the proximal end of the insertion section 30. The flexible section 36 is formed of a long, flexible material and connects the control section 28 and the bending section 34. The bending section 34 partially bends and rotates around the axis of the insertion section 30 when the control section 28 is operated. As a result, the insertion section 30 is advanced into the hollow organ while bending and rotating around the axis of the insertion section 30 in accordance with the shape of the hollow organ (e.g., the shape of the duodenal tract)

[0043] The distal end portion 32 is provided with an ultrasonic probe 38 and a treatment tool opening 40. The ultrasonic probe 38 is provided on the distal end side of the distal end portion 32. The ultrasonic probe 38 is formed in a cylindrical shape, and emits ultrasonic waves concentrically around the axis of the ultrasonic probe 38, and receives reflected waves obtained when the emitted ultrasonic waves are reflected from the observation target region.

[0044] The treatment tool opening 40 is formed closer to the base end of the distal end portion 32 than the ultrasonic probe 38. The treatment tool opening 40 is an opening for allowing a treatment tool 42 to protrude from the distal end portion 32. A treatment tool insertion port 44 is formed in the operation unit 28, and the treatment tool 42 is inserted into the insertion portion 30 from the treatment tool insertion port 44. The treatment tool 42 passes through the insertion portion 30 and protrudes from the treatment tool opening 40 to the outside of the ultrasonic endoscope 16. The treatment tool opening 40 also functions as a suction port for sucking blood, internal waste, and the like.

[0045] 2, a puncture needle is shown as the treatment tool 42. However, this is merely an example, and the treatment tool 42 may also be grasping forceps and / or a sheath, etc.

[0046] 2, an illumination device 46 and a camera 48 are provided at the distal end portion 32. The illumination device 46 emits light. Examples of the type of light emitted from the illumination device 46 include visible light (e.g., white light), invisible light (e.g., near-infrared light), and / or special light. Examples of the special light include light for BLI and / or light for LCI.

[0047] The camera 48 optically captures images of the inside of a hollow organ. An example of the camera 48 is a CMOS camera. The CMOS camera is merely an example, and other types of cameras such as a CCD camera may also be used. The image captured by the camera 48 may be displayed on the display device 14, on a display device other than the display device 14 (e.g., the display of a tablet terminal), or stored in a storage medium (e.g., a flash memory, a HDD, and / or a magnetic tape).

[0048] The ultrasonic endoscope device 12 includes a processing device 18 and a universal cord 50. The universal cord 50 has a base end 50A and a tip end 50B. The base end 50A is connected to the operation unit 28. The tip end 50B is connected to the processing device 18.

[0049] The endoscope system 10 includes a reception device 52. The reception device 52 is connected to the processing device 18. The reception device 52 receives instructions from a user. Examples of the reception device 52 include an operation panel having a plurality of hard keys and / or a touch panel, a keyboard, a mouse, a trackball, a foot switch, a smart device, and / or a microphone.

[0050] The processing device 18 performs various signal processing operations and transmits and receives various signals to and from the ultrasonic endoscope 16, etc., in accordance with instructions received by the receiving device 52. For example, the processing device 18 causes the ultrasonic probe 38 to emit ultrasonic waves in accordance with instructions received by the receiving device 52, and generates and outputs a radial ultrasonic image 24 (see FIG. 1 ) based on the reflected waves received by the ultrasonic probe 38.

[0051] The display device 14 is also connected to the processing device 18. The processing device 18 controls the display device 14 in accordance with instructions received by the reception device 52. As a result, for example, a radial ultrasound image 24 generated by the processing device 18 is displayed on the screen 26 of the display device 14 (see FIG. 1).

[0052] As an example, and as shown in FIG. 3, the processing device 18 includes a computer 54 , an input / output interface 56 , a transmitting / receiving circuit 58 , and a communication module 60 .

[0053] The computer 54 includes a processor 62, a RAM 64, and an NVM 66. The input / output interface 56, the processor 62, the RAM 64, and the NVM 66 are connected to a bus 68.

[0054] The processor 62 controls the entire processing device 18. For example, the processor 62 has a CPU and a GPU, and the GPU operates under the control of the CPU and is mainly responsible for executing image processing. The processor 62 may be one or more CPUs that integrate GPU functionality, or one or more CPUs that do not integrate GPU functionality. The processor 62 may also include a multi-core CPU or a TPU. The processor 62 is an example of a "second processor" and a "third processor" according to the technology of the present disclosure.

[0055] The RAM 64 is a memory that temporarily stores information and is used as a work memory by the processor 62. The NVM 66 is a nonvolatile storage device that stores various programs, various parameters, and the like. Examples of the NVM 66 include flash memory (e.g., EEPROM) and / or an SSD. Note that flash memory and SSD are merely examples, and the NVM 66 may be other nonvolatile storage devices such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0056] The input / output interface 56 is connected to the reception device 52, and the processor 62 acquires instructions received by the reception device 52 via the input / output interface 56 and executes processing according to the acquired instructions.

[0057] A transmission / reception circuit 58 is connected to the input / output interface 56. The transmission / reception circuit 58 generates a pulsed ultrasonic emission signal 70 in accordance with instructions from the processor 62 and outputs the signal to the ultrasonic probe 38. The ultrasonic probe 38 converts the ultrasonic emission signal 70 input from the transmission / reception circuit 58 into ultrasonic waves and emits the ultrasonic waves toward an observation region 72 of the subject 22. The ultrasonic waves are emitted concentrically from the ultrasonic probe 38. The ultrasonic probe 38 receives reflected waves obtained when the ultrasonic waves emitted from the ultrasonic probe 38 are reflected by the observation region 72, converts the reflected waves into a reflected wave signal 74, which is an electrical signal, and outputs the reflected wave to the transmission / reception circuit 58. The transmission / reception circuit 58 digitizes the reflected wave signal 74 input from the ultrasonic probe 38 and outputs the digitized reflected wave signal 74 to the processor 62 via the input / output interface 56. The processor 62 generates a radial ultrasound image 24 (see Figure 1) as an ultrasound image showing the cross-sectional appearance of the observation area 72 based on the reflected wave signal 74 input from the transmission / reception circuit 58 via the input / output interface 56.

[0058] Although not shown in Fig. 3 , the lighting device 46 (see Fig. 2 ) is also connected to the input / output interface 56. The processor 62 controls the lighting device 46 via the input / output interface 56 to change the type of light emitted from the lighting device 46 and adjust the amount of light. Although not shown in Fig. 3 , the camera 48 (see Fig. 2 ) is also connected to the input / output interface 56. The processor 62 controls the camera 48 via the input / output interface 56 and acquires, via the input / output interface 56, images of the inside of the body of the subject 22 captured by the camera 48.

[0059] A communication module 60 is connected to the input / output interface 56. The communication module 60 is an interface including a communication processor, an antenna, etc. The communication module 60 is connected to a network (not shown) such as a LAN or a WAN, and controls communication between the processor 62 and external devices.

[0060] The display device 14 is connected to the input / output interface 56, and the processor 62 controls the display device 14 via the input / output interface 56 to cause the display device 14 to display various information.

[0061] The input / output interface 56 is connected to the reception device 52, and the processor 62 acquires instructions received by the reception device 52 via the input / output interface 56 and executes processing according to the acquired instructions.

[0062] The NVM 66 stores a lesion detection program 76 and a trained model 78. The processor 62 reads the lesion detection program 76 from the NVM 66 and executes the read lesion detection program 76 on the RAM 64 to perform lesion detection processing. The lesion detection processing is processing for detecting a lesion from the observation region 72 using an AI method. By performing the lesion detection processing, the processor 62 detects a location corresponding to a lesion from the radial ultrasound image 24 (see FIG. 1 ) in accordance with the trained model 78, thereby detecting a lesion from the observation region 72. The lesion detection processing is realized by the processor 62 operating as a generation unit 62A, a detection unit 62B, and a control unit 62C in accordance with the lesion detection program 76 executed on the RAM 64.

[0063] The trained model 78 is a trained model having a data structure used in processing to identify a lesion from a radial ultrasound image. The trained model 78 is also an example of a “trained model” according to the technology of the present disclosure.

[0064] The trained model 78 is a neural network (NN) used to detect a location corresponding to a lesion from the radial ultrasound image 24. Therefore, in order to obtain the trained model 78, the ideal ultrasound image to train an untrained NN is a radial ultrasound image generated by a radial scanning ultrasound endoscope.

[0065] However, currently, the number of radial-type endoscopic ultrasound examinations performed is overwhelmingly smaller than the number of convex-type endoscopic ultrasound examinations performed, making it difficult to collect the number of radial-type ultrasound images used for training the neural network necessary to achieve the target detection accuracy.

[0066] On the other hand, the number of convex endoscopic ultrasound examinations performed is overwhelmingly greater than the number of radial endoscopic ultrasound examinations performed. This means that the number of convex ultrasound images generated by convex endoscopic ultrasound examinations is overwhelmingly greater than the number of radial ultrasound images. In other words, it is easier to acquire convex ultrasound images than radial ultrasound images.

[0067] Therefore, in this embodiment, an image that is generated based on a convex ultrasound image and that mimics at least a part of a radial ultrasound image is used as a training ultrasound image for obtaining the trained model 78. This will be specifically described below.

[0068] As an example, as shown in FIG. 4 , a learning device 80 includes a computer 82, an input / output interface 84, a reception device 86, a display device 88, and a communication module 90. The computer 82 includes a processor 92, a RAM 94, and an NVM 96. The input / output interface 84, the processor 92, the RAM 94, and the NVM 96 are connected to a bus 97. The learning device 80 is an example of a "learning device" according to the technology of the present disclosure. The computer 82 is an example of a "computer" according to the technology of the present disclosure. The processor 92 is an example of a "first processor" according to the technology of the present disclosure. The NVM 96 is an example of a "first memory," a "second memory," and a "third memory" according to the technology of the present disclosure.

[0069] Note that the multiple hardware resources (i.e., processor 92, RAM 94, and NVM 96) included in the computer 82 shown in Fig. 4 are of the same type as the multiple hardware resources included in the computer 54 shown in Fig. 3, and therefore a description of the overlapping parts will be omitted. Also, the input / output interface 84 shown in Fig. 4 is of the same type as the input / output interface 56 shown in Fig. 3, the reception device 86 shown in Fig. 4 is of the same type as the reception device 52 shown in Fig. 3, the display device 88 shown in Fig. 4 is of the same type as the display device 14 shown in Fig. 3, and the communication module 90 shown in Fig. 4 is of the same type as the communication module 60 shown in Fig. 3, and therefore a description thereof will be omitted here.

[0070] The NVM 96 stores an untrained model 98 and a learning execution program 100. An example of the model 98 is a mathematical model using a neural network (NN). Examples of types of NN include YOLO, R-CNN, and FCN. The NN used in the model 98 may be, for example, YOLO, R-CNN, or a combination of FCN and RNN. RNN is suitable for learning multiple images obtained in a time series. Note that the types of NN listed here are merely examples, and other types of NNs that enable object detection by learning images may also be used.

[0071] The processor 92 controls the entire learning device 80. The processor 92 performs a learning execution process by reading a learning execution program 100 from the NVM 96 and executing the read learning execution program 100 on the RAM 94. The learning execution process is a process for creating a trained model 78 (see FIG. 3 ) by executing learning using teacher data for the model 98. The learning execution process is realized by the processor 92 operating as an acquisition unit 92A and a learning execution unit 92B in accordance with the learning execution program 100 executed on the RAM 94.

[0072] The model 98 is an example of a "model" according to the technology of the present disclosure. The learning execution program 100 is an example of a "program" according to the technology of the present disclosure. The learning execution process is an example of a "process" according to the technology of the present disclosure.

[0073] As an example, as shown in Fig. 5, a plurality of medical images 102 are pre-stored in the NVM 96. The plurality of medical images 102 are images obtained from a plurality of subjects (e.g., a plurality of subjects other than the subject 22 shown in Fig. 1, or a plurality of subjects including the subject 22). The plurality of medical images 102 include an image generated based on at least one convex ultrasonic image 104 and simulating at least a portion of a radial ultrasonic image. The convex ultrasonic image 104 is an ultrasonic image obtained in the same image mode (here, B-mode, as an example) as the radial ultrasonic image 24 shown in Fig. 1.

[0074] The types of medical images 102 are broadly classified into five types: circular images 102A, scaled images 102B, rotated images 102C, radial ultrasound images 102D, and virtual images 102E. The NVM 96 stores a plurality of different circular images 102A, a plurality of different scaled images 102B, a plurality of different rotated images 102C, a plurality of different radial ultrasound images 102D, and a plurality of different virtual images 102E as medical images 102. The circular images 102A, the scaled images 102B, and the rotated images 102C are generated based on at least one convex ultrasound image 104, and are images that mimic at least a portion of the radial ultrasound image. An image that mimics at least a portion of a radial ultrasound image refers to, for example, an image that is closer in shape to a radial ultrasound image than an unprocessed convex ultrasound image obtained from a convex ultrasound endoscopic examination, and / or an image obtained by adjusting a convex ultrasound image to a scale close to or the same as that of a radial ultrasound image.

[0075] In this embodiment, the circular image 102A, the scale-adjusted image 102B, and the rotated image 102C are examples of "medical images" according to the technology of the present disclosure. The circular image 102A is an example of a "circular image" according to the technology of the present disclosure. The scale-adjusted image 102B is an example of a "scale-adjusted image" according to the technology of the present disclosure. The rotated image 102C is an example of a "rotated image" according to the technology of the present disclosure. The radial ultrasound image 102D is an example of a "first ultrasound image" according to the technology of the present disclosure. The virtual image 102E is an example of a "virtual ultrasound image" according to the technology of the present disclosure.

[0076] As will be described in detail later, the circular image 102A is an image generated based on the convex ultrasound images 104A and 104B. The outer shape of the circular image 102A does not need to be a perfect circle, and may be an incomplete circle. An incomplete circle refers to a shape that is closer to the outer shape of a radial ultrasound image (i.e., a circle) than to the outer shape of an unprocessed convex ultrasound image obtained from a convex ultrasound endoscopy (i.e., a sector shape). An example of an incomplete circle is a circle with a partial gap (e.g., a circle with a partial cutout) as shown in FIG. 5.

[0077] Circular image 102A depicts a lesion. That is, circular image 102A has lesion area 110A, which is a location corresponding to the lesion. Annotation 106A is attached to circular image 102A. Annotation 106A is information that can identify the position of lesion area 110A within circular image 102A (e.g., information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing lesion area 110A).

[0078] For ease of explanation, information capable of identifying the position of the lesion region 110A within the circular image 102A is illustrated as an example of the annotation 106A, but this is merely an example. For example, the annotation 106A may include other types of information that identify the lesion depicted in the circular image 102A, such as information capable of identifying the type of lesion depicted in the circular image 102A.

[0079] As will be described in more detail below, scale-adjusted image 102B is an image generated based on convex ultrasound image 104C. A lesion is captured in scale-adjusted image 102B. That is, scale-adjusted image 102B has lesion region 110B, which is a location corresponding to the lesion. Annotation 106B is added to scale-adjusted image 102B. Annotation 106B is information that can identify the position of lesion region 110B within scale-adjusted image 102B (e.g., information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing lesion region 110B).

[0080] For ease of explanation, the annotation 106B shown here is information that can identify the position of the lesion region 110B in the scaled image 102B, but this is merely an example. For example, the annotation 106B may include other types of information that identify the lesion shown in the scaled image 102B, such as information that can identify the type of lesion shown in the scaled image 102B.

[0081] As will be described in more detail below, rotated image 102C is an image generated based on convex ultrasound image 104D. Rotated image 102C depicts a lesion. That is, rotated image 102C includes lesion area 110C, which is a location corresponding to the lesion. Annotation 106C is added to rotated image 102C. Annotation 106C is information that can identify the position of lesion area 110C within rotated image 102C (e.g., information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing lesion area 110C).

[0082] For ease of explanation, information capable of identifying the position of the lesion region 110C in the rotated image 102C is illustrated as an example of the annotation 106C here, but this is merely an example. For example, the annotation 106C may include other types of information that identify the lesion shown in the rotated image 102C, such as information capable of identifying the type of lesion shown in the rotated image 102C.

[0083] As will be described in detail later, the radial type ultrasound image 102D is an ultrasound image obtained by an actual radial type endoscopic ultrasound examination. A lesion is captured in the radial type ultrasound image 102D. That is, the radial type ultrasound image 102D has a lesion area 110D, which is a location corresponding to the lesion. An annotation 106D is added to the radial type ultrasound image 102D. The annotation 106D is information that can identify the position of the lesion area 110D within the radial type ultrasound image 102D (for example, information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing the lesion area 110D).

[0084] Here, for convenience of explanation, information capable of identifying the position of the lesion region 110D in the radial ultrasound image 102D is illustrated as an example of the annotation 106D, but this is merely an example. For example, the annotation 106D may include other types of information for identifying the lesion shown in the radial ultrasound image 102D, such as information capable of identifying the type of lesion shown in the radial ultrasound image 102D.

[0085] As will be described in more detail below, virtual image 102E is a virtual ultrasound image simulating a radial ultrasound image. A lesion is depicted in virtual image 102E. That is, virtual image 102E has lesion region 110E, which is a location corresponding to the lesion. Annotation 106E is attached to virtual image 102E. Annotation 106E is information that can identify the position of lesion region 110E within virtual image 102E (e.g., information including a plurality of coordinates that can identify the position of a rectangular frame circumscribing lesion region 110E).

[0086] For ease of explanation, information capable of identifying the position of lesion region 110E in virtual image 102E is illustrated as an example of annotation 106E here, but this is merely an example. For example, annotation 106E may include other types of information that identify a lesion shown in virtual image 102E, such as information capable of identifying the type of lesion shown in virtual image 102E.

[0087] For ease of explanation, the circular image 102A, the scaled image 102B, the rotated image 102C, the radial ultrasound image 102D, and the virtual image 102E will be referred to as the "medical image 102" when there is no need to distinguish between them. For ease of explanation, the annotations 106A to 106E will be referred to as the "annotation 106" when there is no need to distinguish between them. For ease of explanation, the lesion areas 110A to 110E will be referred to as the "lesion area 110." The annotation 106 is an example of an "annotation" according to the technology of the present disclosure.

[0088] The acquisition unit 92A acquires multiple medical images 102 from the NVM 96. For example, the acquisition unit 92A acquires, frame by frame, medical images 102 that have not yet been used in training the model 98 from the NVM 96. The learning execution unit 92B causes the model 98 to learn the medical images 102 acquired by the acquisition unit 92A. Note that, for ease of explanation, in the following, in order to facilitate understanding of the processing performed by the learning execution unit 92B, some of the processing performed by the learning execution unit 92B in accordance with the model 98 will be described as processing actively performed by the model 98. In other words, for ease of explanation, the model 98 will be described as a function that performs processing (e.g., processing including image recognition processing) on ​​input information (e.g., information including an image) and outputs the processing results.

[0089] The acquisition unit 92A inputs a training image 112, which is a portion of the medical image 102 other than the annotation 106 (i.e., the main body of the medical image 102), to the model 98. When the training image 112 is input, the model 98 predicts the position of the lesion region 110 and outputs a prediction result 116. The prediction result 116 is information that can identify the position predicted by the model 98 as the position of the lesion region 110 within the training image 112. An example of information that can identify the position predicted by the model 98 is information that includes a plurality of coordinates that can identify the position of a bounding box surrounding the region predicted to contain the lesion region 110 (i.e., the position of the bounding box within the training image 112).

[0090] The learning execution unit 92B calculates the error between the prediction result 116 and the annotation 106 corresponding to the prediction result 116 (i.e., the annotation 106 that was attached to the learning image 112 that was input to the model 98 for outputting the prediction result 116). The learning execution unit 92B then makes adjustments to the model 98 in accordance with the calculated error. That is, the learning execution unit 92B adjusts multiple optimization variables (e.g., multiple connection weights and multiple offset values) in the model 98 so as to minimize the calculated error.

[0091] The acquisition unit 92A and the learning execution unit 92B optimize the model 98 by repeatedly performing a learning process, which is a series of processes including inputting the training images 112 into the model 98, calculating the error, and adjusting multiple optimization variables, for each of the multiple medical images 102 (e.g., all of the medical images 102) stored in the NVM 96. For example, the multiple optimization variables in the model 98 are adjusted to minimize the error, thereby optimizing the model 98 and generating the trained model 78. In other words, the data structure of the trained model 78 is obtained by having the model 98 learn multiple medical images 102 to which annotations 106 have been added.

[0092] The learning execution unit 92B transmits the trained model 78 to the processing device 18 via the communication module 90 (see FIG. 4 ). In the processing device 18, the processor 62 receives the trained model 78 via the communication module 60 (see FIG. 3 ) and stores the received trained model 78 in the NVM 66.

[0093] 6, a convex ultrasound image 104 is generated under a specific image mode by a convex ultrasound endoscopic device 118 having a convex ultrasound probe 118A. There are multiple convex ultrasound images 104, and they are obtained from convex ultrasound endoscopic examinations (e.g., examinations using one or more convex ultrasound endoscopic devices 118) performed on multiple subjects, respectively. The multiple subjects are different from subject 22. Note that the multiple subjects may include subject 22.

[0094] The circular image 102A is an image generated by combining the convex ultrasound images 104A and 104B. In the example shown in Fig. 6, the convex ultrasound image 104B includes a lesion area 110A. Note that this is merely an example, and there may be cases where a lesion is captured in the convex ultrasound image 104A, or in both the convex ultrasound images 104A and 104B.

[0095] The convex ultrasonic image 104A and the convex ultrasonic image 104B are ultrasonic images obtained by emitting ultrasonic waves in opposite directions from the ultrasonic probe 118A, and are in a linearly symmetrical positional relationship. The scales of the convex ultrasonic images 104A and 104B are adjusted based on the scale of the radial ultrasonic image 24. That is, since the scale of the radial ultrasonic image 24 is smaller than the scale of the convex ultrasonic images 104A and 104B, the convex ultrasonic images 104A and 104B are reduced so that the scales of the convex ultrasonic images 104A and 104B match the scale of the radial ultrasonic image 24. The reduced convex ultrasonic images 104A and 104B are then combined while maintaining the linearly symmetrical positional relationship.

[0096] In this way, the convex ultrasound images 104A and 104B are reduced, and the reduced convex ultrasound images 104A and 104B are combined while maintaining the linear symmetrical positional relationship to obtain the circular image 102A. An annotation 106A corresponding to the lesion area 110A is added to the circular image 102A.

[0097] In the example shown in FIG. 6, the convex ultrasound images 104A and 104B are arranged linearly symmetrically in the vertical direction, but this is merely an example, and a pair of convex ultrasound images may be arranged linearly symmetrically in the horizontal or diagonal direction.

[0098] In addition, although an example in which a pair of convex ultrasound images are arranged line-symmetrically and combined has been described here, this is merely an example. For example, three or more frames of convex ultrasound images obtained by performing convex scanning in different directions (e.g., three or more directions) may be combined. In this case, if image areas overlap between adjacent convex ultrasound images, the overlapping image area may be removed from one of the adjacent convex ultrasound images, and the adjacent convex ultrasound images may be combined.

[0099] In addition, although an example in which a pair of convex ultrasound images are combined after adjusting the scale has been described here, this is merely an example, and the scale may be adjusted after the pair of convex ultrasound images are combined.

[0100] The scale-adjusted image 102B is an image obtained by adjusting the scale of the convex ultrasound image 104C including the lesion region 110B based on the scale of the radial ultrasound image 24. That is, because the scale of the radial ultrasound image 24 is smaller than the scale of the convex ultrasound image 104C, the convex ultrasound image 104C is reduced so that the scale of the convex ultrasound image 104C matches the scale of the radial ultrasound image 24. The image obtained by reducing the convex ultrasound image 104C in this manner is the scale-adjusted image 102B. An annotation 106B corresponding to the lesion region 110B is added to the scale-adjusted image 102B.

[0101] The rotated image 102C is an image obtained by rotating the convex ultrasound image 104D including the lesion area 110C. The rotation angle is, for example, a rotation angle that is specified in advance. There are multiple rotation angles, and a rotated image 102C exists for each rotation angle. The rotation angle is, for example, determined in units of a predetermined angle (for example, in 1-degree increments) within a range of 0 degrees or more and less than 360 degrees. In other words, for one convex ultrasound image 104D, rotated images 102C are generated for the number of rotation angles.

[0102] Similarly to the circular image 102A and the scale-adjusted image 102B, the rotated image 102C is also adjusted based on the scale of the radial ultrasound image 24. That is, the convex ultrasound image 104D is reduced so that the scale of the convex ultrasound image 104D matches the scale of the radial ultrasound image 24. In this way, the convex ultrasound image 104D is rotated by a specified rotation angle, and the image obtained by reducing the rotated convex ultrasound image 104D is the rotated image 102C. An annotation 106C corresponding to the lesion area 110C is added to the rotated image 102C.

[0103] Note that, although an example is given here in which the convex ultrasound image 104D is rotated and then the scale is adjusted, this is merely an example, and the rotated image 102C may also be generated by rotating the convex ultrasound image 104D whose scale has been adjusted.

[0104] The radial ultrasound image 102D is a radial ultrasound image obtained by a radial ultrasound endoscopic examination of a subject (see FIG. 1) different from the subject 22. There are a plurality of radial ultrasound images 102D, and a plurality of radial ultrasound images 102D are generated for each of a plurality of subjects different from the subject 22. The plurality of subjects may include the subject 22.

[0105] Radial type ultrasonic endoscopic examination uses, for example, a radial type ultrasonic endoscopic device 120. The radial type ultrasonic endoscopic device 120 is preferably a device with the same specifications as the ultrasonic endoscopic device 12. Furthermore, it is preferable that the radial type ultrasonic endoscopic device 120 has the same parameters set as the various parameters set in the ultrasonic endoscopic device 12 as parameters controlling image quality.

[0106] The radial type ultrasound image 102D is a radial type ultrasound image obtained by the radial type ultrasound endoscopic device 120 under a specific image mode. The radial type ultrasound image 102D is also a radial type ultrasound image obtained earlier than the radial type ultrasound image 24 (see FIG. 1). In other words, the radial type ultrasound image 102D is a radial type ultrasound image obtained from a radial type ultrasound endoscopic examination performed earlier than the radial type ultrasound endoscopic examination shown in FIG. 1. The observation region shown in the radial type ultrasound image 102D is anatomically the same region as the observation region shown in the radial type ultrasound image 24 (see FIG. 1). The radial type ultrasound image 102D includes a lesion region 110D, and an annotation 106D corresponding to the lesion region 110D is added.

[0107] The virtual image 102E is generated based on the volume data 122 representing the subject, and is a virtual ultrasound image simulating a radial ultrasound image. There are a plurality of virtual images 102E, and each virtual image 102E is generated for each of the volume data 122 representing the subject. The plurality of subjects may include the subject 22.

[0108] The volume data 122 is a three-dimensional image defined by voxels, which is a stack of multiple two-dimensional slice images 124 obtained by imaging the entire body or a part of the subject (e.g., the abdomen) using a modality. The position of each voxel is specified by three-dimensional coordinates. An example of a modality is a CT device. The CT device is just one example, and other examples of modalities include an MRI device, an ultrasound diagnostic device, etc.

[0109] The virtual image 102E includes a lesion area 110E, and an annotation 106E corresponding to the lesion area 110E is added.

[0110] 7, in the ultrasonic endoscope device 12, the generation unit 62A acquires a reflected wave signal 74 from the transmission / reception circuit 58 and generates a radial ultrasonic image 24 based on the acquired reflected wave signal 74. In the example shown in Fig. 7, the generation unit 62A generates a radial ultrasonic image 24 including a lesion area 126, which is a location corresponding to a lesion, but of course, the generation unit 62A may also generate a radial ultrasonic image 24 that does not show a lesion.

[0111] The detection unit 62B acquires the trained model 78 from the NVM 66. Then, the detection unit 62B detects a lesion from the radial ultrasound image 24 generated by the generation unit 62A according to the acquired trained model 78. That is, the detection unit 62B determines the presence or absence of a lesion area 126 in the radial ultrasound image 24 according to the trained model 78, and generates position identification information 128 (e.g., information including a plurality of coordinates identifying the position of the lesion area 126) that identifies the position of the lesion area 126 if the lesion area 126 is present in the radial ultrasound image 24. Note that, for ease of explanation, in the following, in order to facilitate understanding of the processing performed by the detection unit 62B, some of the processing performed by the detection unit 62B according to the trained model 78 will be described as processing actively performed mainly by the trained model 78. That is, for ease of explanation, the trained model 78 will be described as a function that performs processing (e.g., processing including image recognition processing) on ​​input information (e.g., information including an image) and outputs the processing result.

[0112] The detection unit 62B inputs the radial type ultrasound image 24 generated by the generation unit 62A to the trained model 78. When the radial type ultrasound image 24 is input, the trained model 78 determines whether or not a lesion area 126 exists in the radial type ultrasound image 24. Here, if the trained model 78 determines that a lesion area 126 exists in the radial type ultrasound image 24 (i.e., if a lesion shown in the radial type ultrasound image 24 is detected), it outputs position identification information 128. The detection unit 62B generates a detection frame 128A based on the position identification information 128 output from the trained model 78. The detection frame 128A is a rectangular frame corresponding to a bounding box (e.g., a bounding box with the highest reliability score) used when the trained model 78 detects the lesion area 126 from the radial type ultrasound image 24. In other words, the detection frame 128A is a frame surrounding the lesion area 126 detected by the trained model 78.

[0113] The detection unit 62B assigns a detection frame 128A to the radial type ultrasound image 24 corresponding to the position identification information 128 output from the trained model 78 (i.e., the radial type ultrasound image 24 input to the trained model 78 for output of the position identification information 128) in accordance with the position identification information 128. That is, the detection unit 62B assigns the detection frame 128A to the radial type ultrasound image 24 corresponding to the position identification information 128 output from the trained model 78 by superimposing the detection frame 128A on the radial type ultrasound image 24 so as to surround the lesion area 126. When the trained model 78 determines that the lesion area 126 exists in the radial type ultrasound image 24, the detection unit 62B outputs the radial type ultrasound image 24 with the assigned detection frame 128A to the control unit 62C. In addition, if the learned model 78 determines that no lesion area 126 exists in the radial ultrasound image 24, the detection unit 62B outputs the radial ultrasound image 24 without the detection frame 128A to the control unit 62C.

[0114] The control unit 62C displays the radial type ultrasound image 24 input from the detection unit 62B (i.e., the radial type ultrasound image 24 reflecting the detection result of the detection unit 62B) on the screen 26 of the display device 14. When a lesion is captured in the radial type ultrasound image 24, the radial type ultrasound image 24 to which a detection frame 128A surrounding the lesion area 126 is added (i.e., the radial type ultrasound image 24 to which the detection frame 128A is superimposed) is displayed on the screen 26. On the other hand, when a lesion is not captured in the radial type ultrasound image 24, the radial type ultrasound image 24 to which the detection frame 128A is not added (i.e., the radial type ultrasound image 24 output from the trained model 78) is displayed on the screen 26.

[0115] Next, the operation of the learning device 80 will be described with reference to FIG.

[0116] 8 shows an example of the flow of a learning execution process performed by the processor 92 of the learning device 80. The flow of the learning execution process shown in FIG. 8 is an example of a "learning method" according to the technology of the present disclosure.

[0117] 8 , first, in step ST10, the acquisition unit 92A acquires one frame of medical image 102 that has not yet been used in learning the model 98 from the NVM 96. After the process of step ST10 is executed, the learning execution process proceeds to step ST12.

[0118] In step ST12, the learning execution unit 92B inputs the learning image 112 obtained from the medical image 102 acquired in step ST10 into the model 98. After the processing of step ST12 is executed, the learning execution processing proceeds to step ST14.

[0119] In step ST14, the learning execution unit 92B calculates the error between the annotation 106 added to the medical image 102 acquired in step ST10 and the prediction result 116 output from the model 98 by executing the process of step ST12. After the process of step ST14 is executed, the learning execution process proceeds to step ST16.

[0120] In step ST16, the learning execution unit 92B performs adjustments to the model 98 in accordance with the errors calculated in step ST14. After the processing of step ST14 is performed, the learning execution processing proceeds to step ST18.

[0121] In step ST18, the learning execution unit 92B determines whether a condition for terminating the learning execution process (hereinafter referred to as the "learning termination condition") has been satisfied. A first example of the learning termination condition is that all medical images 102 in the NVM 96 have been used in training the model 98. A second example of the learning termination condition is that an instruction to terminate the learning execution process has been received by the reception device 86.

[0122] In step ST18, if the learning termination condition is not satisfied, the determination is negative and the learning execution process proceeds to step ST 10. In step ST18, if the learning termination condition is satisfied, the determination is positive and the learning execution process ends.

[0123] The processes of steps ST10 to ST18 are repeatedly executed to optimize the model 98 and generate the trained model 78. The trained model 78 thus generated is stored in the NVM 66 (see FIGS. 3 and 5).

[0124] Next, the operation of the endoscope system 10 will be described with reference to FIG.

[0125] FIG. 9 shows an example of the flow of the lesion detection process performed by the processor 62 of the processing device 18.

[0126] 9 , first, in step ST50, the generator 62A acquires the reflected wave signal 74 from the transmission / reception circuitry 58 and generates one frame of the radial ultrasound image 24 based on the acquired reflected wave signal 74. After the processing of step ST50 is performed, the lesion detection processing proceeds to step ST52.

[0127] In step ST52, the detection unit 62B inputs the radial ultrasound image 24 generated in step ST50 into the trained model 78. After the processing of step ST52 is executed, the lesion detection processing proceeds to step ST54.

[0128] In step ST54, the detection unit 62B determines whether or not a lesion is captured in the radial ultrasound image 24 input to the trained model 78 in step ST52, using the trained model 78. If a lesion is captured in the radial ultrasound image 24, the trained model 78 outputs position identification information 128.

[0129] In step ST54, if a lesion is not captured in the radial ultrasound image 24, the determination is negative and the lesion detection process proceeds to step ST58. In step ST54, if a lesion is captured in the radial ultrasound image 24, the determination is positive and the lesion detection process proceeds to step ST56.

[0130] If the determination in step ST54 is affirmative, the detection unit 62B generates a detection frame 128A based on the position identification information 128 output from the trained model 78, and superimposes the detection frame 128A on the radial ultrasound image 24 generated in step ST50 so as to surround the lesion area 126. Then, in step ST56, the control unit 62C displays the radial ultrasound image 24, in which the lesion area 126 is surrounded by the detection frame 128A, on the screen 26 of the display device 14. Because the lesion area 126 in the radial ultrasound image 24 is surrounded by the detection frame 128A, the doctor 20 can visually determine where the lesion is located in the radial ultrasound image 24. After the processing of step ST56 is executed, the lesion detection processing proceeds to step ST60.

[0131] In step ST58, the control unit 62C displays the radial ultrasonic image 24 generated in step ST50 on the screen 26 of the display device 14. In this case, since the detection frame 128A is not added to the radial ultrasonic image 24, the doctor 20 can visually recognize that no lesion is visible in the radial ultrasonic image 24. After the processing of step ST58 is executed, the lesion detection processing proceeds to step ST60.

[0132] In step ST60, control unit 62C determines whether a condition for terminating the lesion detection process (hereinafter referred to as a "lesion detection termination condition") has been satisfied. One example of the lesion detection termination condition is that an instruction to terminate the lesion detection process has been accepted by acceptance device 52. If the lesion detection termination condition has not been satisfied in step ST60, the determination is negative, and the lesion detection process proceeds to step ST50. If the lesion detection termination condition has been satisfied in step ST60, the determination is positive, and the lesion detection process ends.

[0133] As described above, the annotations 106 are attached to each of the multiple medical images 102 stored in the NVM 96 of the learning device 80. The annotations 106 are information that can identify the position of the lesion region 110 within the medical image 102. The medical image 102 is generated based on at least one convex ultrasound image 104 and is an image that mimics at least a portion of a radial ultrasound image. The multiple medical images 102 configured in this manner are used to train the model 98. That is, the model 98 is optimized by repeatedly performing a training process for each of the multiple medical images 102 stored in the NVM 96, including inputting the training image 112, which is the main body of the medical image 102, into the model 98, calculating an error, and adjusting multiple optimization variables. Optimizing the model 98 generates a trained model 78, which is used to detect lesions captured in the radial ultrasound image 24. In this way, according to the learning device 80, it is possible to obtain a trained model 78 that contributes to identifying a lesion shown in the radial ultrasound image 24, even if the model 98 does not learn only radial ultrasound images (for example, the radial ultrasound image 102D shown in Figures 5 and 6).

[0134] The multiple medical images 102 stored in the NVM 96 of the learning device 80 include a circular image 102A. The circular image 102A is an image generated by combining convex ultrasound images 104A and 104B. Because the outer shape of the radial ultrasound image 24 is also circular, the trained model 78 obtained by training the model 98 on the circular image 102A, which has the same outer shape, can contribute to highly accurate identification of lesions appearing in the radial ultrasound image 24.

[0135] The scale of the convex ultrasound images 104A and 104B on which the circular image 102A is based is different from the scale of the radial ultrasound image 24. Therefore, the learning device 80 generates the circular image 102A by combining the convex ultrasound images 104A and 104B whose scales have been adjusted based on the scale of the radial ultrasound image 24 (for example, the convex ultrasound images 104A and 104B whose scales have been matched to that of the radial ultrasound image 24), and trains the model 98. This can improve the accuracy of identifying a lesion from the radial ultrasound image 24 according to the trained model 78, compared to when the model 98 is trained with the circular image 102A whose scale is the same as that of the convex ultrasound images 104A and 104B.

[0136] A plurality of mutually different circular images 102A are stored in advance in the NVM 96 of the learning device 80. The processor 92 then acquires the circular images 102A from the NVM 96 and causes the acquired circular images 102A to be learned by the model 98. Therefore, the learning device 80 can cause the model 98 to learn the circular images 102A without causing the processor 92 to generate a circular image 102A every time the model 98 is trained.

[0137] The number of directions in which ultrasound is emitted in radial scanning is greater than the number of directions in which ultrasound is emitted in convex scanning. Therefore, in order to enable detection of lesions in areas corresponding to directions in which ultrasound is not emitted in convex scanning, the multiple medical images 102 stored in the NVM 96 of the learning device 80 include a rotated image 102C. The rotated image 102C is an image obtained by rotating the convex ultrasound image 104D. Therefore, by having the model 78 learn the rotated image 102C, it is possible to improve the accuracy of identifying lesions from the radial ultrasound image 24 according to the trained model 98, even if lesions are captured in various positions within the radial ultrasound image 24 used for diagnosis.

[0138] The scale of the convex ultrasound image 104D on which the rotated image 102C is based is different from the scale of the radial ultrasound image 24. Therefore, the learning device 80 generates the rotated image 102C by adjusting the scale of the rotated convex ultrasound image 104D based on the scale of the radial ultrasound image 24 (for example, by matching the scale of the rotated convex ultrasound image 104D to the scale of the radial ultrasound image 24), and trains the generated rotated image 102C in the model 98. This can improve the accuracy of identifying a lesion from the radial ultrasound image 24 according to the trained model 78, compared to when the model 98 is trained on a rotated image 102C having the same scale as the convex ultrasound image 104D.

[0139] A plurality of mutually different rotated images 102C are stored in advance in the NVM 96 of the learning device 80. Then, the processor 92 acquires the rotated images 102C from the NVM 96 and causes the acquired rotated images 102C to be learned by the model 98. Therefore, the learning device 80 can cause the model 98 to learn the rotated images 102C without causing the processor 92 to generate the rotated images 102C every time the model 98 is trained.

[0140] The scale of the convex ultrasound image 104C is different from the scale of the radial ultrasound image 24. Therefore, the learning device 80 adjusts the scale of the convex ultrasound image 104C based on the scale of the radial ultrasound image 24 (for example, by matching the scale of the convex ultrasound image 104C to the scale of the radial ultrasound image 24), thereby generating a scale-adjusted image 102B and having the model 98 learn the image. This can improve the accuracy of identifying a lesion from the radial ultrasound image 24 according to the trained model 78, compared to when the model 98 learns the convex ultrasound image 104C without adjusting the scale of the convex ultrasound image 104C.

[0141] A plurality of different scaled images 102B are stored in advance in the NVM 96 of the learning device 80. The processor 92 then acquires the scaled images 102B from the NVM 96 and causes the acquired scaled images 102B to be learned by the model 98. Therefore, the learning device 80 can cause the model 98 to learn the scaled images 102B without causing the processor 92 to generate the scaled images 102B every time the model 98 is trained.

[0142] In the learning device 80, the processor 92 acquires the radial type ultrasound image 102D from the NVM 96, and trains the acquired radial type ultrasound image 102D into the model 98. Therefore, compared to a case where the radial type ultrasound image 102D is not trained into the model 98 (i.e., compared to a case where only images generated based on the convex type ultrasound image 104 are trained into the model 98), it is possible to obtain a trained model 78 that contributes to highly accurate detection of lesions from the radial type ultrasound image 24.

[0143] A virtual image 102E is stored in the NVM 96 of the learning device 80. The virtual image 102E is generated based on the volume data 122 and is a virtual ultrasound image that mimics a radial ultrasound image. The processor 92 acquires the virtual image 102E from the NVM 96 and trains the model 98 on the acquired virtual image 102E. Therefore, even if there are insufficient actual ultrasound images (e.g., convex ultrasound images 104 and / or radial ultrasound images 102D) used to train the model 98, the shortage can be made up for by the virtual image 102E.

[0144] [First Modification] In the above embodiment, an example has been described in which the acquisition unit 92A acquires multiple medical images 102 pre-stored in the NVM 96 and causes the model 98 to learn from the acquired multiple medical images 102. However, the technology of the present disclosure can be implemented even if the multiple medical images 102 are not pre-stored in the NVM 96. For example, as shown in FIG. 10 , the acquisition unit 92A may acquire the medical image 102 by randomly selecting one generation method 129 from multiple generation methods 129 for generating the medical image 102 based on at least one convex ultrasound image 104 and generating the medical image 102 according to the selected generation method 129. In this case, too, the model 98 may be trained on the medical image 102 acquired by the acquisition unit 92A in the same manner as in the above embodiment.

[0145] 10 shows a first generation method 129A, a second generation method 129B, and a third generation method 129C as examples of the plurality of generation methods 129. The first generation method 129A is a method that includes generating a circular image 102A as the medical image 102 by combining a plurality of convex ultrasound images 104, and adjusting the scale of the circular image 102A based on the scale of the radial ultrasound image 24. The second generation method 129B is a method that includes generating a rotated image 102C as the medical image 102 by rotating the convex ultrasound image 104, and generating the scale of the rotated image 102C based on the scale of the radial ultrasound image 24. The third generation method 129C is a method that includes adjusting the scale of the convex ultrasound image 104 based on the scale of the radial ultrasound image 24, to generate a scale-adjusted image 102B as the medical image 102.

[0146] The NVM 96 stores a sample image group 130. The sample image group 130 is made up of a plurality of convex ultrasound images 104. The plurality of convex ultrasound images 104 constituting the sample image group 130 are images that serve as the basis for the medical images 102 (e.g., a circular image 102A, a scaled image 102B, and a rotated image 102C). That is, the sample image group 130 includes convex ultrasound images 104A, 104B, 104C, and 104D shown in FIG. 6 . The plurality of convex ultrasound images 104 constituting the sample image group 130 also includes a convex ultrasound image 104 that has a lesion area 110 and to which an annotation 106 corresponding to the lesion area 110 has been added.

[0147] The acquisition unit 92A randomly selects one generation method 129 from the first generation method 129A, the second generation method 129B, and the third generation method 129C, and acquires at least one convex ultrasound image 104 to be used in the selected generation method 129 from the sample image group 130.

[0148] When the acquisition unit 92A selects the first generation method 129A, it acquires convex ultrasound images 104A and 104B from the sample image group 130. Each time the acquisition unit 92A selects the first generation method 129A, it acquires a different combination of convex ultrasound images 104A and 104B from the sample image group 130. In this case, at least one of the convex ultrasound images 104A and 104B has a lesion area 110. That is, an annotation 106 is added to at least one of the convex ultrasound images 104A and 104B. The acquisition unit 92A generates a circular image 102A in a manner similar to the example shown in FIG. 6 using the convex ultrasound images 104A and 104B acquired from the sample image group 130.

[0149] When the acquisition unit 92A selects the second generation method 129B, it acquires a convex ultrasound image 104D from the sample image group 130. Every time the acquisition unit 92A selects the second generation method 129B, it randomly acquires a convex ultrasound image 104D from the sample image group 130 and randomly determines the rotation angle by which to rotate the convex ultrasound image 104D. Then, the acquisition unit 92A generates a rotated image 102C in a manner similar to the example shown in FIG. 6 using the convex ultrasound image 104D acquired from the sample image group 130.

[0150] When the third generation method 129C is selected, the acquisition unit 92A acquires a convex ultrasound image 104C that has not yet been used to generate the scaled image 102B from the sample image group 130. Then, the acquisition unit 92A generates the scaled image 102B in a manner similar to the example shown in FIG. 6 using the convex ultrasound image 104C acquired from the sample image group 130.

[0151] An example of the flow of the learning execution process according to the first modified example is shown in Fig. 11. The flowchart shown in Fig. 11 differs from the flowchart shown in Fig. 8 in that the process of step ST10 is replaced by the process of step ST10A and the process of step ST10B.

[0152] 11, in step ST10A, the acquisition unit 92A selects one generation method 129 from a plurality of generation methods 129 (for example, first to third generation methods 129A to 129C). After the process of step ST10A is executed, the learning execution process proceeds to step ST10B.

[0153] In step ST10B, the acquisition unit 92A generates a medical image 102 (e.g., a circular image 102A, a scaled image 102B, or a rotated image 102C) according to the generation method 129 selected in step ST10B (e.g., any of the first to third generation methods 129A to 129C).

[0154] In step ST12, the learning image 112 (see FIG. 5) obtained from the medical image 102 generated in step ST10B is input to the model 98. Note that the radial ultrasound image 102D and / or the virtual image 102E may be input to the model 98 in the same manner as in the above embodiment.

[0155] As described above, in the first modified example, the acquisition unit 92A randomly selects one generation method 129 from a plurality of generation methods 129 and generates the medical image 102 according to the selected generation method 129, thereby acquiring the medical image 102. The medical image 102 acquired by the acquisition unit 92A is then trained in the model 98 in the same manner as in the above embodiment. This makes it possible to suppress bias in learning of the model 98 compared to a case where the model 98 is always trained on medical images 102 generated by only one generation method 129. Furthermore, since it is not necessary to store the medical image 102 in advance in a memory such as the NVM 96, it is possible to suppress memory capacity shortages.

[0156] The multiple generation methods 129 include a first generation method 129A, a second generation method 129B, and a third generation method 129C. The first generation method 129A is a method that includes generating a circular image 102A, the second generation method 129B is a method that includes generating a rotated image 102C, and the third generation method 129C is a method that includes generating a scaled image 102B. The first generation method 129A, the second generation method 129B, and the third generation method 129C are randomly selected by the acquisition unit 92A, and the selected generation method 129 randomly generates the circular image 102A, the scaled image 102B, or the rotated image 102C. Therefore, the model 98 can randomly learn the circular image 102A, the scaled image 102B, and the rotated image 102C. As a result, it is possible to suppress bias in learning of the model 98 compared to when the model 98 is always trained on medical images 102 generated by only one generation method 129. Furthermore, since it is not necessary to store the medical images 102 in advance in a memory such as the NVM 96, it is possible to suppress the occurrence of memory capacity shortages.

[0157] 6 , the scale of the circular image 102A is adjusted based on the scale of the radial ultrasound image 24. For example, the circular image 102A is reduced so that the scale of the circular image 102A matches the scale of the radial ultrasound image 24. Therefore, the accuracy of identifying a lesion from the radial ultrasound image 24 according to the trained model 78 can be improved compared to when the scale of the circular image 102A remains the same as the scales of the convex ultrasound images 104A and 104B.

[0158] 6 , in the third generation method 129C, the scale of the rotated image 102C is adjusted based on the scale of the radial ultrasound image 24. For example, the rotated image 102C is reduced so that the scale of the rotated image 102C matches the scale of the radial ultrasound image 24. Therefore, compared to when the scale of the rotated image 102C remains the same as the scale of the convex ultrasound image 104D, the accuracy of identifying a lesion from the radial ultrasound image 24 according to the trained model 78 can be improved.

[0159] In this first modified example, the first generation method 129A, the second generation method 129B, and the third generation method 129C are illustrated. However, the multiple generation methods 129 may include generation methods 129 other than the first generation method 129A, the second generation method 129B, and the third generation method 129C. Examples of generation methods 129 other than the first generation method 129A, the second generation method 129B, and the third generation method 129C include a method of generating an image corresponding to a partial region of the convex ultrasound image 104 (hereinafter referred to as a "partial image") as the medical image 102. An example of an image corresponding to a partial region is a segmented image including the lesion region 110 among multiple segmented images obtained by segmenting the convex ultrasound image 104. Another example of the generation method 129 is a method of generating an image obtained by rotating the circular image 102A (hereinafter referred to as a "rotated circular image").

[0160] The partial image and / or the rotated circular image may be stored in advance as a medical image 102 in the NVM 96 shown in Fig. 5. In this case, the partial image and / or the rotated circular image is acquired from the NVM 96 by the acquisition unit 92A, similar to the other medical images 102, and is used for training the model 98.

[0161] [Other Modifications] In the above embodiment, an example was given in which the radial ultrasonic image 24 and the detection frame 128A generated by the processing device 18 are displayed on the screen 26 of the display device 14. However, the radial ultrasonic image 24 with the detection frame 128A added may be transmitted to various devices such as a server, a PC, and / or a tablet terminal and stored in the memory of the various devices. Furthermore, the radial ultrasonic image 24 with the detection frame 128A added may be recorded in a report. Furthermore, the position identification information 128 may also be stored in the memory of various devices or recorded in a report. It is preferable that the radial ultrasonic image 24, the detection frame 128A, and / or the position identification information 128 be stored in the memory or recorded in a report for each subject 22.

[0162] In the above embodiment, an example has been given in which the convex ultrasound image 104 is reduced so that the scale of the convex ultrasound image 104 matches the scale of the radial ultrasound image 24. However, if the scale of the radial ultrasound image 24 is larger than the scale of the convex ultrasound image 104, the convex ultrasound image 104 may be enlarged so that the scale of the convex ultrasound image 104 matches the scale of the radial ultrasound image 24.

[0163] In the above embodiment, an example has been described in which the lesion detection process is performed by the processing device 18 and the learning execution process is performed by the learning device 80, but the technology of the present disclosure is not limited to this. The lesion detection process may be performed by the processing device 18 and at least one device provided external to the processing device 18, or may be performed only by at least one device provided external to the processing device 18 (for example, an auxiliary processing device connected to the processing device 18 and used to expand the functions of the processing device 18). Furthermore, the learning execution process may be performed by the learning device 80 and at least one device provided external to the learning device 80, or may be performed only by at least one device provided external to the learning device 80.

[0164] An example of the at least one device provided outside the processing device 18 and the at least one device provided outside the learning device 80 is a server. The server may be realized by cloud computing. Cloud computing is merely an example, and network computing such as fog computing, edge computing, or grid computing may also be used. Furthermore, the server cited as the at least one device provided outside the processing device 18 and the at least one device provided outside the learning device 80 is merely an example, and instead of a server, it may be at least one PC and / or at least one mainframe, or at least one server, at least one PC, and / or at least one mainframe, etc.

[0165] In the above embodiment, an example has been given in which the radial ultrasound image 24 with the detection frame 128A superimposed thereon is displayed on the screen 26 of the display device 14, but this is merely an example. For example, the radial ultrasound image 24 with the detection frame 128A superimposed thereon and the radial ultrasound image 24 without the detection frame 128A superimposed thereon (i.e., the radial ultrasound image 24 in which the result of detecting the lesion area 126 is not visualized) may be displayed on separate screens.

[0166] In the above embodiment, the detection frame 128A is displayed superimposed on the radial ultrasound image 24 so that the doctor 20 can visually recognize the presence or absence of a lesion and its location. However, the presence or absence of a lesion and its location may also be notified using a notification method other than the detection frame 128A (for example, a text image or audio information).

[0167] In the above embodiment, the doctor 20 is made aware of the presence or absence of a lesion and the location of the lesion, but the doctor 20 may also be made aware of the type of lesion and / or the progression of the lesion, etc. In this case, the medical image 102 may be used as training data for training the model 98, with the annotation 106 including information that can identify the type of lesion and / or the progression of the lesion, etc.

[0168] In the above embodiment, an example in which the learning execution program 100 is stored in the NVM 96 has been described, but the technology of the present disclosure is not limited to this. For example, the learning execution program 100 may be stored in a portable storage medium such as an SSD or USB memory. The storage medium is a non-transitory computer-readable storage medium. The learning execution program 100 stored in the storage medium is installed in the computer 82. The processor 92 executes the learning execution process in accordance with the learning execution program 100.

[0169] In the above embodiment, an example in which the lesion detection program 76 is stored in the NVM 66 has been described, but the technology of the present disclosure is not limited to this. For example, the lesion detection program 76 may be stored in a portable storage medium such as an SSD or USB memory. The storage medium is a non-transitory computer-readable storage medium. The lesion detection program 76 stored in the storage medium is installed in the computer 54. The processor 62 executes the lesion detection process in accordance with the lesion detection program 76.

[0170] In the above embodiment, the computers 54 and 82 are exemplified, but the technology of the present disclosure is not limited to this, and devices including ASICs, FPGAs, and / or PLDs may be applied instead of the computers 54 and / or 82. Furthermore, a combination of hardware and software configurations may be used instead of the computers 54 and / or 82.

[0171] The following various processors can be used as hardware resources for executing the various processes described in the above embodiments (i.e., the learning execution process and the lesion detection process). Examples of processors include general-purpose processors that function as hardware resources for executing various processes by executing software, i.e., programs. Examples of processors also include dedicated electronic circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute various processes.

[0172] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a processor and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0173] As an example of a system configured with a single processor, first, one processor is configured by combining one or more processors with software, and this processor functions as a hardware resource that executes various processes. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as typified by SoC. In this way, various processes are realized using one or more of the above-mentioned various processors as hardware resources.

[0174] Furthermore, the hardware structure of these various processors can be, more specifically, electronic circuits that combine circuit elements such as semiconductor devices. The various processes described above are merely examples. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be rearranged, without departing from the spirit of the invention.

[0175] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0176] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0177] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A learning device comprising a first processor, which acquires a plurality of medical images annotated to identify lesions, and trains a model using the acquired medical images, wherein the medical images are generated based on at least one convex ultrasound image and are images that mimic at least a portion of a radial ultrasound image.

2. The learning device according to claim 1, wherein the plurality of medical images includes a circular image generated by combining a plurality of the convex ultrasound images.

3. The learning device according to claim 2, wherein the scale of the circular image is adjusted based on the scale of the radial ultrasound image.

4. The learning device according to claim 2, wherein the circular image is stored in advance in a first memory, and the first processor acquires the circular image from the first memory and trains the model on the acquired circular image.

5. The learning device according to claim 1, wherein the plurality of medical images includes a rotated image obtained by rotating the convex ultrasound image.

6. The learning device according to claim 5, wherein the scale of the rotated image is adjusted based on the scale of the radial ultrasound image.

7. The learning device according to claim 5, wherein the rotated image is stored in advance in a second memory, and the first processor acquires the rotated image from the second memory and trains the model on the acquired rotated image.

8. The learning device according to claim 1, wherein the plurality of medical images include a scale-adjusted image obtained by adjusting the scale of the convex ultrasound image based on the scale of the radial ultrasound image.

9. The learning device according to claim 8, wherein the scaled image is stored in advance in a third memory, and the first processor acquires the scaled image from the third memory, and causes the model to learn the acquired scaled image.

10. The learning device described in claim 1, wherein the first processor randomly selects one generation method from a plurality of generation methods for generating the medical image based on the at least one convex ultrasound image, obtains the medical image by generating it according to the selected generation method, and trains the model using the obtained medical image.

11. The learning device described in claim 10, wherein the plurality of generation methods include a first generation method, a second generation method, and a third generation method, wherein the first generation method includes generating a circular image as the medical image by synthesizing a plurality of the convex ultrasound images, the second generation method includes generating a rotated image by rotating the convex ultrasound image as the medical image, and the third generation method includes generating a scale-adjusted image as the medical image by adjusting the scale of the convex ultrasound image based on the scale of the radial ultrasound image.

12. The learning device according to claim 11, wherein the first generation method includes adjusting the scale of the circular image based on the scale of the radial ultrasound image.

13. The learning device according to claim 11, wherein the second generation method includes adjusting the scale of the rotated image based on the scale of the radial ultrasound image.

14. The learning device according to claim 1, wherein the first processor acquires at least one first ultrasound image obtained by a first radial ultrasound endoscope, and trains the model on the acquired first ultrasound image.

15. The learning device according to claim 1, wherein the first processor acquires a virtual ultrasound image that is generated based on volume data representing a subject and that mimics at least a portion of the radial ultrasound image, and causes the model to learn the acquired virtual ultrasound image.

16. A trained model obtained by the learning device according to claim 1 training the model using the plurality of medical images.

17. A trained model having a data structure used in the process of identifying a lesion from a second ultrasound image obtained by a second radial ultrasound endoscope, wherein the data structure is obtained by training the model with a plurality of medical images annotated to identify the lesion, and the medical images are generated based on at least one convex ultrasound image and are images that mimic at least a portion of the radial ultrasound image.

18. A medical diagnostic device comprising: the trained model according to claim 17; and a second processor, wherein the second processor acquires a third ultrasound image obtained by a third radial type ultrasound endoscope; and detects a location corresponding to the lesion from the acquired third ultrasound image in accordance with the trained model.

19. An ultrasound endoscope device comprising: the trained model according to claim 17; a fourth radial type ultrasound endoscope; and a third processor, wherein the third processor acquires a fourth ultrasound image obtained by the fourth radial type ultrasound endoscope; and detects a location corresponding to the lesion from the acquired fourth ultrasound image in accordance with the trained model.

20. A learning method comprising: acquiring a plurality of medical images annotated to identify lesions; and training a model using the acquired plurality of medical images, wherein the medical images are generated based on at least one convex ultrasound image and are images that mimic at least a portion of a radial ultrasound image.

21. A program for causing a computer to execute a process, the process including: acquiring a plurality of medical images annotated to identify a lesion; and training a model using the acquired plurality of medical images, wherein the medical images are generated based on at least one convex ultrasound image and are images that mimic at least a portion of a radial ultrasound image.

Images