Image generation method for machine learning, machine learning method, endoscope processor, image generation program for machine learning, and nonvolatile storage medium storing image generation program for machine learning

Abstract

In this machine learning image generation method, a first ultrasonic image captured by a first type ultrasonic imaging device is processed to change at least a shape of a vibrator region of the first ultrasonic image and a position of the vibrator region, and a simulation image, simulating a second ultrasonic image captured by a second type ultrasonic imaging device of which a vibrator array direction is different from that of the first type, is generated as a machine learning image.

Description

Machine learning image generation method, machine learning method, endoscope processor, machine learning image generation program, and non-volatile storage medium storing machine learning image generation program

[0001] The present invention relates to a machine learning image generation method, a machine learning method, an endoscope processor, a machine learning image generation program, and a non-volatile storage medium storing a machine learning image generation program, which are suitable for constructing an image recognition processing model for ultrasound images.

[0002] Ultrasound endoscopes include radial endoscopes, which are primarily used for screening purposes, and convex endoscopes, which can simultaneously observe and collect tissue samples using a biopsy needle for patients suspected of having a disease. Because the direction of ultrasound transmission and reception differs depending on the endoscope type, the appearance of the resulting images (ultrasound images), including the shape and position of the ultrasound transducer shadow and the displayed ultrasound image range, differs. For example, radial endoscopes are suitable for imaging large areas, while convex endoscopes are often used when a biopsy target is identified, as it is easier to determine the relative position of the biopsy needle than radial endoscopes. For this reason, images taken with convex endoscopes (hereinafter referred to as convex images) often contain lesions, while images taken with radial endoscopes (hereinafter referred to as radial images) often do not.

[0003] Computer-aided diagnosis (CAD) has also been developed, which provides support information for identifying lesions and other areas of interest in images obtained by an ultrasound endoscope. When constructing an image recognition processing model for such ultrasound endoscopic CAD using deep learning technology, it is desirable to use many ultrasound images as training data, and it is also desirable to use training data that evenly includes ultrasound images showing lesions and ultrasound images not showing lesions. However, due to differences in use depending on the type of ultrasound endoscope, there is a tendency for differences in the ease of obtaining images. As described above, radial images not showing lesions and convex images showing lesions are easy to obtain. Conversely, there is a problem in that radial images showing lesions and convex images not showing lesions are difficult to obtain.

[0004] JP 2024-004597 A (hereinafter referred to as Patent Document 1) discloses a technology for developing an image recognition processing model dedicated to radial images by machine learning an image that mimics a radial image created from a convex image.

[0005] Special Publication No. 2024-004597

[0006] For example, when attempting to develop an image recognition processing model that enables CAD of both radial images and convex images using a single model, a bias in the amount of training data can adversely affect learning to build the image recognition processing model.The present invention aims to provide a machine learning image generation method, a machine learning method, an endoscope processor, a machine learning image generation program, and a non-volatile storage medium storing the machine learning image generation program, which are capable of building a highly accurate image recognition processing model by enabling conversion of images obtained by an ultrasonic endoscope.

[0007] One aspect of the present invention provides a method for generating an image for machine learning by processing a first ultrasound image captured by a first type of ultrasound imaging device, thereby changing at least the shape and position of the transducer region of the first ultrasound image, and generating a simulated image as an image for machine learning that simulates a second ultrasound image captured by a second type of ultrasound imaging device having a transducer arrangement direction different from that of the first type.

[0008] In addition, a machine learning method of one embodiment of the present invention uses, as training images, a radial image that is the first ultrasound image, a simulated convex image that is the simulated image, a convex image that shows at least one of a biological part and a lesion, and a simulated radial image that is the simulated image.

[0009] Another aspect of the present invention is an endoscope processor that can access artificial intelligence learned by the machine learning method, receives an image transmitted from an ultrasonic endoscope, inputs the received image into the artificial intelligence, receives an output result from the artificial intelligence, and outputs the received output result to a monitor.

[0010] One aspect of the machine learning image generation program of the present invention causes a computer to execute a procedure for processing a first ultrasound image captured by a first type of ultrasound imaging device, thereby changing at least the shape and position of the transducer region of the first ultrasound image, and generating a simulated image as a machine learning image that simulates a second ultrasound image captured by a second type of ultrasound imaging device having a transducer arrangement direction different from that of the first type.

[0011] A non-volatile storage medium storing an image generation program for machine learning according to one embodiment of the present invention stores the image generation program for machine learning, which causes a computer to execute a procedure for processing a first ultrasound image captured by a first type of ultrasound imaging device, thereby changing at least the shape and position of the transducer region of the first ultrasound image, and generating, as an image for machine learning, a simulated image that simulates a second ultrasound image captured by a second type of ultrasound imaging device having a transducer arrangement direction different from that of the first type.

[0012] According to the present invention, it is possible to convert an image obtained by an ultrasonic endoscope, thereby achieving the effect of constructing a highly accurate image recognition processing model.

[0013] 1. FIG. 1 is a block diagram showing an image processing device that realizes a machine learning image generation method according to an embodiment of the present invention. FIG. 2 is an explanatory diagram illustrating the appearances of a convex type ultrasonic endoscope 31 and a radial type ultrasonic endoscope 32, and a convex image from the convex type ultrasonic endoscope 31 and a radial image from the radial type ultrasonic endoscope 32. FIG. 3 is an explanatory diagram showing an example of a convex image and a radial image displayed on a display screen 22. FIG. 4 is an explanatory diagram illustrating an example of processing a convex image into a simulated radial image. FIG. 5 is an explanatory diagram illustrating another example of processing a convex image into a simulated radial image. FIG. 6 is an explanatory diagram illustrating another example of processing a convex image into a simulated radial image. FIG. 7 is an explanatory diagram illustrating an example of processing a radial image into a simulated convex image. FIG. 8 is an explanatory diagram illustrating an example of processing a radial image into a simulated convex image. FIG. 9 is an explanatory diagram illustrating an example of processing a radial image into a simulated convex image. FIG. 10 is a block diagram showing an example of a specific configuration of the learning circuit 19 in FIG. 1. FIG. 11 is a flowchart illustrating machine learning for generating an image recognition processing model in an embodiment. Fig. 13 is a flowchart for explaining the generation of images for machine learning in S2 in Fig. 12. Fig. 14 is a block diagram showing an endoscope processor that uses an image recognition processing model.

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0015] 1 is a block diagram illustrating an image processing device that realizes a machine learning image generation method according to one embodiment of the present invention. This embodiment processes convex images obtained by a convex-type ultrasonic endoscope to generate simulated radial images similar to radial images obtained by a radial-type ultrasonic endoscope, or processes radial images obtained by a radial-type ultrasonic endoscope to generate simulated convex images similar to convex images obtained by a convex-type ultrasonic endoscope. This not only increases the number of images but also suppresses bias in the distribution of lesions and the like in ultrasound images, enabling the generation of machine learning images (training data) that enable the construction of a more accurate image recognition processing model. In this embodiment, simulated radial images and simulated convex images (hereinafter, these may be simply referred to as simulated images) are generated by changing the shape and position of transducer regions included in ultrasound images obtained from the ultrasonic endoscope.

[0016] In this embodiment, an ultrasonic endoscope is used as an example of an apparatus for acquiring ultrasonic images, but various ultrasonic imaging apparatuses other than the ultrasonic endoscope may be used. Also, although a convex type and a radial type are used as examples of ultrasonic endoscope types, the present invention is not limited to these and can be applied to ultrasonic images acquired by ultrasonic imaging apparatuses such as sector type and linear type.

[0017] 1, an ultrasound image obtained by a convex type ultrasound endoscope (hereinafter referred to as a convex image) or a radial type ultrasound endoscope (hereinafter referred to as a radial image) is input to an image processing device 10. Ultrasound endoscopes such as convex type ultrasound endoscopes and radial type ultrasound endoscopes are inserted into the body, transmit ultrasound beams toward a target area to be observed, and receive reflected waves from the boundary of acoustic impedance of the target area to obtain echo signals. An ultrasound image is generated based on these echo signals.

[0018] FIG. 2 shows the appearance of a convex type ultrasonic endoscope 31 and a radial type ultrasonic endoscope 32 as examples of ultrasonic endoscopes, and is an explanatory diagram for explaining the convex image obtained by the convex type ultrasonic endoscope 31 and the radial image obtained by the radial type ultrasonic endoscope 32.

[0019] The convex type ultrasonic endoscope 31 includes a thin and long probe 33 to be inserted into the body, etc., and a transducer 34 that is provided at the tip of the probe 33 (the tip of the endoscope) and transmits and receives an ultrasonic beam. The transducer 34 is configured as a transducer array in which a plurality of elements are arranged. The tip of the probe 33 is provided with an opening 35a of a forceps channel provided within the probe 33, and a biopsy needle 35 protrudes from this opening 35a. The radial type ultrasonic endoscope 32 includes a thin and long probe 36 to be inserted into the body, etc., and a transducer 37 that is provided at the tip of the probe 36 (the tip of the endoscope) and transmits and receives an ultrasonic beam. The transducer 37 is configured as a transducer array in which a plurality of ultrasonic elements are arranged.

[0020] In the convex type ultrasonic endoscope 31, the ultrasonic elements of the transducer 34 are arranged in the shape of a convex curved surface, and generate ultrasonic waves by scanning an object in an arc shape. That is, the transducer 34 of the convex type ultrasonic endoscope 31 scans in a sector shape within a plane parallel to the probe axis, as shown by the convex scanning plane 34a.

[0021] On the other hand, in the radial type ultrasonic endoscope 32, the ultrasonic elements of the transducer 37 are arranged circumferentially around the axis of the probe 36, and generate ultrasonic waves by scanning an object circumferentially. That is, the transducer 37 of the radial type ultrasonic endoscope 32 scans in an arc shape within a plane perpendicular to the probe axis, as shown in the radial scanning plane 37a.

[0022] The transducers 34 and 37 perform scanning by, for example, electronically driving each ultrasonic element. In this embodiment, the purpose is to increase the number of images used as training data while suppressing uneven distribution of lesions and the like in the ultrasound images. The images used as training data are not limited to convex or radial images. For example, ultrasound images obtained by a transducer that mechanically changes the direction of transmission and reception of ultrasound using a single ultrasonic element may be used as training data. Alternatively, ultrasound images obtained by a sector transducer or a linear transducer in which transducers are linearly arranged may also be used. In contrast to radial ultrasound endoscopes, convex, sector, and linear ultrasound endoscopes are also referred to as non-radial ultrasound endoscopes.

[0023] The bottom panel of Figure 2 shows a convex image Pc and a radial image Pr displayed on the display screen. The convex image Pc obtained by the convex ultrasonic endoscope 31 has a fan-shaped image portion at the top of the screen. This image portion is a transducer image P34 representing the transducer 34 of the convex ultrasonic endoscope 31. Furthermore, multiple semicircular patterns spreading from the transducer image P34 in the convex image Pc indicate that scanning is performed in a semicircular (fan-shaped) manner centered on the transducer 34. To obtain a good ultrasound image, an ultrasound balloon may be provided to bring the transducer image P34 into close contact with an organ, and a balloon image P34a of the ultrasound balloon is located around the transducer image P34. While Figure 2 depicts the fan as being 180 degrees open, the present invention is not limited to this and may be any angle smaller than 360 degrees.

[0024] The radial image Pr obtained by the radial type ultrasonic endoscope 32 has a circular image portion in the center of the screen. This image portion is a transducer image P37 that shows the transducer 37 of the radial type ultrasonic endoscope 32. Furthermore, multiple circular patterns spreading from the transducer image P37 in the radial image Pr indicate that scanning is performed concentrically around the transducer 37. Furthermore, a balloon image P37a of an ultrasound balloon is present around the transducer image P37. Note that the transducer images P34, P37 and the balloon images P34a, P37a are generally displayed in a uniform black color with a predetermined pixel value.

[0025] In FIG. 1 , the image processing device 10 includes a control unit 11. The control unit 11 may be configured with a processor using a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or the like. The control unit 11 may operate according to a program stored in a memory (not shown) to control each unit, or may implement some or all of its functions using a hardware electronic circuit. The control unit 11 comprehensively controls each unit of the image processing device 10 based on user operations on an operation input unit 16. The operation input unit 16 is configured with various interfaces such as a touch panel and a keyboard, and outputs operation signals based on user operations to the control unit 11.

[0026] The image acquisition unit 12 of the image processing device 10 receives convex images and radial images under the control of the control unit 11. The image processing device 10 may include a convex image memory 13 that stores convex images and a radial image memory 14 that stores radial images, or may be configured to input the images acquired by the image acquisition unit 12 to the image processing unit 15 without going through a memory. The image acquisition unit 12 provides the convex images to the convex image memory 13 and the radial images to the radial image memory 14 under the control of the control unit 11.

[0027] Although Fig. 1 shows an example in which two memories are used, different memory areas of a single memory may be used. Also, Fig. 1 shows an example in which both a convex image and a radial image are handled simultaneously, and if only one of them is handled, a single memory or a single memory area may be provided.

[0028] The ultrasound images stored in the convex image memory 13 and the radial image memory 14 are supplied to the image processing unit 15. The image processing unit 15 is controlled by the control unit 11 to perform image processing on at least one of the convex image and the radial image. The image processing unit 15 may have a working memory for image processing. The control unit 11 sets the image processing process for generating a simulated convex image or a simulated radial image in the image processing unit 15. The image processing unit 15 generates a simulated image by image processing on the ultrasound image. The image processing unit 15 may include a configuration for automatically processing an input image into an arbitrary simulated image, or may be a component including a program for manually processing an image, such as Adobe (registered trademark). The image processing method of the image processing unit 15 will be described later.

[0029] The processed simulated image output from the image processing unit 15 is supplied to the recording device 25. The convex image and radial image recorded in the convex image memory 13 and radial image memory 14 are also supplied to the recording device 25.

[0030] A recording device 25 may be connectable to the image processing device 10. The recording device 25 is configured with a predetermined recording medium (not shown) and records images before processing by the image processing unit 15 and simulated images generated by processing by the image processing unit 15. The images recorded in the recording device 25 are used as training data for learning the image recognition processing model. Examples of the recording device 25 include storage media such as an SD card and a USB memory.

[0031] The simulated images generated by the image processing device 10 are input to the learning device 19 as training data. In this case, the simulated images may be input to the learning device 19 via the recording device 25, or may be input directly from the image processing device 10 to the learning device 19 by a known method such as a cable or wireless communication. The image processing device 10 and the learning device 19 may be integrated. In this case, for example, a learning circuit serving as the learning device 19 is disposed within the image processing device 10, and the learning circuit is controlled by the control unit 11 to perform machine learning using images recorded in the recording device 25 as training data, thereby generating an image recognition processing model. The learning circuit may also independently perform machine learning using images recorded in the recording device 25 to construct an image recognition processing model, without being controlled by the control unit 11.

[0032] The display device 21 is a display device such as an LCD (liquid crystal display device), and is capable of displaying various types of information such as a menu display according to display control by the control unit 11. The display device 21 is also capable of displaying on a display screen 22 an image processed by the image processing unit 15 and an image before processing by the image processing unit 15.

[0033] FIG. 3 is an explanatory diagram showing an example of a convex image and a radial image.

[0034] The observation region of a convex ultrasonic endoscope is fan-shaped, and the convex image Pc1 shown on the left side of Fig. 3 has a transducer image IcV1 at the top of the screen. In the example of Fig. 3, the convex image Pc1 also shows a tissue image IcT1 showing tissue (biological part) such as blood vessels and organs, and a lesion image IcL1 showing a lesion. The bottom of the screen is a blind region PcB that does not include any images.

[0035] Examples of lesions include cancer, tumors, masses, pancreatitis, inflammation, liposarcoma, lymphoma, and fibrosis, and examples of biological sites include the upper gastrointestinal tract (stomach, esophagus, duodenum), large intestine, lung, pancreas, liver, gallbladder, bile duct, pancreatic duct, prostate, blood vessels, lymph nodes, muscle layer, fat layer, and bone.

[0036] On the other hand, the observation region of a radial ultrasound endoscope is concentric, and the radial image Pr1 shown on the right side of Fig. 3 has a transducer image IrV1 at the center of the screen. In the example of Fig. 3, tissue images IrT1 to IrT3 representing tissues are also captured in the radial image Pr1. As described above, lesions are often not captured in the radial image Pr1. Conventionally, due to the uneven distribution of lesions, etc., an image recognition processing model generated by learning using ultrasound images such as those shown in Fig. 3 as training data may erroneously learn to identify lesions based on the shape and position of the transducer shadow.

[0037] Therefore, in this embodiment, one of the convex image and the radial image is processed to generate a simulated image that simulates the other image. By using the generated simulated image as training data, bias in the lesion distribution is eliminated, and erroneous learning is prevented.

[0038] Hereinafter, a specific type of endoscope among the convex type, radial type, sector type, and linear type ultrasonic endoscopes will be referred to as a "first type" ultrasonic endoscope, and the other type will be referred to as a "second type" ultrasonic endoscope, and ultrasonic images obtained by the first and second ultrasonic endoscopes will be referred to as a "first ultrasonic image" and a "second ultrasonic image," respectively. That is, this embodiment applies image processing to a first ultrasonic image obtained by the first type ultrasonic endoscope, thereby generating a simulated image of one of the first and second ultrasonic images from the other ultrasonic image (hereinafter, sometimes simply referred to as a "simulated image of the other"). Furthermore, this embodiment not only describes a processing method for generating a simulated image of the other ultrasonic image from one ultrasonic image, but also a method for generating a simulated image of one ultrasonic image from one ultrasonic image (hereinafter, sometimes simply referred to as a "simulated image of the one").

[0039] In this embodiment, when generating the other simulated image, image processing is performed so that the shape and position of the shadow of the transducer in the first ultrasound image resembles the shape and position of the shadow of the tip in the second ultrasound image. For example, in this embodiment, simulated images such as a simulated radial image or a simulated convex image are generated by changing the shape and position of the transducer region included in the radial image or the convex image.

[0040] (Generating a simulated radial image from a convex image) (Example C-R1) Figure 4 is an explanatory diagram for explaining an example of processing a convex image into a simulated radial image. The left side of Figure 4 shows the convex image stored in the convex image memory 13, the center of Figure 4 shows the image processing process, and the right side of Figure 4 shows the simulated radial image.

[0041] In this embodiment, the control unit 11 controls the image processing unit 15 to process the shape and position of the shadow of the transducer of the convex ultrasonic endoscope on the convex image Pc1 so that it resembles the shape and position of the shadow of the tip of the transducer on the radial image. That is, the control unit 11 controls the image processing unit 15 to process the shape and position of the transducer image IcV1 on the convex image so that it becomes circular in the center of the screen, similar to the radial image.

[0042] 4 shows this processing, in which the image processing unit 15 creates an inverted convex image PcR1 by vertically inverting the convex image Pc1 read from the convex image memory 13, and generates a simulated convex image PrS1 by combining the original convex image Pc1 and the inverted convex image PcR1. The inverted convex image PcR1 includes a transducer image IcVR1, a tissue image IcTR1, a lesion image IcLR1, and a blind region PcBR, which are obtained by vertically inverting the transducer image IcV1, tissue image IcT1, lesion image IcL1, and blind region PcB of the convex image Pc1, respectively.

[0043] As shown in the right column of FIG. 4 , the image processing unit 15 synthesizes (connects) the convex image Pc1 and the inverted convex image PcR1 by synthesizing (connecting) the fan-shaped transducer image IcV1 of the original convex image Pc1 with the fan-shaped transducer image IcVR1 included in the inverted convex image PcR1 to obtain a circular simulated transducer image IrVS1 (IcV1 + IcVR1). The image processing unit 15 clips the top and bottom edges of the synthesized image to adjust it to the size of the display screen, thereby obtaining a simulated radial image PrS1 in which the circular simulated transducer image IrVS1 is positioned at the center of the screen. Like the radial image, the simulated radial image PrS1 includes a circular simulated transducer image IcVS1 at the center of the screen and also includes a lesion image IcL1 and a simulated lesion image IcLR1 of the lesion area.

[0044] As described above, convex images and radial images are examples of ultrasound images. Generalizing the images, (Example C-R1) involves processing a first ultrasound image captured by a first type of ultrasound imaging device to change at least the shape and position of the transducer region of the first ultrasound image, thereby generating a simulated image that simulates a second ultrasound image captured by a second type of ultrasound imaging device whose transducer arrangement direction differs from that of the first type. One of the first and second ultrasound imaging devices is a radial ultrasound endoscope in which transducers are arranged around the entire axis, while the other is a non-radial ultrasound endoscope in which transducers are not arranged around the entire axis. The non-radial ultrasound endoscope is, for example, a convex ultrasound endoscope or a sector ultrasound endoscope.

[0045] The image processing unit 15 provides the simulated radial image PrS1 to the recording device 25 for recording. In this way, the image processing unit 15 generates a simulated radial image that imitates the difficult-to-obtain image of a lesion captured by a radial ultrasonic endoscope, by combining the original convex image and an inverted convex image of the lesion captured by a convex ultrasonic endoscope so that the shape and position of the transducer shadow are similar. In this way, an ultrasound image that eliminates bias in the distribution of lesions, etc., can be obtained.

[0046] (Example C-R2) Figure 5 is an explanatory diagram for explaining another example of processing from a convex image to a simulated radial image. The left side of Figure 5 shows the convex image stored in the convex image memory 13, the center of Figure 5 shows the image processing process, and the right side of Figure 5 shows the simulated radial image. In Figure 5, the same components as in Figure 4 are assigned the same reference numerals.

[0047] The example of FIG. 5 differs from the example of FIG. 4 only in that the image processing unit 15 is controlled by the control unit 11 to generate a simulated radial image PrSs1 by scrolling the display area of ​​the simulated radial image PrS1 of FIG. 4. In some cases, the screen can be scrolled during ultrasound image observation, and this example corresponds to this. While FIG. 5 illustrates an example of a simulated radial image scrolled vertically, it may also be scrolled horizontally or diagonally. In this way, in example C-R2, an ultrasound image that eliminates uneven distribution of lesions, etc., can be obtained.

[0048] (Example C-R3) Figure 6 is an explanatory diagram for explaining another example of processing from a convex image to a simulated radial image. The left side of Figure 6 shows two convex images stored in the convex image memory 13, the center of Figure 6 shows the image processing process, and the right side of Figure 6 shows the simulated radial image. In Figure 6, the same components as in Figure 4 are assigned the same reference numerals.

[0049] In the example of Fig. 6, the image processing unit 15, under the control of the control unit 11, creates a simulated radial image PrS2 by image processing using two convex images Pc2 and Pc3. Fig. 6 illustrates this processing. Under the control of the control unit 11, the image processing unit 15 reads the convex images Pc2 and Pc3 from the convex image memory 13, flips the convex image Pc3 upside down to create an inverted convex image PcR3 (center column of Fig. 6), and generates a simulated convex image PrS2 (right column of Fig. 6) by combining the convex image Pc2 and the inverted convex image PcR3. Note that the convex image Pc2 includes a transducer image IcV2, a lesion image IcL2, tissue images IcT2a and IcT2b, and a blind region PcB, and the convex image Pc3 includes a transducer image IcV3, a lesion image IcL3, a tissue image IcT3, and a blind region PcB. The inverted convex image PcR3 includes a transducer image IcVR3, a lesion image IcLR3, a tissue image IcTR3, and a blind region PcBR.

[0050] That is, the image processing unit 15 combines (links) the convex image Pc2 and the inverted convex image PcR3 to obtain a circular simulated oscillator image IrVS2 (IcV2 + IcVR3) by combining the oscillator image IcV2 in the convex image Pc2 with the oscillator image IcVR3 included in the inverted convex image PcR3. The image processing unit 15 clips the top and bottom edges of the combined image to obtain a simulated radial image PrS2 adjusted to the size of the display screen. Like the radial image, the simulated radial image PrS2 has a circular simulated oscillator image IcVS2 in the center of the screen and also includes lesion images IcL2 and IcLR3 of the lesion area.

[0051] In this way, (Example C-R3) also processes a first ultrasonic image captured by a first type ultrasonic imaging device to change at least the shape and position of the transducer region of the first ultrasonic image, thereby generating a simulated image that simulates a second ultrasonic image captured by a second type ultrasonic imaging device having a transducer array direction different from that of the first type. Note that (Example C-R3) uses a convex type ultrasonic endoscope as the first type ultrasonic imaging device and a radial type ultrasonic endoscope as the second type ultrasonic imaging device. Multiple first ultrasonic images are prepared, and the fan-shaped transducer region at the tip of the endoscope located at the edge of the first ultrasonic image becomes a circle, and the multiple first ultrasonic images are stitched together to form the circle at the center. In this case, the multiple first ultrasonic images include a first convex image captured by the convex type ultrasonic endoscope and a second convex image captured by the convex type ultrasonic endoscope that is different from the first convex image.

[0052] The image processing unit 15 provides the simulated radial image PrS1 to the recording device 25 for recording. In this way, also in the example C-R3, a simulated radial image for eliminating uneven distribution of lesions and the like can be obtained.

[0053] (Example C-R4) Figure 7 is an explanatory diagram for explaining another example of processing from a convex image to a simulated radial image. The left side of Figure 7 shows the convex image stored in the convex image memory 13, the center of Figure 7 shows the image processing process, and the right side of Figure 7 shows the simulated radial image. In Figure 7, the same components as in Figure 4 are assigned the same reference numerals.

[0054] The example in Figure 7 differs from the example in Figure 4 only in that the image processing unit 15, under the control of the control unit 11, performs processing to add a balloon image IB representing an ultrasound balloon to the simulated radial image PrS1 of Figure 4. In this way, even in example C-R4, an ultrasound image that eliminates bias in the distribution of lesions, etc. is obtained. Note that while Figure 7 shows an example in which processing is performed to add the balloon image IB to the simulated radial image PrS1, conversely, processing may be performed to delete the balloon image when an ultrasound image is displayed in the simulated radial image PrS1.

[0055] (Generating a simulated convex image from a radial image) (Example R-C1) Fig. 8 is an explanatory diagram for explaining an example of processing a radial image into a simulated convex image. The left side of Fig. 8 shows the radial image stored in the radial image memory 14, the center of Fig. 8 shows the image processing process, and the right side of Fig. 8 shows the simulated convex image.

[0056] In this embodiment, the control unit 11 controls the image processing unit 15 to process the image so that the shape and position of the shadow of the transducer of the probe 33 on the radial image Pr1 resembles the shape and position of the shadow of the tip of the probe 33 on the convex image. That is, the control unit 11 controls the image processing unit 15 to process the shape and position of the transducer image IrV1 on the radial image on the screen so that it has a semicircular shape at the edge of the screen, similar to the convex image.

[0057] FIG. 8 illustrates this processing. The radial image Pr1 includes a transducer image IrV1 and tissue images IrT1a, IrT1b, and IrT1c. The image processing unit 15 detects the transducer image IrV1 included in the radial image Pr1 read from the radial image memory 14. For example, the image processing unit 15 detects the transducer image IrV1 located at the center of the screen based on brightness and circularity. The center column of FIG. 8 shows the detection region IrVD1 of the transducer image IrV1 detected by the image processing unit 15. The image processing unit 15 cuts out a portion of the radial image Pr1 by deleting the portion of the radial image Pr1 above the horizontal line passing through the center of the detection region IrVD1. The image processing unit 15 then creates a black blind region PcBS with a predetermined pixel value at the bottom of the cut-out radial image Pr1 to fit the size of the display screen, thereby generating a simulated convex image PcS1.

[0058] The simulated convex image PcS1 has a semicircular simulated transducer image IrVH1 at the top of the screen, as well as tissue images IrT1b and IrT1c and a blind region PcBS, similar to the convex image.

[0059] In this way, also in (Example R-C1), a first ultrasonic image captured by a first type ultrasonic imaging device is processed to change at least the shape and position of the transducer region of the first ultrasonic image, thereby generating a simulated image that simulates a second ultrasonic image captured by a second type ultrasonic imaging device having a transducer array direction different from that of the first type. Note that in (Example R-C1), the first type ultrasonic imaging device is a radial type ultrasonic endoscope, and the second type ultrasonic imaging device is a convex type ultrasonic endoscope, and the circular transducer region of the tip of the endoscope located in the center of the first ultrasonic image becomes fan-shaped, and processing is performed to partially remove the first ultrasonic image so that this fan-shaped region is located at the edge of the image.

[0060] The image processing unit 15 provides the simulated convex image PcS1 to the recording device 25 for recording. In this way, the image processing unit 15 generates a simulated convex image that imitates an image that is difficult to obtain and does not show lesions, captured with a convex ultrasonic endoscope, by performing image processing on a radial image that does not show lesions and is easily obtained using a radial ultrasonic endoscope, by deleting the upper part of the original radial image and providing a blind area so that the shape and position of the transducer shadow resembles that of the original radial image. In this way, an ultrasound image that eliminates bias in the distribution of lesions, etc., can be obtained.

[0061] In examinations using a convex scope, the position is often adjusted so that the lesion is displayed at the bottom right of the transducer for biopsy purposes. Therefore, when generating a simulated convex image using a radial image showing the lesion, the image is shifted and inverted to adjust the lesion position to the bottom right of the transducer. This makes it possible to generate a simulated convex image that is close to the images that frequently appear in actual examination situations.

[0062] In addition, in radial images, the transducer may be located not in the center of the screen but above or below. In such cases, the smaller area of ​​the transducer is removed so that the transducer fan is positioned at the edge of the image. This makes it possible to generate a simulated convex image using an image that captures a wider ultrasound image area.

[0063] (Example R-C2) Figure 9 is an explanatory diagram for explaining another example of processing from a radial image to a simulated convex image. The left side of Figure 9 shows the radial image stored in the radial image memory 14, the center of Figure 9 shows the image processing process, and the right side of Figure 9 shows the simulated convex image. In Figure 9, the same components as in Figure 8 are assigned the same reference numerals.

[0064] The example of Figure 9 differs from the example of Figure 8 only in that the image processing unit 15, under the control of the control unit 11, generates a simulated convex image PcSF1 in which the effective observation range of the simulated convex image PcS1 of Figure 8 is limited to a fan shape. In a convex ultrasound endoscope, the scanning range of the transducer is relatively narrow, and a convex image with a narrow effective observation range may be obtained depending on the central angle of the fan shape of the scanning range. This example addresses this issue, and the image processing unit 15 provides black mask regions PcM1 and PcM2 with predetermined pixel values ​​so that a fan-shaped effective observation range can be obtained from both ends of the transducer image IrVH1 at the top of the screen. In this way, in example R-C2, an ultrasound image that eliminates bias in the distribution of lesions, etc., can also be obtained.

[0065] (Example R-C3) Figure 10 is an explanatory diagram for explaining another example of processing from a radial image to a simulated convex image. The left side of Figure 10 shows the radial image stored in the radial image memory 14, the center of Figure 10 shows the image processing process, and the right side of Figure 10 shows the simulated convex image. In Figure 10, the same components as in Figure 8 are assigned the same reference numerals.

[0066] The example of FIG. 10 differs from the example of FIG. 8 only in that the image processing unit 15, under the control of the control unit 11, generates a simulated convex image PcSA1 by adding a multiple reflection image IcA1 indicating multiple reflections to the simulated convex image PcS1 of FIG. 8 . In an ultrasound endoscope, multiple reflections (ring artifacts) can occur due to interference between ultrasonic wave packets reflected along the ultrasound propagation path. This example addresses this issue, and the image processing unit 15 creates a semicircular black multiple reflection image IcA1 with a predetermined pixel value around the transducer image IrVH1 at the top of the screen. Note that if a multiple reflection image appears in the simulated convex image, the image processing unit 15 may perform processing to remove the multiple reflection image from the simulated convex image. In this way, in example R-C3, an ultrasound image is obtained that eliminates the bias of the multiple reflection image.

[0067] In this way, various types of image processing are used to appropriately adjust the training data for machine learning. As described above, if there is a bias in the training images, there is a possibility that images with different appearances will be misjudged. For example, if there is a difference between the images used during training and the images actually taken at a hospital for diagnosis, the image recognition processing model may make an erroneous judgment. Therefore, for appropriate adjustment, image processing is performed to equalize the number of various images. For example, it is better to use as training data (training images) an equal number of ultrasound images regarding the presence or absence of a lesion. Furthermore, it is better to create training data so that the number of images is as equal as possible, ranging from images without the influence of multiple reflections to images with a large influence of multiple reflections, as in the example of FIG. 10 .

[0068] (Other Examples) In each of the following examples, a simulated image of one ultrasound image is generated from the other ultrasound image by performing image processing so that the shape and position of the shadow of the transducer in one ultrasound image, either a convex image or a radial image, resembles the shape and position of the shadow of the tip in the other ultrasound image.

[0069] (Example E1: Convex + Inverted Convex) As in the example of Figure 4, a simulated radial image is generated by concatenating a convex image and an inverted convex image and clipping them. In this case, two images of the same lesion or tissue may be displayed as is in the simulated radial image, or one of the two corresponding images may be deleted by filling in the background color, for example. For example, in the example of Figure 4, an example was described in which the simulated radial image PrS1 includes images IcT1 and IcTR1 of the same tissue and lesion images IcL1 and IcLR1 of the same lesion. However, one of the images IcT1 and IcTR1 may be deleted, and one of the lesion images IcL1 and IcLR1 may be deleted.

[0070] (Example E2: Convex + Convex Rotation) A simulated radial image may be generated by combining an original convex image with a rotated convex image obtained by rotating the original convex image by 180 degrees. In this case, two images of the same tissue or two images of the same lesion may be displayed as they are in the generated simulated radial image, or one of the images corresponding to the same lesion or the same tissue may be deleted.

[0071] (Example E3: Convex + Radial Halving) A simulated radial image may be generated by combining a convex image with a halved radial image obtained by dividing the original radial image in half along a horizontal line passing through the center of the transducer image. Even in this case, two images of the same tissue or two lesion images of the same lesion may be displayed as is in the generated simulated radial image, or one of the images corresponding to the same lesion or the same tissue may be deleted. Furthermore, a convex image showing a lesion image may be combined with a halved radial image not showing a lesion image.

[0072] (Example E4: Convex + Another Convex Image) A simulated radial image is generated by concatenating a convex image and another convex image and clipping them, as in the example of Fig. 6. In this case, although Fig. 6 describes an example in which two images of the same lesion or tissue may appear as they are in the simulated radial image, one of the two corresponding images may be deleted.

[0073] (Example E5: Dividing Radial Image in Half) A halved radial image obtained by dividing the radial image in half along a horizontal line passing through the center of the transducer image may be used as the simulated convex image.

[0074] (Example E6: Dividing Radial Image into Four) A simulated convex image may be created by combining two four-division radial images obtained by dividing the radial image into four parts along horizontal and vertical lines passing through the center of the transducer image.

[0075] (Example E7: Convex image + 2 half convex images) A ​​simulated radial image may be generated by combining three images: a convex image, a half convex image obtained by dividing the convex image in half along a vertical line passing through the center of the transducer image, and the half convex image.

[0076] As described above, by performing image processing so that the shape and position of the shadow of the transducer in a first ultrasonic image obtained by a first type of ultrasonic endoscope resembles the shape and position of the shadow of the tip in a second ultrasonic image obtained by a second type of ultrasonic endoscope, it is possible not only to generate a simulated image of one ultrasonic image from another, but also to generate a simulated image of one ultrasonic image from another. Furthermore, it is also possible to generate one or other simulated images from one ultrasonic image by image processing using the ultrasonic images used above (Examples C-R1 to C-R4, Examples R-C1 to C3, Examples E1 to E7) and the ultrasonic images obtained during the processing of the ultrasonic images.

[0077] (Learning) FIG. 11 is a block diagram showing an example of a specific configuration of the learning circuit 19 in FIG.

[0078] The learning circuit 19 in FIG. 11 is composed of a neural network (NN) 40, a neural network control circuit (hereinafter referred to as the NN control circuit) 50, and a learning loss calculation unit 55. Note that all or each of the learning loss calculation unit 55 and the NN control circuit 50 may be composed of one or more processors using a CPU, GPU, FPGA, or the like. These one or more processors may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of the functions using hardware electronic circuits. Furthermore, the neural network 40 may be composed of hardware, or the functions of the neural network 40 may be realized by a program. The NN control circuit 50 is controlled by the control unit 11.

[0079] As described above, the recording device 25 records the convex images and radial images recorded in the convex image memory 13 and the radial image memory 14 as training data (learning images), and also records the simulated convex images and simulated radial images obtained by image processing by the image processing unit 15 as training data (learning images). Furthermore, each image recorded in the recording device 25 is annotated. For example, each image recorded in the recording device 25 is provided with information indicating the location (area) of the lesion, such as information such as a mask image indicating the lesion area and information on the lesion differentiation result, as correct answer information. As described above, many of the simulated radial images generated based on the convex images contain lesion images, while many of the simulated convex images generated based on the radial images do not contain lesion images. As a result, when the radial images and the simulated radial images are combined, the distribution of lesion images is approximately uniform, and when the convex images and the simulated convex images are combined, the distribution of lesion images is approximately uniform. It is preferable that the total number of radial images and simulated radial images is approximately equal to the total number of convex images and simulated convex images.

[0080] For example, it is preferable that the ratio of the number of radial images, pseudo-convex images, convex images, and pseudo-radial images is such that the number of images with the largest number is within twice the number of images with the smallest number, and more preferably, all numbers are equal.

[0081] The neural network 40 is composed of an input layer, an intermediate layer (hidden layer), and an output layer, each of which is made up of multiple nodes indicated by circles. Each node is connected to nodes in the previous and next layers, and each connection is assigned a parameter called a weight coefficient. Learning is a process of updating parameters to minimize the learning loss, which will be described later. For example, a convolutional neural network (CNN) may be used as the neural network 40.

[0082] The NN control circuit 50 is composed of an input control unit 51, an initialization unit 52, an NN application unit 53, and an update unit 54. The input control unit 51 controls the input of training data and teacher data including correct answer information recorded in the recording device 25. The initialization unit 52 initializes the parameters of the neural network 40. The NN application unit 53 applies the training data read from the recording device 25 to the neural network 40, causing the neural network 40 to output a classification output. The update unit 54 updates the parameters of the neural network 40 based on the learning loss.

[0083] The neural network 40 is controlled by the NN control circuit 50 and outputs, as a classification output, a probability value (hereinafter referred to as a score) indicating which classification each input image has a high probability of belonging to. This classification output is provided to a learning loss calculation unit 55. The learning loss calculation unit 55 is provided with correct answer information assigned to each image corresponding to each classification output from the recording device 25, and calculates the error between each classification output and each correct answer information as a learning loss. This learning loss is provided to the NN control circuit 50.

[0084] The update unit 54 of the NN control circuit 50 updates the parameters of the neural network 40 so as to reduce the received learning loss. For example, the update unit 54 may update the parameters in accordance with an existing SGD (Stochastic Gradient Descent) algorithm. The update formula for this SGD is publicly known, and each parameter of the neural network 40 is calculated by substituting the learning loss value into the SGD update formula. Under the control of the NN control circuit 50, the neural network 40 classifies the input image using the updated parameters. Thereafter, similar operations are repeated to perform learning.

[0085] By inputting an unknown ultrasound image into the neural network 40 (image recognition processing model) obtained by such learning, results (classification output) such as differentiation of lesions in the input ultrasound image can be obtained.

[0086] Next, the operation of the embodiment configured as above will be described with reference to Fig. 12 to Fig. 14. Fig. 12 is a flowchart for explaining machine learning for generating an image recognition processing model in the embodiment, and Fig. 13 is a flowchart for explaining the generation of images for machine learning in S2 in Fig. 12. Fig. 14 is a block diagram showing an endoscope processor that uses the image recognition processing model.

[0087] In S1 of Fig. 12, the initialization unit 52 of the NN control circuit 50 initializes the parameters of the neural network 40. However, the initialization unit 52 is not an essential component, and parameter initialization is not an essential step. In Fig. 3, the NN is initialized after start-up, but the present invention is not limited to this. For example, the present invention can be applied to an NN constructed by another learning method without initialization.

[0088] Next, in S2, an image for machine learning is generated. That is, in S11 of Fig. 13, the control unit 11 acquires a processing method and controls the image processing unit 15 according to the acquired processing method. As a result, the image processing unit 15 reads an image corresponding to the processing method from the convex image memory 13 or the radial image memory 14 (S12).

[0089] The image processing unit 15 processes the read image according to the set processing method (S13). For example, when the image processing method (Example C-R1) described above is specified, the image processing unit 15 reads the convex image from the convex image memory 13, creates an inverted convex image by flipping the read convex image upside down, and synthesizes (links) the original convex image and the inverted convex image. During this synthesis, the image processing unit 15 synthesizes the images so that the transducer image in the synthesized image has the same shape, position, and brightness as the transducer image in the radial image. The image processing unit 15 clips the synthesized image to fit the size of the display screen to generate a simulated image (simulated radial image) (S14).

[0090] In this way, the image processing unit 15 processes the first ultrasound image captured by the first type of ultrasound imaging device to change at least the shape and position of the transducer region of the first ultrasound image, thereby generating a simulated image that simulates a second ultrasound image captured by a second type of ultrasound imaging device whose transducer arrangement direction is different from that of the first type.Furthermore, the image processing unit 15 may generate a simulated image by combining each of the above examples.

[0091] For example, as shown in (Example C-R1), the first ultrasound image may be a convex image Pc1 showing at least one of a tissue (biological part) and a lesion, or an inverted convex image PcR1 obtained by inverting this convex image Pc1. Alternatively, the first ultrasound image may be a rotated convex image (Example E2) obtained by rotating the convex image Pc1, or a first radial image (Example R-C1) obtained by cutting out a region not showing a lesion from an image captured by a radial-type ultrasonic endoscope. In this case, when using the inverted convex image or the rotated convex image, the lesion or tissue (biological part) shown in either the convex image Pc1 or the inverted convex image may be removed.

[0092] Furthermore, when an ultrasound balloon is not shown in the first ultrasound image, the image processing unit 15 may add an ultrasound balloon to the image. Conversely, when an ultrasound balloon is shown in the first ultrasound image, the image processing unit 15 may remove the image of the ultrasound balloon.

[0093] Furthermore, the image processing unit 15 may perform processing to add a multiple reflection image when multiple reflection lines due to the influence of multiple reflections are not captured in the first ultrasound image. Conversely, the image processing unit 15 may perform processing to remove a multiple reflection image when a multiple reflection image is captured in the first ultrasound image.

[0094] In this way, the image processing unit 15 generates the simulated convex images and the simulated radial images by employing various techniques. The control unit 11 controls the image processing unit 15 so that the ratio of the number of radial images, simulated convex images, convex images, and simulated radial images recorded in the recording device 25 is within two times the number of the least number of images.

[0095] When manually processing images, the image processing is performed so that the ratio of the number of radial images, simulated convex images, convex images, and simulated radial images is within two times the number of the most numerous images relative to the least numerous images.

[0096] The image processing unit 15 determines whether the required types and number of simulation images have been generated in S15 (S15). If the termination condition is not satisfied, the image processing unit 15 returns to S11 and repeats S11 to S15. If the termination condition is satisfied, the image processing unit 15 terminates the process.

[0097] Annotation is performed in S3 of FIG. 12 . For example, information indicating the position of the lesion image and correct information such as the differentiation result are added to the convex image, simulated convex image, radial image, and simulated radial image. Note that annotations corresponding to the simulated image may be automatically added by performing the same processing on the annotation image, which is an unprocessed image with annotations added, as when generating the simulated image. The input control unit 51 of the NN control circuit 50 inputs images serving as training data stored in the recording device 25 to the neural network 40 (S4). The input control unit 51 also inputs correct information stored in the recording device 25 to the learning loss calculation unit 55 (S4). Note that the neural network extracts a predetermined number of images (hereinafter referred to as a mini-batch) from a large number of images, and performs learning on the extracted mini-batch images. This mini-batch learning is performed for the number of data items to perform one unit of learning (hereinafter referred to as an epoch). For example, the number of epochs performed in learning may be predetermined.

[0098] The NN application unit 53 applies this mini-batch to the neural network 40 (S5). As a result, the neural network 40 outputs a classification output. The classification output of the neural network 40 is provided to the learning loss calculation unit 55, which calculates the learning loss (S6). The learning loss calculation unit 55 outputs the calculated learning loss to the NN control circuit 50 (backpropagation of error) (S7). The update unit 54 of the neural network 40 updates the parameters of the neural network 40 based on the input learning loss, for example, by the SGD method (S8).

[0099] Next, the NN application unit 53 determines whether the learning termination condition is met (S9). As described above, the process of extracting mini-batch training data and performing learning is repeated the number of times equal to the number of data items, and learning is performed until the specified number of epochs is reached. The NN application unit 53 determines whether the specified number of epochs has been reached, and if not (NO in S9), the process returns to S4 and S4 to S9 are repeated. Furthermore, if the specified number of epochs has been reached (YES in S9), the NN application unit 53 ends the process.

[0100] 12, image generation for machine learning and annotation are performed outside the iterative process consisting of S4 to S9, but these can also be performed one by one for the number of mini-batch images after image input during the iterative process. In this case, it is possible to save memory and HDD (hard disk) capacity for storing simulated images.

[0101] Artificial intelligence that enables diagnosis of ultrasound images, etc., is obtained using the image recognition processing model obtained by learning in the learning device 19. This image recognition processing model uses training data that is not biased in the distribution of lesions, etc., during learning, making it less likely to erroneously learn in a way that uses only the shape and position of the transducer or the range of the displayed ultrasound image as information for determining a lesion, and enables highly accurate inference.

[0102] FIG. 14 is a block diagram illustrating an endoscope processor that utilizes such artificial intelligence.

[0103] An ultrasonic endoscope 71, such as a convex ultrasonic endoscope 71a or a radial ultrasonic endoscope 71b, is connected to the endoscope processor 60. The endoscope processor 60 includes a control unit 61. The control unit 61 may be configured with a processor using a CPU, GPU, FPGA, or the like. The control unit 61 may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of its functions using hardware electronic circuits. The control unit 61 comprehensively controls each unit of the endoscope processor 60 based on user operations on the operation input unit 65.

[0104] The transmission / reception processing unit 62 of the endoscope processor 60 is controlled by the control unit 61 to drive the ultrasonic endoscope 71 to perform ultrasonic scanning. The transmission / reception processing unit 62 receives echo signals obtained by the ultrasonic endoscope 71 and performs predetermined signal processing such as detection. The image generation unit 63 generates convex images, radial images, etc. from the data obtained by the transmission / reception processing unit 62. The ultrasonic images from the image generation unit 63 are supplied to the image recognition unit 64 and also to the artificial intelligence 70 under the control of the control unit 61.

[0105] The artificial intelligence 70 performs image recognition processing on the input image to perform inferences such as the location and differentiation of a lesion, and outputs the inference results to the image recognition unit 64. Under the control of the control unit 61, the image recognition unit 64 provides the inference results of the artificial intelligence 70, together with the radial image and the convex image, to a display device 66 serving as a monitor, and displays them on a display screen 67.

[0106] The display device 66 is a display device such as an LCD (liquid crystal display device), and displays the ultrasound image from the image recognition unit 64 and the inference results of the artificial intelligence 70 on a display screen 67. In this way, the surgeon can confirm, from the display on the display screen 67, the diagnosis results for the ultrasound image obtained by the ultrasound endoscope 71, which are highly accurate and less susceptible to erroneous learning.

[0107] In this manner, in this embodiment, simulated images such as simulated radial images and simulated convex images are generated by, for example, changing the shape and position of transducer regions included in ultrasound images obtained by various types of ultrasound endoscopes, such as convex and radial types. This makes it possible to obtain training data that suppresses bias in the distribution of lesions and the like in ultrasound images used for machine learning, and to construct a highly accurate image recognition processing model.

[0108] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

[0109] Furthermore, among the technologies described herein, many of the controls and functions, mainly those described in the flowcharts, can be set by a program, and the above-described controls and functions can be realized by a computer reading and executing the program. The program can be recorded or stored, in whole or in part, as a computer program product on a portable medium such as a flexible disk, CD-ROM, or nonvolatile memory, or on a storage medium such as a hard disk or volatile memory, and can be distributed or provided at the time of product shipment, via a portable medium, or via a communication line. A user can easily realize the machine learning image generation method, machine learning method, endoscope processor, machine learning image generation program, and nonvolatile storage medium storing the machine learning image generation program of the present embodiment by downloading the program via a communication network and installing it on a computer, or by installing it on a computer from a recording medium.

Claims

1. A method for generating an image for machine learning, which processes a first ultrasound image captured by a first type of ultrasound imaging device to change at least the shape and position of the transducer region of the first ultrasound image, and generates, as an image for machine learning, a simulated image that simulates a second ultrasound image captured by a second type of ultrasound imaging device having a transducer arrangement direction different from that of the first type.

2. The method for generating an image for machine learning according to claim 1, wherein the second ultrasound image captured by the second type ultrasound imaging device is processed to change at least the shape and position of the transducer region of the second ultrasound image, thereby generating a simulated image as an image for machine learning that simulates the first ultrasound image captured by the first type ultrasound imaging device having a transducer arrangement direction different from that of the second type.

3. The method for generating images for machine learning described in claim 2, wherein images are processed so that the ratio of the number of the first ultrasound image, the first simulated image, the second ultrasound image, and the second simulated image is within two times the number of the most numerous image relative to the least numerous image.

4. The method for generating images for machine learning according to claim 1, wherein one of the first type ultrasound imaging device and the second type ultrasound imaging device is a radial type ultrasound endoscope in which transducers are arranged around the entire circumference of the axis, and the other is a non-radial type ultrasound endoscope in which transducers are not arranged around the entire circumference of the axis.

5. The method for generating images for machine learning according to claim 4, wherein the non-radial ultrasonic endoscope is a convex ultrasonic endoscope or a sector ultrasonic endoscope.

6. The method for generating images for machine learning described in claim 4, wherein, when the first type of ultrasound imaging device is the radial type ultrasound endoscope and the second type of ultrasound imaging device is a convex type ultrasound endoscope, the processing comprises partially removing the first ultrasound image so that a circular transducer region at the tip of the endoscope located in the center of the first ultrasound image becomes fan-shaped and the fan-shaped region is located at the edge of the image.

7. The method for generating images for machine learning according to claim 4, wherein, when the first type of ultrasound imaging device is a convex type ultrasound endoscope and the second type of ultrasound imaging device is the radial type ultrasound endoscope, the processing comprises preparing a plurality of the first ultrasound images, and stitching the plurality of first ultrasound images together so that a fan-shaped transducer region at the tip of the endoscope located at the edge of the first ultrasound images becomes a circle, and further the circle is located at the center.

8. The method for generating images for machine learning according to claim 7, wherein the plurality of first ultrasound images include a first convex image captured by the convex ultrasound endoscope, and a second convex image captured by the convex ultrasound endoscope and different from the first convex image.

9. The method for generating images for machine learning described in claim 8, wherein the plurality of first ultrasound images are a third convex image showing at least one of a biological part and a lesion, a fourth convex image obtained by inverting the third convex image, a fifth convex image obtained by rotating the third convex image, or a first radial image obtained by cutting out a part not showing a lesion from an image captured by a radial type ultrasonic endoscope, and when the fourth convex image or the fifth convex image is used, a lesion or biological part shown in one of the third convex image and the fourth convex image has been removed.

10. The method for generating images for machine learning according to claim 9, wherein the lesion is cancer, a tumor, a mass, pancreatitis, inflammation, liposarcoma, lymphoma, or fibrosis, and the biological site is the stomach, esophagus, duodenum, lung, pancreas, liver, gallbladder, bile duct, pancreatic duct, prostate, blood vessel, lymph node, muscle layer, fat layer, or bone.

11. The method for generating images for machine learning according to claim 1, further comprising, in addition to the processing, processing to add an ultrasonic balloon to the image if an ultrasonic balloon is not visible in the image, or processing to remove an ultrasonic balloon from the image if an ultrasonic balloon is visible in the image.

12. The method for generating images for machine learning according to claim 1, further comprising, in addition to the processing, performing processing to add multiple reflection images to the image if the image does not contain multiple reflection images, or performing processing to remove multiple reflection images from the image if the image contains multiple reflection images.

13. A machine learning method using, as training images, a radial image that is the first ultrasound image described in claim 6, a simulated convex image that is the simulated image described in claim 6, a convex image that is the third convex image described in claim 9, and a simulated radial image that is the simulated image described in claim 9.

14. The machine learning method according to claim 13, wherein the ratio of the number of radial images to the number of simulated convex images, the number of convex images to the number of simulated radial images is within two times the number of the least numerous images.

15. An endoscope processor capable of accessing artificial intelligence trained by the machine learning method of claim 13, which receives captured images transmitted from an ultrasonic endoscope, inputs the received captured images into the artificial intelligence, receives output results from the artificial intelligence, and outputs the received output results to a monitor.

16. A machine learning image generation program for causing a computer to execute the steps of: processing a first ultrasound image captured by a first type of ultrasound imaging device to change at least the shape and position of the transducer region of the first ultrasound image, and generating, as an image for machine learning, a simulated image that simulates a second ultrasound image captured by a second type of ultrasound imaging device having a transducer arrangement direction different from that of the first type.

17. A non-volatile storage medium storing an image generation program for machine learning that causes a computer to execute the following steps: by processing a first ultrasound image captured by a first type of ultrasound imaging device, changing at least the shape and position of the transducer region of the first ultrasound image, and generating, as an image for machine learning, a simulated image that simulates a second ultrasound image captured by a second type of ultrasound imaging device whose transducer arrangement direction is different from that of the first type.

Images