Endoscope system, operation method and program for medical image processing device, and recording medium

Abstract

A medical image processing device, an endoscope system, a method and program for operating the medical image processing device, and a recording medium are provided that detect an optimal lesion area according to the position in the living body of an acquired image. [Solution] Images are acquired from a medical device that sequentially captures images of multiple positions within the subject's body and displays them in real time, position information indicating the position within the body of the acquired images is acquired, and an attention area detection unit detects an attention area from the input image, selecting an attention area detection unit corresponding to the position indicated by the position information from multiple attention area detection units that respectively correspond to multiple positions within the body, and having the selected attention area detection unit detect the attention area from the acquired image.

Description

[Technical Field]

[0001] The present invention relates to a medical image processing apparatus, an endoscope system, an operating method and program for a medical image processing apparatus, and a recording medium, and more particularly to a technique for automatically detecting a lesion area from a medical image. [Background technology]

[0002] When capturing endoscopic images using an endoscope system, the position (site) within the lumen to be captured changes sequentially depending on the insertion depth of the endoscope into the living body. For this reason, the endoscope system captures images of multiple positions within the lumen in chronological order from the start to the end of capture.

[0003] Patent Document 1 discloses a technology for acquiring internal position identification information when an endoscopic image is captured, and identifying and displaying a position on a model corresponding to the internal position identification information on a part model, which is a model of a part inside a subject.

[0004] This technique makes it possible to match the position of the endoscope inside the body with its position on the guide image with high accuracy. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-206251 Summary of the Invention [Problem to be solved by the invention]

[0006] Image diagnostic technology is known for automatically detecting lesion areas from medical images. Known methods for automatic detection include detecting the difference between a medical image and a previous medical image, or between a medical image and a standard image, detecting lesion areas by detecting matches with lesion patterns using pattern matching, and using a pre-trained model detector that has learned the features of lesion areas accumulated in the past.

[0007] The structure of the mucosa and the characteristics of lesions vary depending on the position inside the lumen imaged by an endoscopic system. Therefore, systems that automatically detect lesion areas from endoscopic images have the problem that it is difficult to optimally detect lesion areas depending on the position at which the endoscopic image is taken. The same problem exists when automatically detecting lesion areas from images taken not only by endoscopic images but also by medical devices such as ultrasound diagnostic devices that sequentially take images at multiple positions inside the subject's body.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide a medical image processing device, an endoscopic system, an operating method and program for the medical image processing device, and a recording medium that detects the optimal lesion area according to the position in the living body of an acquired image. [Means for solving the problem]

[0009] In order to achieve the above object, one aspect of a medical image processing device is a medical image processing device that includes an image acquisition unit that acquires images from a medical device that sequentially captures images of multiple positions within a subject's body and displays them in real time; a position information acquisition unit that acquires position information indicating the position of the acquired image within the body; a region of interest detection unit that detects a region of interest from an input image, the region of interest detection unit being multiple region of interest detection units each corresponding to a multiple position within the body; a selection unit that selects from the multiple region of interest detection units the region of interest detection unit that corresponds to the position indicated by the position information; and a control unit that causes the selected region of interest detection unit to detect the region of interest from the image acquired.

[0010] According to this aspect, a medical device sequentially captures images of multiple positions within the subject's body and displays them in real time, and the acquired image is used by a medical device to acquire position information indicating the position within the body, and an area of ​​interest detection unit detects an area of ​​interest from the input image. From multiple area of ​​interest detection units corresponding to multiple positions within the body, an area of ​​interest detection unit corresponding to the position indicated by the position information is selected, and the selected area of ​​interest detection unit is made to detect an area of ​​interest from the acquired image, thereby making it possible to detect an optimal lesion area according to the position within the body of the acquired image.

[0011] The image acquisition unit preferably sequentially acquires a plurality of images taken in time series inside the living body, thereby enabling optimal detection of a lesion area according to the position inside the living body in the plurality of images taken in time series inside the living body.

[0012] Furthermore, it is preferable that the image acquisition unit sequentially acquires a plurality of images captured at a constant frame rate, thereby enabling optimal detection of a lesion area according to the position in the living body of the plurality of images captured at the constant frame rate.

[0013] The plurality of interest region detection units are preferably a plurality of trained models, which allows appropriate detection of lesion regions.

[0014] Furthermore, it is preferable that each of the multiple trained models is trained using a different data set, so that each model can perform different detections.

[0015] Furthermore, it is preferable that the multiple trained models are models trained using datasets consisting of images captured at different positions within a living body, thereby enabling detection of lesion areas according to the positions within the living body of the images.

[0016] The position information acquisition unit preferably includes a position recognition unit that recognizes a position within the living body from the acquired image, thereby making it possible to appropriately recognize the position within the living body of the image.

[0017] The position information acquisition unit may also include an input unit that receives in-vivo position information input from a user, thereby enabling the in-vivo position of the image to be appropriately recognized.

[0018] It is preferable to provide a display control unit that causes the medical device to display the location indicated by the acquired location information, thereby allowing a doctor to know that the location has been properly recognized.

[0019] The medical device preferably includes an ultrasound probe that transmits ultrasound waves from outside the body of a subject into the body of the subject and receives the ultrasound waves reflected from inside the body of the subject, an ultrasound image generator that generates an ultrasound image using the ultrasound waves received by the ultrasound probe, and a display that displays the ultrasound image, thereby enabling optimal detection of a lesion area according to the position inside the body of the subject in the ultrasound image generated by the ultrasound diagnostic device.

[0020] The medical device preferably includes an endoscope inserted into the body cavity of a subject, a camera for capturing images of the inside of the body cavity, and a display unit for displaying the captured images, thereby enabling optimal detection of a lesion area according to the position within the living body in the endoscopic image generated by the endoscopic system.

[0021] In order to achieve the above-mentioned object, one aspect of an endoscopic system is an endoscopic system comprising: a medical device having an endoscope to be inserted into a body cavity of a subject, a camera that photographs the inside of the body cavity, and a display unit that displays the photographed images; an image acquisition unit that acquires images from the medical device that sequentially photographs images of multiple positions within the subject's body and displays them in real time; a position information acquisition unit that acquires position information indicating the position of the acquired images within the body; an area of ​​interest detection unit that detects an area of ​​interest from an input image, the area of ​​interest detection unit being multiple area of ​​interest detection units each corresponding to a multiple position within the body; a selection unit that selects from the multiple area of ​​interest detection units the area of ​​interest detection unit that corresponds to the position indicated by the position information; and a medical image processing device that causes the selected area of ​​interest detection unit to detect the area of ​​interest from the image acquired.

[0022] According to this aspect, images are sequentially taken from multiple positions within the subject's body from a medical device having an endoscope inserted into the body cavity of the subject, a camera that takes images inside the body cavity, and a display unit that displays the taken images. Position information indicating the position of the image within the body is obtained, and an area of ​​interest detection unit detects an area of ​​interest from the input image. From multiple area of ​​interest detection units that correspond respectively to multiple positions within the body, an area of ​​interest detection unit corresponding to the position indicated by the position information is selected, and the selected area of ​​interest detection unit is made to detect an area of ​​interest from the acquired image, so that it is possible to detect an optimal lesion area according to the position within the body of the acquired image.

[0023] In order to achieve the above-mentioned object, one aspect of a medical image processing method is a medical image processing method comprising: an image acquisition step of acquiring images from a medical device that sequentially captures images of multiple positions within a subject's living body and displays them in real time; a position information acquisition step of acquiring position information indicating the position of the acquired images within the living body; a region of interest detection step of detecting a region of interest from an input image, wherein the region of interest detection steps correspond to multiple positions within the living body, respectively; a selection step of selecting from the multiple region of interest detection steps the region of interest detection step that corresponds to the position indicated by the position information; and a control step of detecting the region of interest from the image acquired by the selected region of interest detection step.

[0024] According to this aspect, a region of interest detection process obtains position information indicating the position within the body of the image obtained from a medical device that sequentially captures images of multiple positions within the body of a subject and displays them in real time, and detects a region of interest from the input image.From multiple region of interest detection processes that respectively correspond to multiple positions within the body, a region of interest detection process corresponding to the position indicated by the position information is selected, and the region of interest is detected from the image obtained by the selected region of interest detection process, thereby making it possible to detect an optimal lesion region according to the position within the body of the obtained image.

[0025] One aspect of a program executed by a computer to achieve the above-mentioned object is a program that causes a computer to execute the following steps: an image acquisition process for acquiring images from a medical device that sequentially captures images of multiple positions within a subject's body and displays them in real time; a position information acquisition process for acquiring position information indicating the position of the acquired images within the body; a region of interest detection process for detecting a region of interest from an input image, wherein the region of interest detection process corresponds to multiple positions within the body, respectively; a selection process for selecting from the multiple region of interest detection processes the region of interest detection process that corresponds to the position indicated by the position information; and a control process for detecting the region of interest from the image acquired by the selected region of interest detection process.

[0026] According to this aspect, the area of ​​interest detection process acquires position information indicating the position within the body of an image acquired from a medical device that sequentially captures images of multiple positions within the body of a subject, and detects an area of ​​interest from the input image.From multiple area of ​​interest detection processes corresponding to multiple positions within the body, an area of ​​interest detection process corresponding to the position indicated by the position information is selected, and the area of ​​interest is detected from the acquired image by the selected area of ​​interest detection process, so that it is possible to detect an optimal lesion area according to the position within the body of the acquired image. [Effects of the Invention]

[0027] According to the present invention, it is possible to detect an optimal lesion area according to the position in the living body of an acquired image. [Brief explanation of the drawings]

[0028] [Figure 1] External view of the endoscope system [Figure 2] Block diagram showing the functions of the endoscope system [Figure 3] Graph showing light intensity distribution [Figure 4] 1 is a flowchart showing the processing of an image diagnosis method performed by the endoscope system 10. [Figure 5] FIG. 10 is a diagram showing an example of a display unit that displays a captured image and acquired location information. [Figure 6]FIG. 10 is a diagram showing an example of a display unit that displays a captured image and acquired location information. [Figure 7] Block diagram showing the functions of the endoscope system [Figure 8] Block diagram showing the functions of an ultrasound diagnostic device DETAILED DESCRIPTION OF THE INVENTION

[0029] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] First Embodiment [Configuration of endoscope system] 1 is an external view showing an endoscope system 10 (an example of a medical device) according to a first embodiment. As shown in FIG. 1, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 16, a display unit 18, and an input unit 20.

[0031] In this embodiment, the endoscope 12 is a lower endoscope that is inserted through the anus of a subject and used to observe the rectum, large intestine, etc. The endoscope 12 is optically connected to a light source device 14. The endoscope 12 is also electrically connected to a processor device 16.

[0032] The endoscope 12 has an insertion section 12A that is inserted into the body cavity of the subject, an operation section 12B provided at the base end portion of the insertion section 12A, a bending section 12C provided at the tip side of the insertion section 12A, and a tip section 12D.

[0033] The operation unit 12B is provided with an angle knob 12E and a mode changeover switch 13.

[0034] By operating the angle knob 12E, the bending portion 12C is bent, and the distal end portion 12D is oriented in a desired direction by this bending operation.

[0035] The mode selector switch 13 is used to switch between observation modes. The endoscope system 10 has multiple observation modes, each with a different wavelength pattern of irradiated light. A doctor can set a desired observation mode by operating the mode selector switch 13. The endoscope system 10 generates an image corresponding to the set observation mode by combining the wavelength pattern and image processing, and displays the image on the display unit 18.

[0036] The operation unit 12B is also provided with an acquisition instruction input unit (not shown). The acquisition instruction input unit is an interface for a doctor to input an instruction to acquire a still image. The acquisition instruction input unit accepts an instruction to acquire a still image. The instruction to acquire a still image accepted by the acquisition instruction input unit is input to the processor device 16.

[0037] The processor device 16 is electrically connected to a display unit 18 and an input unit 20. The display unit 18 is a display device that outputs and displays an image of an observation target and information related to the image of the observation target. The input unit 20 functions as a user interface that accepts input operations such as function settings and various instructions for the endoscope system 10.

[0038] 2 is a block diagram showing the functions of the endoscope system 10. As shown in FIG. 2, the light source device 14 includes a first laser light source 22A, a second laser light source 22B, and a light source control unit 24.

[0039] The first laser light source 22A is a blue laser light source with a center wavelength of 445 nm. The second laser light source 22B is a violet laser light source with a center wavelength of 405 nm. Laser diodes can be used as the first laser light source 22A and the second laser light source 22B. The light emissions of the first laser light source 22A and the second laser light source 22B are individually controlled by a light source control unit 24. The emission intensity ratio between the first laser light source 22A and the second laser light source 22B is freely changeable.

[0040] As shown in FIG. 2, the endoscope 12 includes an optical fiber 28A, an optical fiber 28B, a phosphor 30, a diffusing member 32, an imaging lens 34, an imaging element 36, and an analog-to-digital converter 38.

[0041] The first laser light source 22A, the second laser light source 22B, the optical fiber 28A, the optical fiber 28B, the phosphor 30, and the diffusing member 32 constitute an irradiation unit.

[0042] The laser light emitted from the first laser light source 22A is irradiated onto a phosphor 30 arranged at the tip 12D of the endoscope 12 via an optical fiber 28A. The phosphor 30 is configured to contain multiple types of phosphors that absorb part of the blue laser light from the first laser light source 22A and emit green to yellow excitation light L. As a result, the light emitted from the phosphor 30 is a green to yellow excitation light L, which is excited by the blue laser light from the first laser light source 22A. 11 and blue laser light L that is transmitted without being absorbed by the phosphor 30. 12 These are combined to form white (pseudo-white) light L1.

[0043] It should be noted that the white light referred to here is not limited to light that strictly includes all wavelength components of visible light. For example, it may include light in a specific wavelength band such as R, G, or B, and in a broad sense, it also includes light that includes wavelength components from green to red, or light that includes wavelength components from blue to green, etc.

[0044] On the other hand, the laser light emitted from the second laser light source 22B is irradiated via the optical fiber 28B onto a diffusing member 32 disposed at the tip portion 12D of the endoscope 12. The diffusing member 32 may be made of a translucent resin material or the like. The light emitted from the diffusing member 32 becomes narrow-band wavelength light L2 with a uniform light intensity within the irradiation area.

[0045] 3 is a graph showing the intensity distribution of light L1 and light L2. The light source control unit 24 (an example of a wavelength pattern changing unit) changes the light intensity ratio between the first laser light source 22A and the second laser light source 22B. This changes the light intensity ratio between light L1 and light L2, and changes the wavelength pattern of irradiation light L0, which is a composite light of light L1 and light L2. Therefore, irradiation light L0 with different wavelength patterns can be emitted depending on the observation mode.

[0046] 2, the imaging unit (camera) is made up of the imaging lens 34, the imaging element 36, and the analog-to-digital converter 38. The imaging unit is disposed at the tip 12D of the endoscope 12.

[0047] The imaging lens 34 forms an image of the incident light on the imaging element 36. The imaging element 36 generates an analog signal corresponding to the received light. A CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used as the imaging element 36. The analog signal output from the imaging element 36 is converted into a digital signal by an analog-to-digital converter 38 and input to the processor device 16.

[0048] As shown in FIG. 2, the processor device 16 includes an imaging control unit 40, an image processing unit 42, an image acquisition unit 44, a lesion area detection unit 46, a position information acquisition unit 48, a selection unit 52, a lesion area detection control unit 54, a display control unit 58, a memory control unit 60, and a memory unit 62.

[0049] The imaging control unit 40 controls the light source control unit 24 of the light source device 14, the image sensor 36 and analog-to-digital conversion unit 38 of the endoscope 12, and the image processing unit 42 of the processor device 16, thereby providing overall control over the imaging of moving images and still images by the endoscopic system 10.

[0050] The image processing unit 42 performs image processing on the digital signal input from the analog-to-digital conversion unit 38 of the endoscope 12 to generate image data (hereinafter referred to as image) that represents an image. The image processing unit 42 performs image processing according to the wavelength pattern of the irradiated light at the time of imaging.

[0051] The image acquisition unit 44 acquires the images generated by the image processing unit 42. That is, the image acquisition unit 44 sequentially acquires a plurality of images obtained by capturing the inside of the subject's body cavity (an example of the inside of a living body) in time series at a constant frame rate. The image acquisition unit 44 may acquire images input from the input unit 20 or images stored in the storage unit 62. The image acquisition unit 44 may also acquire images from an external device such as a server connected to a network (not shown). In these cases, the images are also preferably a plurality of images captured in time series.

[0052] Lesion area detection unit 46 detects a lesion area from an input image (input image). Lesion area detection unit 46 includes first detection unit 46A, second detection unit 46B, third detection unit 46C, fourth detection unit 46D, fifth detection unit 46E, sixth detection unit 46F, seventh detection unit 46G, and eighth detection unit 46H (examples of multiple interest area detection units) corresponding to multiple positions within the body cavity, respectively. Here, as an example, first detection unit 46A corresponds to the rectum, second detection unit 46B corresponds to the sigmoid colon, third detection unit 46C corresponds to the descending colon, fourth detection unit 46D corresponds to the transverse colon, fifth detection unit 46E corresponds to the ascending colon, sixth detection unit 46F corresponds to the cecum, seventh detection unit 46G corresponds to the ileum, and eighth detection unit 46H corresponds to the jejunum (an example of a multiple interest area detection process).

[0053] The first detection unit 46A, the second detection unit 46B, the third detection unit 46C, the fourth detection unit 46D, the fifth detection unit 46E, the sixth detection unit 46F, the seventh detection unit 46G, and the eighth detection unit 46H are each a trained model. The multiple trained models are each a model trained using a different data set. More specifically, the multiple trained models are each a model trained using a data set consisting of images captured at different positions within a body cavity.

[0054] That is, the first detection unit 46A is a model trained using a dataset consisting of images of the rectum, the second detection unit 46B is a dataset consisting of images of the sigmoid colon, the third detection unit 46C is a dataset consisting of images of the descending colon, the fourth detection unit 46D is a dataset consisting of images of the transverse colon, the fifth detection unit 46E is a dataset consisting of images of the ascending colon, the sixth detection unit 46F is a dataset consisting of images of the cecum, the seventh detection unit 46G is a dataset consisting of images of the ileum, and the eighth detection unit 46H is a model trained using a dataset consisting of images of the jejunum.

[0055] These trained models preferably use support vector machines, convolutional neural networks, or the like.

[0056] The position information acquiring unit 48 acquires position information I that indicates the position within the body cavity of the image acquired by the image acquiring unit 44. Here, the doctor inputs the position information I using the input unit 20. The position information acquiring unit 48 acquires the position information I input from the input unit 20.

[0057] As position information I indicating the position of the image within the body cavity, the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, cecum, ileum, jejunum, etc. may be input. These position candidates may be displayed on the display unit 18 in a selectable manner, and the doctor may select one using the input unit 20.

[0058] The selection unit 52 selects a detection unit corresponding to the position indicated by the position information I acquired by the position information acquisition unit 48 from the lesion area detection unit 46. That is, the selection unit 52 selects the first detection unit 46A if the position indicated by the position information I is the rectum, the second detection unit 46B if the position is the sigmoid colon, the third detection unit 46C if the position is the descending colon, the fourth detection unit 46D if the position is the transverse colon, the fifth detection unit 46E if the position is the ascending colon, the sixth detection unit 46F if the cecum, the seventh detection unit 46G if the position is the ileum, or the eighth detection unit 46H if the position is the jejunum.

[0059] The lesion area detection control unit 54 (an example of a control unit) causes the detection unit selected by the selection unit 52 to detect a lesion area (an example of a region of interest) from the image acquired by the image acquisition unit 44. The lesion area here is not limited to areas caused by disease, but also includes areas that appear to be in a state different from a normal state. Examples of lesion areas include polyps, cancer, colonic diverticula, inflammation, treatment scars such as EMR (Endoscopic Mucosal Resection) scars or ESD (Endoscopic Submucosal Dissection) scars, clip sites, bleeding points, perforations, and vascular atypia.

[0060] The image acquisition unit 44, the lesion area detection unit 46, the position information acquisition unit 48, the selection unit 52, and the lesion area detection control unit 54 constitute a medical image processing device 56.

[0061] Display control unit 58 causes display unit 18 to display the image generated by image processing unit 42. Display control unit 58 also displays (the position indicated by) position information I acquired by position information acquisition unit 48. Furthermore, the lesion area detected by lesion area detection unit 46 may be superimposed on the image so that it can be recognized.

[0062] The storage control unit 60 stores the image generated by the image processing unit 42 in the storage unit 62. For example, the storage unit 62 stores information such as an image captured in accordance with an instruction to acquire a still image and information on the wavelength pattern of the irradiated light L0 when the image was captured.

[0063] The storage unit 62 is, for example, a storage device such as a hard disk. Note that the storage unit 62 is not limited to one built into the processor device 16. For example, it may be an external storage device (not shown) connected to the processor device 16. The external storage device may be connected via a network (not shown).

[0064] The endoscope system 10 configured in this manner normally captures moving images at a constant frame rate and displays the captured images and position information I of the images on the display unit 18. In addition, a lesion area is detected from the captured moving image, and the detected lesion area is displayed on the display unit 18 so as to be superimposed on the moving image in a recognizable manner.

[0065] [Operation method of endoscope system] FIG. 4 is a flowchart showing the processing of an image diagnosis method (an example of an operation method of an endoscope system, an example of a medical image processing method) by the endoscope system 10.

[0066] To start image diagnosis using the endoscope system 10, in step S1 (an example of an image acquisition step), video is captured and displayed under the control of the imaging control unit 40. That is, the light source control unit 24 sets the light emitted from the first laser light source 22A and the light emitted from the second laser light source 22B to a light intensity ratio corresponding to a desired observation mode. As a result, the illumination light L0 of a desired wavelength pattern is irradiated onto the observation area inside the body cavity of the subject.

[0067] The imaging control unit 40 controls the image sensor 36, the analog-to-digital conversion unit 38, and the image processing unit 42 to capture an image of the observation area by receiving reflected light from the observation area. The image acquisition unit 44 acquires the captured image.

[0068] In this way, the endoscope system 10 captures images (moving images) at a constant frame rate.

[0069] Next, in step S2 (an example of a position information acquisition step), position information I is acquired by the position information acquisition unit 48. Here, the doctor uses the input unit 20 to input the position information I indicating the position within the lumen of the image captured in step S1. The position information acquisition unit 48 acquires the position information I input from the input unit 20. The doctor may input the position information I before capturing the image.

[0070] If new position information I is not input from the input unit 20, the last input position information I is acquired as the current position information I. The doctor only needs to input position information I from the input unit 20 when the imaging position (site) of the endoscope 12 changes.

[0071] Subsequently, in step S3 (an example of a selection step), selector 52 selects one of detectors 1 to 8, 46A based on position information I acquired in step S2. That is, if the position within the lumen indicated by position information I is the rectum, selector 52 selects first detector 46A; if it is the sigmoid colon, selector 52 selects second detector 46B; if it is the descending colon, selector 52 selects third detector 46C; if it is the transverse colon, selector 52 selects fourth detector 46D; if it is the ascending colon, selector 52 selects sixth detector 46F; if it is the cecum, selector 52 selects seventh detector 46G; if it is the ileum, selector 52 selects eighth detector 46H.

[0072] Next, in step S4 (an example of a control step), lesion area detection control unit 54 causes the selected detection unit (an example of a selected attention area detection step) to detect a lesion area in the image acquired by image acquisition unit 44. For example, if first detection unit 46A is selected, first detection unit 46A detects a lesion area from the image. Also, if second detection unit 46B is selected, second detection unit 46B detects a lesion area from the image.

[0073] The first to eighth detectors 46A to 46H are each trained according to a position (site) within the lumen. Therefore, by detecting the lesion area using a detector according to the acquired position information I, appropriate detection becomes possible.

[0074] In step S5, the display control unit 58 causes the images captured in step S1 to be displayed in real time on the display unit 18. Note that real-time display refers to a process of updating and displaying images captured sequentially as needed, and includes displaying images with a time lag such as the time required for image processing and the time required for communication to the display unit 18.

[0075] Furthermore, the display control unit 58 causes the display unit 18 to display the position information I acquired in step S3.

[0076] Furthermore, if a lesion area is detected in step S4, display control unit 58 superimposes and displays the detected lesion area on the image displayed on display unit 18 so that it can be recognized.

[0077] Figures 5 and 6 show the captured image G E 5 is a diagram showing an example of the display unit 18 displaying the acquired position information I and the character "Rectum" corresponding to the position indicated by the position information I on the image G. E Here, the location information I is displayed in English in the upper right corner of the image, but the display position and language are not limited to this example.

[0078] In the case shown in FIG. 6, the position information I is E Schematic diagram of the lumen G S is displayed on the display unit 18, and a schematic diagram G S A circle is displayed at the position indicated by the upper position information I. Here, a circle is used as the graphic, but the shape and color are not limited to this and it is sufficient if the display allows the doctor to recognize the position.

[0079] By displaying the position information I in this way, the doctor can confirm that the position information I has been set correctly.

[0080] The storage control unit 60 stores the image G E may be stored in the storage unit 62. E and Image G E The position information I may be stored in the storage unit 62 in association with the position information I.

[0081] Finally, in step S6, it is determined whether or not to end the image diagnosis by the endoscope system 10. The doctor can use the input unit 20 to input an instruction to end the imaging operation of the endoscope system 10.

[0082] If an end instruction is input, the process of this flowchart ends. If an end instruction is not input, the process returns to step S1 and continues shooting.

[0083] In this way, by acquiring the position information I and detecting the lesion area by a detection unit according to the position information I, the accuracy of detecting the lesion area can be improved.

[0084] Here, the endoscopic system 10 acquires the position information I by inputting it from the input unit 20 by the doctor, but shape information of the bending portion 12C of the endoscope 12 may be acquired by an endoscope insertion shape observation device (not shown) using a magnetic coil or the like, and the position information I of the tip portion 12D may be estimated from this shape information. Alternatively, shape information of the bending portion 12C of the endoscope 12 may be acquired by irradiating X-rays from outside the body of the subject, and the position information I of the tip portion 12D may be estimated from this shape information.

[0085] Although the example described here is applied to a lower extremity endoscope, it can also be applied to an upper extremity endoscope that is inserted through the mouth or nose of a subject and used to observe the esophagus, stomach, etc. In this case, the pharynx, esophagus, stomach, duodenum, etc. are input from input unit 20 as position information I indicating the position of the image within the body cavity. Furthermore, lesion area detection unit 46 may be provided with a detection unit that detects the lesion area of ​​the pharynx, esophagus, stomach, duodenum, etc.

[0086] Here, lesion area detection unit 46 includes multiple detection units, but it may also include only one detector and switch the data or parameters used for each position. For example, lesion area detection unit 46 includes only first detection unit 46A, and parameters corresponding to acquired position information I are set in first detection unit 46A. First detection unit 46A detects the lesion area using the set parameters.

[0087] Even with this configuration, the lesion area is detected by a detection unit according to the position information I, so that the accuracy of detecting the lesion area can be improved.

[0088] <Second embodiment> The position information I is not limited to being input from outside the endoscope system 10. For example, it can also be estimated from a captured image.

[0089] Fig. 7 is a block diagram showing the functions of an endoscope system 70. Note that parts that are common to the block diagram shown in Fig. 2 are given the same reference numerals, and detailed description thereof will be omitted.

[0090] The endoscope system 70 includes a position recognition unit 50 in a position information acquisition unit 48. In this embodiment, an image acquired by the image acquisition unit 44 is input to the position information acquisition unit 48. The position recognition unit 50 recognizes (estimates) the position (site) within the lumen where the image was captured, based on the image feature amount of the input image.

[0091] The position recognition unit 50 is a trained model that has trained images of the mucous membrane at each position using a machine learning algorithm such as deep learning.

[0092] In this way, according to the endoscope system 70, the position recognition unit 50 can analyze the image, thereby acquiring the position information I.

[0093] Based on the acquired position information I, the selection unit 52 selects one of the detection units: the first detection unit 46A, the second detection unit 46B, the third detection unit 46C, the fourth detection unit 46D, the fifth detection unit 46E, the sixth detection unit 46F, the seventh detection unit 46G, and the eighth detection unit 46H.

[0094] The lesion area detection control unit 54 causes the selected detection unit to detect a lesion area in the image acquired by the image acquisition unit 44.

[0095] With this configuration, it is possible to immediately acquire the position information I when the position of the tip portion 12D of the endoscope 12 changes, thereby improving the accuracy of detecting the lesion area.

[0096] Furthermore, similarly to the first embodiment, the display control unit 58 displays the position information I on the display unit 18. By displaying the position information I in this manner, the doctor can confirm that the position information I has been correctly recognized. If the displayed position information I is incorrect, it may be configured so that it can be corrected using the input unit 20.

[0097] The position recognition unit 50 may detect characteristic landmarks at each position and estimate the position from the detected landmark information. For example, if bile is detected as a landmark, the position is estimated to be the duodenum, if villi are detected, the position is estimated to be the ileum or jejunum, and if the ileocecal valve is detected, the position is estimated to be the cecum or ascending colon.

[0098] In the case of upper endoscopy, if the vocal cords or epiglottis are detected as landmarks, the location can be estimated as the pharynx, and if squamous epithelium is detected, the location can be estimated as the esophagus.

[0099] <Third embodiment> An ultrasound diagnostic device that generates ultrasound images is a medical device that sequentially captures images at multiple positions inside a living body of a subject. Here, an example in which the present invention is applied to an ultrasound diagnostic device will be described.

[0100] Fig. 8 is a block diagram showing the functions of the ultrasound diagnostic device 100. As shown in Fig. 8, the ultrasound diagnostic device 100 includes a display unit 18, an input unit 20, a medical image processing device 56, a display control unit 58, a memory control unit 60, a memory unit 62, an ultrasound probe 102, a transmission / reception control unit 104, a transmitting unit 106, a receiving unit 108, and an image processing unit 110.

[0101] Note that there are cases where the term "ultrasound diagnostic device" is used without including the ultrasound probe 102. In this case, the ultrasound diagnostic device is connected to the ultrasound probe.

[0102] The ultrasound probe 102 transmits ultrasound waves from outside the body of the subject toward the inside of the subject's body and receives the ultrasound waves reflected from the inside of the subject's body.

[0103] The ultrasonic probe 102 is connected to a transmitting unit 106 and a receiving unit 108. The transmitting unit 106 and the receiving unit 108 transmit and receive ultrasonic waves using the ultrasonic probe 102 under the control of the transmission and reception control unit 104.

[0104] The transmitting unit 106 outputs a transmission signal to an ultrasonic transducer (not shown) included in the ultrasonic probe 102. The ultrasonic transducer of the ultrasonic probe 102 transmits ultrasonic waves corresponding to the transmission signal to the subject.

[0105] Furthermore, the ultrasound waves reflected inside the subject's body are received by the ultrasound transducer that transmitted the ultrasound waves. The ultrasound transducer outputs a reflected wave signal to the receiving unit 108. The receiving unit 108 receives this reflected wave signal. Furthermore, the receiving unit 108 performs amplification processing, analog-to-digital conversion processing, etc. on the reflected wave signal, and outputs a digital signal to the image processing unit 110.

[0106] The image processing unit 110 (an example of an ultrasound image generating unit) performs image processing on the digital signal input from the receiving unit 108, and generates an ultrasound image signal.

[0107] The display unit 18, input unit 20, medical image processing device 56, display control unit 58, storage control unit 60, and storage unit 62 have the same configurations as those of the endoscope system 10 according to the first embodiment.

[0108] The position information acquisition unit 48 included in the medical image processing device 56 acquires, as the position information I, the liver, gallbladder, pancreas, spleen, kidneys, uterus, ovaries, prostate, and the like.

[0109] In addition, the lesion area detection units 46 provided in the medical image processing device 56 have the following functions: the first detection unit 46A detects lesion areas in the liver, the second detection unit 46B detects lesion areas in the gallbladder, the third detection unit 46C detects lesion areas in the pancreas, the fourth detection unit 46D detects lesion areas in the spleen, the fifth detection unit 46E detects lesion areas in the kidneys, the sixth detection unit 46F detects lesion areas in the uterus, the seventh detection unit 46G detects lesion areas in the ovaries, and the eighth detection unit 46H detects lesion areas in the prostate.

[0110] That is, here, the first detection unit 46A is a trained model trained using a dataset consisting of images of the liver, the second detection unit 46B is a dataset consisting of images of the gallbladder, the third detection unit 46C is a dataset consisting of images of the pancreas, the fourth detection unit 46D is a dataset consisting of images of the spleen, the fifth detection unit 46E is a dataset consisting of images of the kidneys, the sixth detection unit 46F is a dataset consisting of images of the uterus, the seventh detection unit 46G is a dataset consisting of images of the ovaries, and the eighth detection unit 46H is a trained model trained using a dataset consisting of images of the prostate.

[0111] The processing of the image diagnosis method by the ultrasound diagnostic apparatus 100 is similar to the flowchart shown in FIG.

[0112] That is, an ultrasound probe is used to transmit and receive ultrasound to and from the subject, and an ultrasound image signal is generated by the image processing unit 110 (step S1). This operation is performed at a constant frame rate. In addition, position information I of the location where the ultrasound image is being taken is acquired by, for example, input by the doctor via the input unit 20 (step S2).

[0113] The selection unit 52 selects one of the detection units from the first detection unit 46A, the second detection unit 46B, the third detection unit 46C, the fourth detection unit 46D, the fifth detection unit 46E, the sixth detection unit 46F, the seventh detection unit 46G, and the eighth detection unit 46H (step S3) based on the position information I. That is, if the position in the subject's living body indicated by the position information I is the liver, the first detection unit 46A is selected; if the position is the gallbladder, the second detection unit 46B is selected; if the position is the pancreas, the third detection unit 46C is selected; if the position is the spleen, the fourth detection unit 46D is selected; if the position is the kidney, the fifth detection unit 46E is selected; if the position is the uterus, the sixth detection unit 46F is selected; if the position is the ovary, the seventh detection unit 46G is selected; or if the position is the prostate, the eighth detection unit 46H is selected.

[0114] Lesion area detection control unit 54 causes the selected detection unit to detect a lesion area in the image acquired by image acquisition unit 44 (step S4).

[0115] Display control unit 58 causes display unit 18 to display in real time the image captured in step S1. Display control unit 58 also causes display unit 18 to display position information I (step S5). If a lesion area is detected in step S4, the detected lesion area is superimposed on the image displayed on display unit 18 so that it can be recognized.

[0116] The above operations are repeated until it is determined in step S6 that the image diagnosis by the ultrasound diagnostic apparatus 100 is to be ended.

[0117] As described above, even in an ultrasound diagnostic apparatus, by acquiring position information I and detecting a lesion area using a detection unit according to the position information I, the accuracy of detecting a lesion area can be improved.

[0118] <Additional Notes> In addition to the above-mentioned aspects and examples, the following configurations are also included within the scope of the present invention.

[0119] (Appendix 1) The medical image analysis processing unit detects an area of ​​interest that is an area that requires attention based on the feature amount of pixels of the medical image, The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0120] (Appendix 2) The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the feature amount of pixels of the medical image, The medical image analysis result acquisition unit is a medical image processing device that acquires the analysis results of the medical image analysis processing unit.

[0121] (Appendix 3) The medical image analysis result acquisition unit: Obtained from a recording device that records the results of medical image analysis; The analysis results are a region of interest, which is an area of ​​interest contained in a medical image, and / or the presence or absence of an object of interest.

[0122] (Appendix 4) A medical image processing apparatus in which the medical image is a normal light image obtained by irradiating white light or light of multiple wavelength bands as white light.

[0123] (Appendix 5) A medical image is an image obtained by irradiating light in a specific wavelength range. A medical imaging device, wherein the specific wavelength band is narrower than the white wavelength band.

[0124] (Appendix 6) A medical image processing device in which the specific wavelength band is the blue or green band of the visible range.

[0125] (Appendix 7) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm.

[0126] (Appendix 8) The specific wavelength band is the red band of the visible range for medical imaging processing equipment.

[0127] (Appendix 9) A medical image processing device, wherein the specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less.

[0128] (Appendix 10) A medical image processing device, wherein the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients.

[0129] (Appendix 11) A medical image processing device in which the specific wavelength band includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light in the specific wavelength band has a peak wavelength in a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

[0130] (Appendix 12) Medical images are in vivo images that show the inside of a living body. In vivo images are medical image processing devices that have information on the fluorescence emitted by fluorescent substances in the living body.

[0131] (Appendix 13) Fluorescence is a medical imaging device that obtains fluorescence by irradiating the inside of a living body with excitation light having a peak wavelength of 390 nm or more and 470 nm or less.

[0132] (Appendix 14) Medical images are in vivo images that show the inside of a living body. The specific wavelength band is a wavelength band of infrared light in a medical imaging device.

[0133] (Appendix 15) A medical image processing device in which the specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light in the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less.

[0134] (Appendix 16) the medical image acquisition unit includes a special light image acquisition unit that acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating white light or light of a plurality of wavelength bands as white light; Medical image processing equipment uses special light for medical images.

[0135] (Appendix 17) A medical image processing device that obtains signals of specific wavelength bands through calculations based on the RGB (Red, Green, Blue) or CMY (Cyan, Magenta, Yellow) color information contained in normal light images.

[0136] (Appendix 18) a feature image generating unit that generates a feature image by calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light in a specific wavelength band; Medical image processing device that processes medical images as feature images.

[0137] (Appendix 19) A medical image processing device according to any one of appendices 1 to 18; an endoscope that acquires an image by irradiating at least one of light in a white wavelength band and light in a specific wavelength band; An endoscope apparatus comprising:

[0138] (Appendix 20) A diagnostic support device comprising the medical image processing device according to any one of appendices 1 to 18.

[0139] (Appendix 21) A medical service support device comprising the medical image processing device according to any one of appendices 1 to 18.

[0140] <Other> The above image processing method can be configured as a program for causing a computer to execute each step, and a non-transitory recording medium such as a CD-ROM (Compact Disk-Read Only Memory) can be configured to store this program.

[0141] In the embodiments described so far, for example, the hardware structure of the processing unit that executes various processes of the processor device 16 is the following various processors: The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes such as an ASIC (Application Specific Integrated Circuit).

[0142] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a form in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers such as servers and clients, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are configured with one or more processors of each type as a hardware structure.

[0143] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0144] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0145] 10 Endoscopy System 12 Endoscopy 12A Insertion section 12B Operation section 12C curved section 12D tip 12E angle knob 13 Mode switch 14 Light source device 16 Processor Unit 18 Display 20 Input section 22A First laser light source 22B Second laser light source 24 Light source control unit 28A Optical Fiber 28B optical fiber 30 Phosphor 32 Diffusion material 34 Imaging lens 36 image sensor 38 Analog-to-digital converter 40 Shooting control unit 42 Image processing section 44 Image acquisition unit 46 Lesion area detection unit 46A First detection unit 46B Second detection unit 46C Third detection unit 46D 4th detection unit 46E 5th detection unit 46F 6th detection unit 46G 7th detector 46H 8th detector 48 Location information acquisition unit 50 Position recognition part 52 Selection section 54 Lesion area detection control unit 56 Medical image processing equipment 58 Display control unit 60 Memory control unit 62 Memory section 70 Endoscopy System 100 Ultrasound diagnostic equipment 102 Ultrasound probe 104 Transmission and reception control section 106 Transmitter 108 Receiving unit 110 Image processing section G E image G S Schematic diagram L0 irradiation light L1 light L 11 Excitation light L 12 laser light L2 light S1~S6 Processing of image diagnostic methods

Claims

1. an endoscope having an insertion portion to be inserted into a body cavity of a subject; a display unit for displaying an image; a medical image processing device; An endoscope system that sequentially captures images of the inside of a living body of a subject and displays them in real time, The medical image processing device includes: an image acquisition unit that acquires images of a plurality of positions in the living body of the subject from an image sensor disposed on the distal end side of the insertion portion of the endoscope; a position information acquisition unit that recognizes position information indicating a position in the living body of the acquired image from the acquired image; a plurality of region-of-interest detection units that detect regions of interest from the acquired image, the plurality of region-of-interest detection units corresponding to a plurality of positions in the living body; a selection unit that selects, from the plurality of attention area detection units, an attention area detection unit that corresponds to a position indicated by the position information recognized by the position information acquisition unit; a control unit that causes the selected attention area detection unit to detect an attention area from the acquired image; a display control unit that causes the display unit to display the position indicated by the position information recognized by the position information acquisition unit and the selected attention area detected by the attention area detection unit; and Equipped with the plurality of attention area detection units are a plurality of trained models, The plurality of trained models are models trained using different datasets, The display control unit displays the acquired image on the display unit in real time, and further superimposes the selected area of ​​interest detected by the area of ​​interest detection unit on the image displayed in real time.

2. The endoscope system according to claim 1 , wherein the position information acquisition unit detects characteristic landmarks at each position within the living body, and estimates the position within the living body from information about the detected landmarks.

3. The endoscope system according to claim 1 or 2, wherein the plurality of trained models are models trained using images of mucous membranes at respective positions within the living body.

4. The endoscope system according to claim 1 , wherein the image acquisition unit sequentially acquires a plurality of images obtained by capturing images of the inside of the living body in time series.

5. The endoscope system according to claim 4 , wherein the image acquisition unit sequentially acquires a plurality of images photographed at a constant frame rate.

6. The endoscope system according to claim 1 , wherein the plurality of trained models are models trained using a dataset consisting of images taken at different positions within a living body.

7. The endoscopic system according to any one of claims 1 to 6, wherein the position information acquisition unit includes an input unit that accepts input from a user to correct the recognized position within the living body and the position information within the living body displayed on the display unit.

8. The display control unit displaying a schematic diagram on the display unit; displaying a graphic at a position indicated by the acquired position information on the schematic diagram; The endoscope system according to claim 1 .

9. The display control unit displays the acquired image. The endoscope system according to claim 8 .

10. The schematic diagram is a diagram of a lumen. The endoscope system according to claim 8 or 9.

11. The endoscope system according to claim 1 , wherein the position information acquisition unit includes an input unit that receives the in-vivo position information as an input from a user.

12. The endoscope is inserted through the mouth or nose of the subject, The endoscope system according to claim 1 , wherein the position information acquisition unit acquires position information indicating at least one of the positions of the pharynx, the esophagus, the stomach, and the duodenum.

13. The endoscope is inserted from the anus of the subject, The endoscopic system according to any one of claims 1 to 11, wherein the position information acquisition unit acquires position information indicating at least one of the positions of the rectum, sigmoid colon, descending colon, transverse colon, ascending colon, cecum, ileum, and jejunum.

14. The endoscope system according to claim 1 , wherein the position information acquisition unit acquires the position information from an endoscope insertion shape observation device, or acquires the position information by irradiating the subject with X-rays from outside the body of the subject.

15. The endoscope system according to claim 1 , further comprising a storage control unit that stores the acquired image and the position information of the acquired image in association with each other.

16. The endoscope system according to claim 1 , wherein the trained model detects a lesion area as a region of interest.

17. The endoscope system of claim 16, wherein the lesion area includes at least one of a polyp, cancer, inflammation, and vascular atypia.

18. A method for operating a medical image processing device that sequentially captures images of the inside of a subject's body and displays them in real time, comprising: an image acquiring step in which an image acquiring unit acquires images of a plurality of positions in a living body of a subject from an imaging element disposed at a distal end of an insertion portion of an endoscope having an insertion portion that is inserted into a body cavity of the subject; a position information acquiring step in which a position information acquiring unit recognizes position information indicating a position of the acquired image within the living body from the acquired image; an attention area detection step in which a plurality of attention area detection units corresponding to a plurality of positions in the living body detect attention areas from the acquired image; a selection step in which a selection unit selects, from the plurality of attention area detection units, an attention area detection unit corresponding to a position indicated by the position information recognized in the position information acquisition step; a control step in which a control unit causes the selected attention area detection unit to detect an attention area from the acquired image; a display control step of causing a display control unit to display on a display unit the position indicated by the position information recognized in the position information acquisition step and the attention area detected by the selected attention area detection unit; Equipped with the plurality of attention area detection units are a plurality of trained models, The plurality of trained models are models trained using different datasets, The display control step is a method for operating a medical image processing device, in which the display control unit displays the acquired image on the display unit in real time, and further superimposes the area of ​​interest detected by the selected area of ​​interest detection unit on the image displayed in real time.

19. A program for sequentially capturing images of the inside of a subject's body and displaying them in real time, an image acquisition step of acquiring images of a plurality of positions in a living body of a subject from an imaging element disposed at a distal end of an insertion portion of an endoscope having an insertion portion that is inserted into a body cavity of the subject; a position information acquiring step of recognizing, from the acquired image, position information indicating a position within the living body of the acquired image; an attention area detection step of detecting attention areas from the acquired image by a plurality of attention area detection units respectively corresponding to a plurality of positions in the living body; a selection step of selecting, from the plurality of attention area detection units, an attention area detection unit corresponding to the position indicated by the position information recognized in the position information acquisition step; a control step of causing the selected attention area detection unit to detect an attention area from the acquired image; a display control step of displaying on a display unit the position indicated by the position information recognized in the position information acquisition step and the selected attention area detected by the attention area detection unit; on the computer, the plurality of attention area detection units are a plurality of trained models, The plurality of trained models are models trained using different datasets, The display control process is a program that displays the acquired image on the display unit in real time, and further superimposes the attention area detected by the selected attention area detection unit on the image displayed in real time.

20. a non-transitory computer-readable recording medium that sequentially captures and displays in-vivo images of a subject in real time when instructions stored in the recording medium are read by a computer; an image acquisition step of acquiring images of a plurality of positions in a living body of a subject from an imaging element disposed at a distal end of an insertion portion of an endoscope having an insertion portion that is inserted into a body cavity of the subject; a position information acquiring step of recognizing, from the acquired image, position information indicating a position within the living body of the acquired image; an attention area detection step of detecting attention areas from the acquired image by a plurality of attention area detection units respectively corresponding to a plurality of positions in the living body; a selection step of selecting, from the plurality of attention area detection units, an attention area detection unit corresponding to the position indicated by the position information recognized in the position information acquisition step; a control step of causing the selected attention area detection unit to detect an attention area from the acquired image; a display control step of displaying on a display unit the position indicated by the position information recognized in the position information acquisition step and the selected attention area detected by the attention area detection unit; on the computer, the plurality of attention area detection units are a plurality of trained models, The plurality of trained models are models trained using different datasets, The display control process displays the acquired image on the display unit in real time, and further displays the selected attention area detected by the attention area detection unit superimposed on the image displayed in real time.

Images