LEARNING DEVICE, LEARNING METHOD, PROGRAM, TRAINED MODEL, AND ENDOSCOPE SYSTEM

Abstract

Provided are a learning device which carries out effective learning using high accuracy correct answer data, a learning method, a program, a learned model, and an endoscope system. A learning device 14 comprises a recognizer composed of a neural network and a processor, wherein the processor (CPU 41) acquires a learning image 53 of a subject being examined and biological examination information which is information associated with the learning image 53 and indicating a position on which biological examination has been performed in the subject being examined, creates correct answer region data which indicates a region including the position on which biological examination has been performed as a correct answer region on the basis of the biological examination information, and carries out learning of the recognizer which recognizes a region of interest using the learning image 53 and the correct answer region data.

Description

[Technical field]

[0001] The present invention relates to a learning device, a learning method, a program, a trained model, and an endoscope system. [Background technology]

[0002] Functions to support examinations and diagnoses have been proposed, such as a function to detect regions of interest such as lesions from medical images of a part of a patient's body, and a function to differentiate malignancies from medical images. It is expected that the use of such support functions will prevent lesions from being overlooked and reduce the burden on doctors, etc.

[0003] To realize the above-mentioned support function, a recognizer using machine learning is used. Specifically, a medical image is input to a recognizer that has undergone machine learning, and the recognizer is made to detect areas of interest such as lesions and to distinguish the malignancy of the lesions.

[0004] Here, in order to train a recognizer by machine learning, it is known that the accuracy of the correct answer data at the time of training has a large effect on the accuracy of the recognizer after training. In other words, when training a recognizer, it is necessary to prepare highly accurate correct answer data in order to obtain a highly accurate recognizer. Therefore, techniques aimed at obtaining highly accurate correct answer data have been proposed so far.

[0005] For example, in the technique described in Patent Document 1, a measurement target area that has been corrected in accordance with a user's instruction is learned as correct data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2019 / 146356 Summary of the Invention [Problem to be solved by the invention]

[0007] Even when professional users such as doctors create correct answer data for medical images, the definition of the correct answer data may vary depending on the user. For example, when multiple doctors each specify a region of interest for the same medical image, the regions of interest do not necessarily match. Therefore, the accuracy of the correct answer data is not necessarily high, and even if a recognizer is trained based on such correct answer data, it is difficult to train the recognizer effectively.

[0008] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a learning device, a learning method, a program, a trained model, and an endoscopic system that perform effective learning by utilizing highly accurate correct answer data. [Means for solving the problem]

[0009] A learning device, which is one aspect of the present invention for achieving the above-mentioned object, is a learning device having a recognizer composed of a neural network and a processor, in which the processor acquires learning images of a subject and biopsy information, which is information associated with the learning images and indicates the location where a biopsy was performed on the subject, generates correct answer region data based on the biopsy information, with the region including the location where the biopsy was performed being the correct answer region, and trains a recognizer that recognizes the region of interest using the learning images and the correct answer region data.

[0010] According to this aspect, the correct answer region data is generated with the region including the location where the biopsy was performed as the correct answer region. Then, learning of the attention region is performed based on the correct answer region data. As a result, since the correct answer region data is generated with the location where the biopsy was actually performed as the attention region at the time of examination or diagnosis, effective learning can be performed using the highly accurate correct answer region data. Note that the highly accurate correct answer region data refers to the correct answer region data that clearly shows the accurate attention region in the learning image. The location where the biopsy was performed is the location where the doctor actually paid attention and determined that further examination (biopsy) is necessary, so that the accuracy as the attention region is high, and by generating the correct answer region data based on the location where the biopsy was performed, highly accurate correct answer region data can be generated. Also, the attention region is the region that the doctor or the like should pay attention to when performing an examination using the endoscope system. For example, the attention region is a location suspected as a lesion or a location where a normal lesion appears.

[0011] Preferably, the biopsy information is image data indicating the position where the biopsy was performed, or position data indicating the position where the biopsy was performed.

[0012] Preferably, when the biopsy information indicates that multiple biopsies were performed, the processor creates correct area data based on at least one of the multiple biopsy locations.

[0013] Preferably, when the biopsy information indicates that a plurality of locations have been subjected to biopsy, the processor creates the correct region data based on a region including the locations where the plurality of biopsies have been subjected to biopsy.

[0014] Preferably, the processor acquires correct answer region candidate data paired with the training image, creates correct answer region data based on the biometric information and the correct answer region candidate data, and trains the recognizer using the training image and the correct answer region data.

[0015] Preferably, the processor calculates a region weight, which is the weight of the region based on the correct region candidate data, from the degree of similarity between the region based on the biopsy information and the region based on the correct region candidate data, and trains the recognizer using the training image, the correct region data, and the region weight.

[0016] Preferably, the processor performs training using the region weights for a portion of the training period.

[0017] Preferably, the processor performs training using the region weights for the entire training period.

[0018] Another aspect of the present invention is a learning method for a learning device having a recognizer composed of a neural network and a processor, and includes the steps of: acquiring a learning image of a subject and biopsy information, which is information associated with the learning image and indicates the location where a biopsy was performed on the subject; generating correct area data based on the biopsy information, in which an area including the location where the biopsy was performed is designated as the correct area; and training a recognizer that recognizes the area of ​​interest using the learning image and the correct area data.

[0019] Another aspect of the present invention is a program for causing a learning device having a recognizer composed of a neural network and a processor to perform a learning method, the program including the steps of: in the processor, acquiring a learning image of a test subject and biopsy information, which is information associated with the learning image and indicates the location where a biopsy was performed on the test subject; generating correct area data based on the biopsy information, in which an area including the location where the biopsy was performed is designated as the correct area; and training a recognizer that recognizes the area of ​​interest using the learning image and the correct area data.

[0020] The trained model of the recognizer, which is another aspect of the present invention, is obtained by the training method described above.

[0021] Another aspect of the present invention is an endoscope system equipped with a trained model of the above-mentioned recognizer. Effect of the Invention

[0022] According to the present invention, correct area data is generated by setting the area including the location where the biopsy was performed as the correct area, and learning of the area of ​​interest is performed based on that correct area data.Therefore, correct area data is generated for the area where the biopsy was actually performed as the area of ​​interest during examination or diagnosis, and effective learning can be performed using highly accurate correct area data. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a schematic diagram showing the overall configuration of an endoscope system. [Diagram 2] FIG. 2 is a block diagram showing the learning device of the first embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a learning image. [Figure 4] FIG. 4 is a diagram showing an example of biopsy information. [Diagram 5] FIG. 5 is a diagram illustrating the processing performed by the CPU. [Figure 6] FIG. 6 is a diagram showing an example of the correct area data. [Figure 7] FIG. 7 is a diagram illustrating an embodiment of a recognizer. [Figure 8] FIG. 8 is a flow diagram showing a learning method using the learning device. [Figure 9] FIG. 9 is a diagram showing the correct area data. [Figure 10] FIG. 10 is a diagram showing the correct area data. [Figure 11] FIG. 11 is a block diagram showing a learning device. [Figure 12] FIG. 12 is a diagram showing an example of the correct area candidate data. [Figure 13] FIG. 13 is a diagram for explaining generation of correct area data. [Figure 14] FIG. 14 is a diagram showing an example of the correct area data. [Figure 15] FIG. 15 is a flow diagram showing the learning method. [Figure 16] FIG. 16 is a diagram for explaining the application of region weights to candidate regions based on correct candidate region data. [Figure 17] FIG. 17 is a flow diagram showing the learning method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, preferred embodiments of a learning device, a learning method, a program, a trained model, and an endoscope system according to the present invention will be described with reference to the accompanying drawings.

[0025] FIG. 1 is a schematic diagram showing the overall configuration of an endoscope system for acquiring learning images (medical images) to be input to a learning device according to the present invention.

[0026] As shown in FIG. 1, an endoscope system 9 includes an endoscope scope 10 which is an electronic endoscope, a light source device 11, an endoscope processor device 12, a display device 13, a learning device 14, an operation unit 15, and a display device 16.

[0027] The endoscope 10 is a device for capturing time-series medical images including an image of a subject, and is, for example, a flexible endoscope. The medical images captured by the endoscope 10 are used as learning images, which will be described later. The endoscope 10 has an insertion section 20 that is inserted into a subject and has a tip and a base end, a handheld operation section 21 that is connected to the base end side of the insertion section 20 and is held by the surgeon to perform various operations, and a universal cord 22 that is connected to the handheld operation section 21.

[0028] The insertion section 20 is formed in a long shape with a small diameter as a whole. The insertion section 20 is configured by sequentially connecting a soft section 25 having flexibility from its base end side to its tip end side, a bending section 26 that can be bent by operating the hand-held operation section 21, and a tip section 27 that has an imaging optical system (objective lens) and an imaging element 28 (not shown) built in.

[0029] The imaging element 28 is a CMOS (complementary metal oxide semiconductor) type or CCD (charge coupled device) type imaging element. Image light of the observed site is incident on the imaging surface of the imaging element 28 via an observation window (not shown) opened on the tip surface of the tip portion 27 and an objective lens (not shown) arranged behind the observation window. The imaging element 28 captures (converts into an electrical signal) the image light of the observed site incident on its imaging surface and outputs an imaging signal.

[0030] The handheld operation section 21 is provided with various operation members operated by the surgeon. Specifically, the handheld operation section 21 is provided with two types of bending operation knobs 29 used for bending the bending section 26, an air / water supply button 30 for air / water supply operation, and a suction button 31 for suction operation. The handheld operation section 21 is also provided with a still image capture instruction section 32 for issuing an instruction to capture a still image 39 of the observation site, and a treatment tool introduction port 33 for inserting a treatment tool (not shown) into a treatment tool insertion passage (not shown) passing through the insertion section 20.

[0031] The universal cord 22 is a connection cord for connecting the endoscope 10 to the light source device 11. The universal cord 22 contains a light guide 35, a signal cable 36, and a fluid tube (not shown) that are inserted through the insertion section 20. In addition, at the end of the universal cord 22, a connector 37a that is connected to the light source device 11, and a connector 37b that branches off from the connector 37a and is connected to the endoscope processor device 12 are provided.

[0032] By connecting the connector 37a to the light source device 11, the light guide 35 and the fluid tube (not shown) are inserted into the light source device 11. As a result, the necessary illumination light, water, and gas are supplied from the light source device 11 to the endoscope 10 via the light guide 35 and the fluid tube (not shown). As a result, the illumination light is irradiated toward the observation site from an illumination window (not shown) on the tip surface of the tip portion 27. In addition, in response to the pressing operation of the air / water supply button 30 described above, gas or water is sprayed from an air / water supply nozzle (not shown) on the tip surface of the tip portion 27 toward an observation window (not shown) on the tip surface.

[0033] Connecting the connector 37b to the endoscope processor device 12 electrically connects the signal cable 36 to the endoscope processor device 12. As a result, an image pickup signal of the observation site (inspection subject) is output from the imaging element 28 of the endoscope scope 10 to the endoscope processor device 12 via the signal cable 36, and a control signal is output from the endoscope processor device 12 to the endoscope scope 10.

[0034] The light source device 11 supplies illumination light to the light guide 35 of the endoscope 10 via the connector 37a. The illumination light is selected from various wavelength bands according to the observation purpose, such as white light (light in a white wavelength band or light in multiple wavelength bands), light in one or multiple specific wavelength bands, or a combination of these. The specific wavelength band is a band narrower than the white wavelength band.

[0035] A first example of the specific wavelength band is, for example, a blue band or a green band in the visible range. The first example wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the first example light has a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm.

[0036] A second example of the specific wavelength band is, for example, the red band of the visible range, and includes a wavelength band of 585 nm to 615 nm or a wavelength band of 610 nm to 730 nm, and the light of the second example has a peak wavelength within the wavelength band of 585 nm to 615 nm or a wavelength band of 610 nm to 730 nm.

[0037] A third example of the specific wavelength band includes a wavelength band where the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and the light of the third example has a peak wavelength in the wavelength band where the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different. The wavelength band of this third example includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example has a peak wavelength in the above-mentioned wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

[0038] A fourth example of the specific wavelength band is the wavelength band (390 nm to 470 nm) of excitation light that is used for observing fluorescence emitted by fluorescent substances in living organisms (fluorescence observation) and excites these fluorescent substances.

[0039] A fifth example of the specific wavelength band is an infrared light wavelength band, which includes a wavelength band of 790 nm to 820 nm or a wavelength band of 905 nm to 970 nm, and the fifth example light has a peak wavelength in the wavelength band of 790 nm to 820 nm or a wavelength band of 905 nm to 970 nm.

[0040] The endoscope processor device 12 controls the operation of the endoscope 10 via the connector 37b and the signal cable 36. The endoscope processor device 12 also generates an image (also called a "video 38") consisting of time-series frame images 38a including a subject image, based on an imaging signal acquired from the imaging element 28 of the endoscope 10 via the connector 37b and the signal cable 36. Furthermore, when the still image shooting instruction unit 32 is operated on the handheld operation unit 21 of the endoscope 10, the endoscope processor device 12, in parallel with the generation of the video 38, turns one frame image in the video 38 into a still image 39 corresponding to the timing of the shooting instruction.

[0041] The moving image 38 and the still image 39 are medical images captured inside the subject, i.e., inside the living body. Furthermore, when the moving image 38 and the still image 39 are images obtained using light (special light) in the above-mentioned specific wavelength band, both are special light images. The endoscope processor device 12 then outputs the generated moving image 38 and still image 39 to the display device 13 and the learning device 14, respectively.

[0042] The endoscope processor device 12 may generate (acquire) a special light image having information of the specific wavelength band based on the normal light image obtained by the white light. In this case, the endoscope processor device 12 functions as a special light image acquisition unit. The endoscope processor device 12 obtains the signal of the specific wavelength band by performing a calculation based on color information of red, green, and blue [RGB (Red, Green, Blue)] or cyan, magenta, and yellow [CMY (Cyan, Magenta, Yellow)] contained in the normal light image.

[0043] The endoscope processor device 12 may generate a feature image such as a publicly known oxygen saturation image based on at least one of the normal light image obtained by the above-mentioned white light and the special light image obtained by the above-mentioned light of the specific wavelength band (special light). In this case, the endoscope processor device 12 functions as a feature image generating unit. Note that the above-mentioned in-vivo image, normal light image, special light image, and video 38 or still image 39 including the feature image are all medical images obtained by imaging or measuring the human body for the purpose of image-based diagnosis or examination.

[0044] The display device 13 is connected to the endoscope processor device 12, and functions as a display unit that displays a moving image 38 and a still image 39 input from the endoscope processor device 12. A user (doctor) performs operations such as moving forward and backward of the insertion portion 20 while checking the moving image 38 displayed on the display device 13, and when a lesion or the like is found in the observed region, the user operates the still image capturing instruction unit 32 to capture a still image of the observed region, and also performs a diagnosis, a biopsy, or the like.

[0045] The learning device 14 is composed of a computer. The operation unit 15 uses a keyboard, mouse, etc. that are connected to the computer by wire or wirelessly, and the display device (display unit) 16 uses various monitors such as a liquid crystal monitor that can be connected to the computer.

[0046] <First embodiment> FIG. 2 is a block diagram showing the learning device 14 of the first embodiment.

[0047] 2 is mainly composed of an information acquisition unit 40, a CPU (Central Processing Unit) (processor) 41, a recognizer 43, a display control unit 46, and a storage unit 48. The display control unit 46 generates image data for display and outputs it to the display device 16. The storage unit 48 includes a storage unit 48 as a working area for the CPU 41, and a storage unit 48 that stores various programs such as an operating system and a medical image processing program.

[0048] The information acquiring unit 40 acquires information used for training the recognizer 43. Specifically, the information acquiring unit 40 acquires training images 53 and biopsy information. The training images 53 are medical images captured by the endoscope system 9. The information acquiring unit 40 acquires training images 53 stored in a training image database (not shown). The biopsy information is information associated with the training images 53 and indicates the location where the biopsy of the subject was performed. The information acquiring unit 40 acquires the biopsy information stored in a biopsy information database 51. Here, the training image database and the biopsy information database 51 may be provided outside the training device 14 as shown in the figure, or may be provided inside the training device 14.

[0049] FIG. 3 is a diagram showing an example of a learning image.

[0050] The learning image 53 is acquired by the endoscope system 9. For example, the learning image 53 is a medical image of a part of a human body to be inspected. The learning image 53 is an image captured by inserting the insertion portion 20 of the endoscope 10 into the large intestine C.

[0051] FIG. 4 is a diagram showing an example of biopsy information.

[0052] In the biopsy information 55, the location B1 where the biopsy was performed is shown in the learning image 53. Thus, an example of the biopsy information is image data in which the location B1 indicating the location where the biopsy was performed is added to the learning image 53. Another example of the biopsy information may be position data related to coordinates indicating the location of the location B1 where the biopsy was performed. The coordinates indicating the location of the location B1 are associated with the learning image 53. The above-mentioned example of the biopsy information is a specific example, and the biopsy information is not particularly limited as long as it is information indicating the location where the biopsy was performed (for example, the location where a part of the human body was removed with forceps, etc.). Here, the location B1 where the biopsy was performed, which is included in the biopsy information 55, is the location where the doctor performed further examination (biopsy) in the actual diagnosis. Therefore, the area including the location B1 where the biopsy was performed is an area that should be noted. Also, the attention area is an area that the doctor, etc. should pay attention to when performing an examination using the endoscope system 9. For example, the attention area is an area suspected to be a lesion or an area where a normal lesion appears.

[0053] 2, the CPU 41 operates based on a program stored in the storage unit 48, and controls the information acquisition unit 40, the recognizer 43, and the display control unit 46, and also functions as a part of each of these units. The CPU 41 also controls the learning of the recognizer 43, which will be described in detail later.

[0054] FIG. 5 is a diagram for explaining the processing performed by the CPU 41.

[0055] The CPU 41 performs a correct area data generation process 45 and a learning process 47 .

[0056] The correct region data generation process 45 generates correct region data, which uses the learning image 53 acquired by the information acquisition unit 40 and the biopsy information, and sets the region including the location B1 where the biopsy was performed as the correct region.

[0057] FIG. 6 is a diagram showing an example of the correct area data.

[0058] 6 shows region R1 including location B1 where the biopsy was performed as the correct region in association with learning image 53. Note that the subject in learning image 53 is omitted.

[0059] The correct region R2 is an area enlarged at a predetermined magnification with the biopsy site B1 at its center. The biopsy site B1 may be, for example, a part of a lesion. Therefore, in order to observe the entire lesion or the lesion more widely, it is necessary to set an area larger than the biopsy site B1 as the area of ​​interest. Therefore, the correct region R2 is an area enlarged at a predetermined magnification with the biopsy site B1 at its center. The predetermined magnification is appropriately determined depending on the part of the subject and the user's settings. In addition, if it is sufficient to set the biopsy site B1 as the area of ​​interest, the correct region R2 and the biopsy site B1 may be the same area.

[0060] 5, the CPU 41 performs a learning process 47 for the recognizer 43. Specifically, the learning process 47 for the CPU 41 performs learning of the recognizer 43 that recognizes the region of interest using the learning image 53 and the correct region data.

[0061] FIG. 7 is a diagram for explaining an embodiment of the recognizer 43, which is a learning model configured by a neural network.

[0062] The recognizer 43 receives a learning image 53 and outputs an estimated image 50 obtained by estimating a region of interest in the learning image 53. The learning processing unit 122 in which the learning process 47 is performed includes a loss value calculation unit 54 and a parameter control unit 56.

[0063] The recognizer 43 uses a convolutional neural network (CNN), which is one of the deep learning models.

[0064] The recognizer 43 has a multi-layer structure and holds a multi-weight parameter. The recognizer 43 changes from an unlearned model to a trained model by updating the weight parameter from the initial value to an optimal value. The initial value of the weight parameter of the recognizer 43 may be any value, or, for example, the weight parameter of an image-based trained model that recognizes images may be applied.

[0065] This recognizer 43 includes an input layer 52A, an intermediate layer 52B having multiple sets of convolutional layers and pooling layers, and an output layer 52C, and each layer has a structure in which multiple “nodes” are connected by “edges.”

[0066] In the case shown in FIG. 7, a learning image 53 is input to the input layer 52A. The intermediate layer 52B has a convolution layer, a pooling layer, and the like, and is a part that extracts features from the image input from the input layer 52A. The convolution layer filters nearby nodes in the previous layer (performing a convolution operation using a filter) to obtain a "feature map." The pooling layer reduces the feature map output from the convolution layer to create a new feature map. The "convolution layer" plays a role in extracting features such as edge extraction from an image, and the "pooling layer" plays a role in providing robustness so that the extracted features are not affected by translation, etc. Note that the intermediate layer 52B is not limited to cases where convolution layers and pooling layers are alternately arranged, but also includes cases where convolution layers are continuous and normalization layers. In addition, the final convolution layer conv is a part that outputs a feature map (image) of the same size as the learning image 53, which is the input image, and indicates a region of interest.

[0067] The output layer 52C is a section that outputs the detection result of the recognizer 43 (in this embodiment, an image showing the region of interest).

[0068] The loss value calculation unit 54 obtains the detection result (estimated image 50) output from the output layer 52C of the recognizer 43 and the correct answer region data 57 corresponding to the input image (learning image 53), and calculates the loss value between them. Specifically, the difference between the attention region output in the estimated image 50 and the correct answer region in the correct answer region data 57 is set as the loss value. The loss value may be calculated using, for example, the Jaccard coefficient or the Dice coefficient.

[0069] Based on the loss value calculated by the loss value calculation unit 54, the parameter control unit 56 adjusts the weight parameters of the recognizer 43 by the backpropagation method to minimize the distance in the feature space between the estimated image 50 and the correct region data 57 or to maximize the similarity.

[0070] This parameter adjustment process is repeated, and learning is repeated until the loss value calculated by the loss value calculation unit 54 converges.

[0071] In this manner, the learning images 53 and the corresponding correct region data 57 are used to proceed with learning, thereby creating a recognizer 43 having a trained learning model with optimized weight parameters.

[0072] FIG. 8 is a flow diagram showing a learning method using the learning device 14.

[0073] First, the learning device 14 acquires a learning image 53 by the information acquisition unit 40 (step S10). After that, the learning device 14 acquires biopsy information 55 by the information acquisition unit 40 (step S11). Next, the learning device 14 generates correct region data 57 by the correct region data generation process 45 performed by the CPU 41 (step S12). After that, the learning device 14 inputs the learning image 53 to the recognizer 43, and outputs an estimated image 50 having a recognition result of the region of interest. Then, the CPU 41 performs learning so that the difference between the recognition result output by the recognizer 43 and the correct region data 57 is minimized (step S13).

[0074] As described above, in this embodiment, biometric information is used to generate highly accurate correct region data 57. Then, learning image 53 and highly accurate correct region data 57 are used to train recognizer 43, so that more effective training can be performed.

[0075] <Modification 1 of the First Embodiment> Next, a first modified example of the first embodiment will be described. In this example, when there are multiple locations where a biopsy has been performed, the correct region data generation process 45 of the CPU 41 generates correct region data having one correct region.

[0076] FIG. 9 is a diagram showing the correct region data of this example.

[0077] Based on the biopsy information of this example, image data showing the positions of the three locations where the biopsy was performed or position data showing the positions are acquired. Specifically, as shown in FIG. 9, the locations where the biopsy was performed are location B3, location B5, and location B7, and these biopsies were performed on the lesion D1. In this case, the biopsy information also includes information that the biopsy was performed on the lesion D1. Then, in the correct region data generation process 45 of the CPU 41, correct region data 57A showing the correct region R3 including the locations B3, B5, and B7 where the biopsy was performed is generated. The correct region data 57A corresponds to the learning image 53A.

[0078] In this way, when biopsies are performed on the same affected area, or when biopsies are performed on adjacent areas, a correct answer region including all of the biopsy-performed areas is generated. This makes it possible to generate correct answer region data 57A that appropriately sets the periphery of the biopsy-performed area as the attention region, and thus makes it possible to generate highly accurate correct answer region data 57A.

[0079] <Modification 2 of the First Embodiment> Next, a second modified example of the first embodiment will be described. In this example, when there are multiple locations where biopsies have been performed, the correct region data generation process 45 of the CPU 41 generates correct region data based on at least one of the locations where the multiple biopsies have been performed.

[0080] FIG. 10 is a diagram showing the correct region data of this example.

[0081] Based on the biometric information of this example, image data showing the positions of three locations where biometric testing was performed or position data showing the positions are acquired. Specifically, as shown in FIG. 10, the locations are B9, B11, and B13 where biometric testing was performed. Then, in the correct answer region data generation process 45 of the CPU 41, correct answer regions are set for each of the locations B9, B11, and B13 where biometric testing was performed. Specifically, a correct answer region R5 is generated corresponding to the location B9, a correct answer region R7 is generated corresponding to the location B11, and a correct answer region R9 is generated corresponding to the location B13. In this way, when biometric testing was performed at multiple locations and the locations are scattered or the biometric testing was not performed at the same lesion, a correct answer region is set corresponding to each location. This makes it possible to generate correct answer region data 57C in which the periphery of the location where biometric testing was performed is appropriately set as the attention region, and it is possible to generate highly accurate correct answer region data 57C. The correct answer region data 57C is generated corresponding to the learning image 53B. In addition, in the above example, correct answer areas are set for all locations where biopsy was performed, but if there are multiple locations where biopsy was performed, correct answer area data only needs to be generated based on at least one of them.

[0082] <Second embodiment> Next, a second embodiment will be described. In this embodiment, correct region candidate data is acquired in addition to biopsy information, and correct region data is generated.

[0083] Fig. 11 is a block diagram showing the learning device 14 of the second embodiment. Note that the same reference numerals are used for the parts that have already been explained in Fig. 2, and the explanation will be omitted.

[0084] The learning device 14 shown in FIG. 11 is mainly composed of an information acquisition unit 40, a CPU 41, a recognizer 43, a display control unit 46, and a storage unit 48.

[0085] The information acquiring unit 40 acquires the learning image 53, the biopsy information, and the correct answer region candidate data. The information acquiring unit 40 acquires the correct answer region candidate data from the correct answer region candidate database 58. The correct answer region candidate database 58 stores the correct answer region candidate data. The correct answer region candidate database 58 may be provided outside the learning device 14 as shown in the figure, or may be provided inside the learning device 14.

[0086] FIG. 12 is a diagram showing an example of the correct area candidate data.

[0087] In the correct answer region candidate data 70, each of the doctors (Doctors) 1 to 4 indicates a region of interest in a learning image in which the large intestine C is photographed. The regions of interest indicated by the doctors 1 to 4 are indicated by different line types. The region indicated by the doctor 1 is an area surrounded by a solid line, the region indicated by the doctor 2 is an area surrounded by a dotted line, the region indicated by the doctor 3 is an area surrounded by a dashed line, and the region indicated by the doctor 4 is an area surrounded by a finer dotted line. The regions indicated by the doctors 1 to 4 include some overlapping regions and some non-overlapping regions. In this way, when a plurality of doctors are made to specify a region of interest based on the same medical image (learning image), there is variation in the region of interest specified by each doctor. Thus, even if the recognizer 43 is made to learn only the correct answer region candidate data 70 having the variation as the correct answer region data, the accuracy of the correct answer region data is not high, and effective learning cannot be performed. Therefore, in this embodiment, highly accurate correct region data is generated based on the correct region candidate data 70 and biopsy information, thereby achieving effective learning.

[0088] FIG. 13 is a diagram for explaining generation of correct area data.

[0089] The correct answer region data 66 is generated by the correct answer region data generation process 45 performed by the CPU 41 based on the biopsy information 62 and the correct answer region candidate data 70. The correct answer region data generation process 45 generates the correct answer region data by various methods. For example, correct answer region data is generated in which the area where the biopsy was performed, which is indicated in the biopsy information 62, is enlarged based on the correct answer region candidate data 70 as the correct answer region. In addition, for example, the area where the biopsy was performed, which is indicated in the biopsy information, and all the areas indicated in the correct answer region candidate data 70 overlap may be set as the correct answer region data.

[0090] FIG. 14 is a diagram showing an example of the correct area data.

[0091] The correct answer region data 66A shows a correct answer region R21 obtained by enlarging the location B15 where the biopsy was performed in the biopsy information 62 based on the correct answer region candidate data 70. The correct answer region R21 is a region that is enlarged based on the correct answer region candidate data 70, with the location B15 where the biopsy was performed at the center. The correct answer region data 66A having such a correct answer region R21 is highly accurate, and more effective learning can be achieved by using the correct answer region data 66A for learning.

[0092] The correct answer region data 66B has a correct answer region R22 which is an area where the location B15 where the biopsy was performed in the biopsy information 62 overlaps with all areas of the correct answer region candidate data 70. The correct answer region data 66B having such a correct answer region R22 is highly accurate, and more effective learning can be achieved by using the correct answer region data 66B for learning.

[0093] FIG. 15 is a flow diagram showing a learning method using the learning device 14 of this embodiment.

[0094] First, the learning device 14 acquires the learning image 53 by the information acquisition unit 40 (step S20). After that, the learning device 14 acquires the biopsy information 62 by the information acquisition unit 40 (step S21). Next, the learning device 14 acquires the correct answer region candidate data 70 by the information acquisition unit 40 (step S22). After that, the learning device 14 generates the correct answer region data 66 by the correct answer region data generation process 45 performed by the CPU 41 (step S23). Then, the learning device 14 inputs the learning image 53 to the recognizer 43 and outputs the recognition result of the attention region. Next, the CPU 41 proceeds with learning so that the difference between the recognition result output by the recognizer 43 and the correct answer region data 66 is minimized (step S24).

[0095] As described above, according to this embodiment, the correct answer region data 66 is generated based on the biopsy information 62 and the correct answer region candidate data 70. The learning device 14 of this embodiment performs learning using the generated correct answer region data 66 and the learning image 53, and therefore can perform effective learning.

[0096] <Modification 1 of the second embodiment> Next, a first modified example of the second embodiment will be described. In this example, a correct region data generation process 45 of the CPU 41 calculates a region weight based on the correct region candidate data from the degree of coincidence between the location where the biopsy was performed based on the biopsy information and the region based on the correct region candidate data. Then, a learning process 47 of the CPU 41 learns the recognizer 43 using the learning image, the correct region data, and the region weight.

[0097] FIG. 16 is a diagram for explaining the application of region weights to candidate regions based on correct candidate region data.

[0098] In the example shown in Fig. 16, a location B17 where a biopsy was performed based on the biopsy information is shown. Also shown are candidate regions H1, H3, and H5 based on the correct candidate region data. Note that candidate regions H1, H3, and H5 are regions that different doctors indicated as attention regions for the same learning image.

[0099] In the correct region data generation process 45 of the CPU 41, a region weight is assigned according to the size of the portion overlapping with the biopsy site B17. In the case shown in FIG. 16, the site B17 overlaps with the candidate region H1 so as to be included therein, so the region weight of the candidate region H1 is set to the heaviest, a. Since half of the site B17 overlaps with the candidate region H5, the region weight of the candidate region H5 is set to b. Since the site B17 does not overlap with the candidate region H3, the region weight of the candidate region H3 is set to the lightest, c. The weights of the region weights are a>b>c. The site B17 where the biopsy was actually performed is highly worthy of being set as a region of interest, and it is also estimated that the candidate region having an overlapping region with the biopsy site B17 is also highly worthy of being set as a region of interest. Therefore, as described above, the region weights a, b, and c are assigned to each candidate region. In this way, in this example, the region weights are set and the correct region data is generated using the region weights. In addition, various methods can be adopted for generating the correct answer area data using the area weight. For example, the candidate areas H1, H3, and H5 may be expanded according to the area weight, and the area where the expanded candidate areas H1, H3, and H5 overlap may be determined as the correct answer area.

[0100] FIG. 17 is a flow diagram showing a learning method using the learning device 14 of this embodiment.

[0101] First, the learning device 14 acquires a learning image by the information acquisition unit 40 (step S30). After that, the learning device 14 acquires biopsy information by the information acquisition unit 40 (step S31). Next, the learning device 14 acquires correct region candidate data by the information acquisition unit 40 (step S32). Next, the learning device 14 acquires weights of each region of the correct region candidate data by the correct region data generation process 45 of the CPU 41 (step S33). After that, the learning device 14 generates correct region data by the correct region data generation process 45 performed by the CPU 41 (step S34). Then, the learning device 14 inputs the learning image to the recognizer 43 and outputs the recognition result of the attention region. Next, the CPU 41 proceeds with learning so that the difference between the recognition result output by the recognizer 43 and the correct region data is minimized (step S35). Note that the learning using the region weights performed by the CPU 41 may be performed for the entire learning period or for a part of the learning period.

[0102] As described above, in this example, a region weight is assigned to each candidate region in the correct region candidate data based on the location where the biopsy was performed, and the correct region data is generated based on the region weight. This makes it possible to generate highly accurate correct region data.

[0103] <Other> In the above embodiment, the endoscope processor device 12 and the learning device 14 are provided separately, but the endoscope processor device 12 and the learning device 14 may be integrated together. That is, the endoscope processor device 12 may be provided with the function of the learning device 14.

[0104] Furthermore, the hardware structure for executing various controls of the learning device 14 of the above embodiment including the recognizer is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various control units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0105] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple control units may be composed of one processor. As an example of configuring multiple control units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple control units. Second, as represented by a system on chip (SoC), there is a form using a processor that realizes the functions of the entire system including multiple control units with one IC (Integrated Circuit) chip. In this way, various control units are configured using one or more of the above various processors as a hardware structure.

[0106] The above-mentioned configurations and functions can be realized by any hardware, software, or a combination of both. For example, the present invention can be applied to a program that causes a computer to execute the above-mentioned processing steps (processing procedures), a computer-readable recording medium (non-transitory recording medium) on which such a program is recorded, or a computer on which such a program can be installed.

[0107] In addition, in the above embodiment, the medical images to be processed are time-series images or still images taken by the endoscope 10, but this is not limited to this, and the images may be medical images taken, for example, by an ultrasound diagnostic device, an X-ray device, a mammography examination, etc.

[0108] Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0109] 9: Endoscope system 10: Endoscope 11:Light source device 12: Endoscope processor device 13:Display device 14: Learning device 15:Operation section 16: Display 20: Insertion section 21: Handheld operation unit 22: Universal Code 25: Soft part 26: Curved section 27:Tip 28: Image sensor 29: Curvature control knob 30: Air / water supply button 31: Suction button 32: Still image shooting instruction section 33: Treatment tool inlet 35: Light guide 36: Signal cable 37a: Connector 37b : Connector 38:Video 38a: Frame image 39: Still image 40: Information acquisition department 41: CPU 43: Recognizer 46: Display control section 48: Storage section 50: Estimated image 51: Biopsy information database 52A: Input layer 52B: Middle layer 52C: Output layer 53: Learning image 54: Loss value calculation unit 55: Biological examination information 56: Parameter control section 58: Correct region candidate database

Claims

1. A learning device having a recognizer configured with a neural network and a processor, The processor, acquiring a learning image of an object to be inspected and biopsy information associated with the learning image, the biopsy information being information indicating a location of the object to be inspected; generating correct region data, based on the biopsy information, with a region including the site where the biopsy was performed as a correct region; A learning device that learns the recognizer that recognizes an attention area by using the learning image and the correct area data, The processor acquires correct answer region candidate data paired with the learning image, creates the correct answer region data based on the biopsy information and the correct answer region candidate data, and trains the recognizer using the learning image and the correct answer region data. The correct area candidate data is data indicating a plurality of the attention areas.

2. The learning device according to claim 1 , wherein the biopsy information is image data indicating a position where the biopsy was performed, or position data indicating a position where the biopsy was performed.

3. The learning device according to claim 1 or 2, wherein when the biopsy information includes multiple locations where the biopsy was performed, the processor creates the correct answer area data based on at least one of the multiple locations where the biopsy was performed.

4. The learning device according to any one of claims 1 to 3, wherein when the biopsy information includes multiple locations where the biopsy was performed, the processor creates the correct area data based on an area including the multiple locations where the biopsy was performed.

5. A learning device described in any one of claims 1 to 4, wherein data indicating multiple regions of interest in the correct answer region candidate data has regions that overlap with each other and regions that do not overlap with each other.

6. The learning device according to any one of claims 1 to 5, wherein the processor calculates a region weight, which is a weight of the region based on the correct region candidate data, from the degree of similarity between the region based on the biopsy information and the region based on the correct region candidate data, and trains the recognizer using the learning image, the correct region data, and the region weight.

7. The learning device according to claim 6 , wherein the processor performs learning using the region weights during a portion of a learning period.

8. The learning device according to claim 6 , wherein the processor performs learning using the region weights throughout the entire learning period.

9. A learning method for a learning device having a recognizer configured with a neural network and a processor, comprising: Executed by the processor, acquiring a learning image of an object to be inspected and biopsy information associated with the learning image, the biopsy information being information indicating a location of the object to be inspected; generating correct region data based on the biopsy information, the correct region being a region including the site where the biopsy was performed; A step of training the recognizer that recognizes an area of ​​interest using the learning image and the correct area data; A learning method comprising: The processor, In the acquiring step, correct region candidate data that is paired with the learning image is acquired; In the generating step, the correct region data is generated based on the biopsy information and the correct region candidate data; In the step of performing learning, the recognizer is trained using the learning image and the correct answer region data; A learning method, wherein the correct area candidate data is data indicating a plurality of the attention areas.

10. A program for causing a learning device having a recognizer configured with a neural network and a processor to perform a learning method, The processor, acquiring a learning image of an object to be inspected and biopsy information associated with the learning image, the biopsy information being information indicating a location of the object to be inspected; generating correct region data based on the biopsy information, the correct region being a region including the site where the biopsy was performed; A step of training the recognizer that recognizes an area of ​​interest using the learning image and the correct area data; A program for carrying out a learning method including: The processor, In the acquiring step, correct region candidate data paired with the learning image is acquired; In the generating step, the correct region data is generated based on the biopsy information and the correct region candidate data; In the step of performing learning, the recognizer is trained using the learning image and the correct answer region data; The correct area candidate data is data indicating a plurality of the attention areas.

11. A trained model of the recognizer obtained by the training method according to claim 9.

12. An endoscope system equipped with the trained model of the recognizer according to claim 11.

Images