Information processing apparatus, learning data generation apparatus, and diagnostic support system

Abstract

To provide an information processor, a learning data generator, and a diagnosis support system that can efficiently increase the accuracy of detection when detecting a noted region by using a medical image.SOLUTION: An information processor 18 detects a noted region in medical image data by a first detector 61, performs control so that medical image data which does not include a noted region to detect according to the result of detection, in a display 16, receives information which specifies medical image data for which detection result has been determined to be false negative from a user, detects a noted region again on the basis of the false negative medical image data by a second detector 62, and executes learning using a first medical image dataset including the medical image data and the re-detection result on the first detector 61.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a learning data generation apparatus, and a diagnostic support system.

Background Art

[0002] In the medical field, image recognition processing is performed using medical images obtained by various modalities such as an endoscope, CT (Computed Tomography), or MRI (Magnetic Resonance Imaging) to obtain information for supporting a doctor's diagnosis. In recent years, various methods for obtaining desired information by image recognition processing using machine learning techniques have been developed.

[0003] Various ideas have been studied when performing image recognition processing using machine learning techniques. For example, in a method for determining defects in manufactured products, a first model for discriminating good product images and a second model generated to discriminate between correct data and incorrect data in which a user has discriminated an abnormal candidate region are used to appropriately determine the presence or absence of an abnormal region, thereby suppressing the false detection of locations that are not defects of the detection target and obtaining sufficient inspection accuracy (Patent Document 1).

[0004] In addition, in order to shorten the expected value of the time required for determination by image recognition processing, an image inspection apparatus is known that determines the state of an object using a first neural network and determines the state of the object using a second neural network when the state of the object does not satisfy a predetermined condition (Patent Document 2).

[0005] In addition, in order to improve the recognition accuracy for rare case data, a data processing apparatus is known in which a user corrects a recognition result recognized by a recognition unit that performs image recognition processing, weights the corrected data, uses it as learning data, and performs learning of the recognition unit including this learning data (Patent Document 3).

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] When performing image recognition processing using machine learning technology, when using a learning model constructed by supervised learning, it is preferable to use a learning model constructed by learning using a large number of high-quality learning data. In addition, appropriate annotations are required for the learning data in this case.

[0008] Therefore, when performing accurate image recognition processing of medical images, it is considered preferable to use a learning model that has learned a variety of and a large number of medical images with appropriate annotations as learning data. Also, when improving the accuracy of image recognition processing in a previously constructed learning model, it is considered preferable to prepare learning data and annotations effective for improving the accuracy and to learn these learning data.

[0009] However, it has been extremely laborious to collect a variety of and a large number of medical images, or medical images effective for improving the accuracy of the learning model, and further to attach appropriate annotations to these.

[0010] An object of the present invention is to provide an information processing apparatus, a learning data generation apparatus, and a diagnostic support system that can efficiently improve the accuracy of detection when detecting a region of interest using medical images.

Means for Solving the Problems

[0011] The information processing apparatus of the present invention includes a processor. The processor acquires medical image data in which an inspection target appears, and based on the medical image data, a first detector detects a region of interest included in the inspection target appearing in the medical image data. The processor controls the display to display the medical image data for which the detection result is that the region of interest has not been detected. The processor receives information for specifying the medical image data for which the detection result is evaluated as a false negative by the user, and based on the medical image data evaluated as a false negative, a second detector re-detects the region of interest included in the inspection target appearing in the medical image data, generates a first medical image data set including the medical image data and the re-detection result of the region of interest of the medical image data, and performs learning on the first detector using the first medical image data set.

[0012] It is preferable that the computational load of the second detector is higher than that of the first detector.

[0013] The second detector is constructed using a machine learning algorithm, and it is preferable that the number of parameters of the second detector is larger than that of the first detector.

[0014] It is preferable that the second detector re-detects the region of interest based on medical image data with a higher resolution when compared with the first detector.

[0015] The second detector re-detects the region of interest based on each of a plurality of medical image data, and it is preferable that the processor integrates the results of a plurality of re-detections based on each of the plurality of medical image data to obtain the re-detection result by the second detector.

[0016] The second detector consists of a plurality, and it is preferable that each of the second detectors re-detects the region of interest based on at least one of the plurality of medical image data.

[0017] It is preferable that the plurality of medical image data includes medical image data with different resolutions from each other.

[0018] The plurality of medical image data preferably includes medical image data that has undergone different image conversion processes from each other.

[0019] The plurality of medical image data preferably includes medical image data with different imaging times from each other.

[0020] The false negative rate of the re-detection result of the second detector is preferably lower than the false negative rate of the detection result of the first detector.

[0021] The first detector is pre-constructed by learning using an initial medical image data set for a machine learning algorithm, and the processor preferably performs learning on the first detector using the first medical image data set and the initial medical image data set.

[0022] The processor preferably performs learning on the first detector after weighting each of the initial medical image data set and the first medical image data set.

[0023] The processor preferably performs learning on the first detector after weighting each of the first medical image data sets included in the plurality of first medical image data sets.

[0024] The medical image data included in the first medical image data set is preferably obtained at a specific facility.

[0025] During the examination for acquiring medical image data, the processor preferably detects a region of interest by the first detector based on the medical image data, and receives information for specifying medical image data evaluated as a false negative during the examination.

[0026] During the examination, the processor preferably re-detects the region of interest by the second detector based on the medical image data.

[0027] The processor preferably performs control to display medical image data and the detection result of the region of interest based on the medical image data on the display.

[0028] The processor preferably receives information for identifying medical image data evaluated by the user as having a false positive detection result, generates a second medical image data set including the medical image data evaluated as having a false positive and the annotation that the medical image data is a non-region of interest, and performs learning on the first detector using the second medical image data set.

[0029] In addition, the learning data generation device of the present invention includes a processor. The processor acquires medical image data in which the inspection target is imaged, detects a region of interest included in the inspection target imaged in the medical image data based on the medical image data by a first detector, performs control to display on the display the medical image data for which the detection result is that the region of interest was not detected, receives information for identifying medical image data evaluated by the user as having a false negative detection result, re-detects a region of interest included in the inspection target imaged in the medical image data based on the medical image data evaluated as having a false negative by a second detector, generates a medical image data set including the medical image data and the re-detection result of the region of interest associated with the medical image data, and stores the medical image data set in a preset storage unit.

[0030] In addition, the diagnostic support system of the present invention includes a processor. The processor acquires medical image data in which an inspection target is imaged, and a first detector detects a target area included in the inspection target imaged in the medical image data based on the medical image data. The processor controls the display to display the medical image data for which the detection result is that the target area has not been detected. The processor receives information for specifying the medical image data for which the detection result is evaluated as a false negative by the user, and a second detector re-detects the target area included in the inspection target imaged in the medical image data based on the medical image data evaluated as a false negative. The processor generates a medical image data set including the medical image data and the re-detection result of the target area associated with the medical image data, and generates a third detector by performing learning on the first detector using the medical image data set. By using the third detector to detect the target area included in the inspection target imaged in the medical image data, diagnostic support information regarding the inspection target is generated.

[0031] The first detector is preferably a third detector constructed in the past.

Advantages of the Invention

[0032] According to the present invention, when detecting a target area using a medical image, it is possible to efficiently improve the accuracy of detection.

Brief Description of the Drawings

[0033]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0034] An example of the basic configuration of the present invention will be described. As shown in FIG. 1, the diagnostic support system 10 includes a learning data generation device 11, a learning device 12, and a detection device 13. The diagnostic support system 10 is connected to an endoscope system 14, various modalities such as X-ray examinations (not shown), a device capable of outputting medical image data such as a PACS (Picture Archiving and Communication System) 15, a display device such as a display 16, and an input device 17 such as a keyboard (not shown) or a touch panel of the display 16.

[0035] Based on the medical image data acquired from the endoscope system 14 or the like, the diagnostic support system 10 detects a region of interest included in the examination object shown in the medical image, and generates diagnostic support information regarding the examination object. The diagnostic support information is output, for example, to be displayed on the display 16. A doctor can use the displayed diagnostic support information as information for diagnosing the examination object. In this specification, the region of interest is a region to be noted in the examination object shown in the medical image, for example, a lesion region.

[0036] The medical image data is, for example, data of medical images handled by a PACS, and specifically includes X-ray images obtained by X-ray examinations, MRIs obtained by MR examinations, CT images obtained by CT examinations, endoscope images obtained by endoscope examinations, or ultrasonic images obtained by ultrasonic examinations.

[0037] The learning data generation device 11 includes a first detector and a second detector, and based on the acquired medical image data, the first detector detects the region of interest. The user evaluates the detection result of the first detector, and the learning data generation device 11 receives information specifying the medical image data evaluated by the user as having a false negative detection result. Then, based on the medical image data evaluated as having a false negative result, the second detector re-detects the region of interest. A first medical image data set associating the medical image data evaluated as having a false negative result with the re-detection result is generated.

[0038] The learning device 12 performs learning of the first detector using the first medical image dataset generated by the learning data generation device 11. The detection device 13 uses the third detector constructed by the learning device 12 performing learning of the first detector to detect a region of interest included in the examination target shown in the medical image data, and generates diagnostic support information regarding the examination target. Note that the first detector, the second detector, and the third detector are learning models constructed using a machine learning algorithm.

[0039] As a configuration of the diagnostic support system 10, the learning data generation device 11, the learning device 12, and the detection device 13 may be executed as a learning data generation unit, a learning unit, and a detection unit in one computer, or via a network, the learning data generation device 11, the learning device 12, and the detection device 13 may be executed by separate computers, respectively.

[0040] Also, as shown in FIG. 2, as an information processing device 18 including a learning data generation unit 21 that executes the function of the learning data generation device 11 and a learning unit 22 that executes the function of the learning device 12, the learning data generation unit 21 and the learning unit 22 may be executed by one computer. In the present embodiment, the information processing device 18 including the learning data generation unit 21 and the learning unit 22 automatically executes the functions of the learning data generation device 11 and the learning device 12. In this case, the diagnostic support system 10 is composed of the information processing device 18 and the detection device 13.

[0041] When performing image recognition processing such as detecting a region of interest in a medical image by the diagnostic support system 10 using a detector composed of a learning model in the technology of machine learning, by training the detector with a medical image dataset composed of medical image data and annotations effective for improving the accuracy, the accuracy of detecting the region of interest by the detector can be improved. The medical image dataset is so-called learning data used for learning of the detector.

[0042] In the present invention, the detection accuracy of the detector refers to the degree to which the detection result of the detector for the region of interest detected in the medical image data including the inspection object matches the region of interest in the actual inspection object. A high or improved accuracy means a high or increasing degree of match. Specifically, when the accuracy is high, it can be said that the detection sensitivity of a smaller region of interest increases, more types of regions of interest can be detected, or the region of interest can be detected even in a medical image with poor imaging conditions.

[0043] For example, it is desired to improve the accuracy of the detector for detecting the region of interest so that the detector can correctly detect medical image data for which the detector introduced in the facility could not correctly detect the region of interest. In this case, medical image data selected from the medical image data stored in the filing system of the facility such as a medical image data server like PACS can be used.

[0044] There are two cases where the detector cannot appropriately detect the region of interest in the medical image. One is a false negative (FN, false negative) in which, for a medical image where the region of interest actually exists, the detector outputs a detection result indicating that the region of interest does not exist. The other is a false positive (FP, false positive) in which, for a medical image where the region of interest actually does not exist, the detector outputs a detection result indicating that the region of interest exists. Since it is the detection of the region of interest based on the medical image, it is preferable to minimize the false negative detection result.

[0045] Among the medical image data possessed by the facility, for the medical image data for which the detection result of the detector is a false positive, for example, it can be used for the learning of the detector after giving an annotation such as "background" uniformly.

[0046] For medical image data where the detection result of the detector is a false negative, annotation cannot be uniformly applied because the regions of interest vary in each medical image, etc. In order to create a medical image dataset of medical image data with false negative results, it is necessary to add an annotation that appropriately indicates the region of interest in each medical image. However, it is actually difficult for users such as doctors to add an annotation that appropriately indicates the region of interest to each medical image in the facility where the diagnostic support system is installed.

[0047] In the diagnostic support system 10, the information processing device 18, or the learning data generation device 11, the first detector is constructed by learning the initially prepared machine learning algorithm using the initial medical image dataset, which is the initially prepared medical image data. In order to improve the detection accuracy of the region of interest of the constructed first detector, a first medical image dataset effective for improving the detection accuracy of the first detector is prepared, and the first detector is made to learn again using these first medical image datasets. Thereby, adjustment of the parameters of the first detector, etc. is performed.

[0048] First detector 6 As a first medical image dataset effective for improving the detection accuracy of 1, when the first detector 6 1 actually executes detection, a dataset of the first medical images that failed in detection, where the detection result is a false negative or a false positive, can be mentioned.

[0049] According to the diagnostic support system 10, the information processing device 18, or the learning data generation device 11, for the medical image data where the detection result of the first detector is a false negative, re-detection is performed by the second detector, and a first medical image dataset with the re-detection result as an annotation is generated. Using the first medical image dataset as learning data, learning of the first detector is carried out. Therefore, in the first detector that has carried out learning using the re-detection result, in particular, the accuracy of detecting the region of interest based on the medical image data is improved in the direction of reducing false negative detection results.

[0050] Also, according to the diagnostic support system 10, the information processing apparatus 18, or the learning data generation apparatus 11, the learning of the first detector can be automatically performed. Therefore, in the diagnostic support system 10 using the first detector, it is possible to efficiently improve the detection accuracy of the region of interest based on the medical image data.

[0051] In particular, in the first detector included in a specific diagnostic support system 10, information processing apparatus 18, or learning data generation apparatus 11 already introduced in a facility, in order to improve the accuracy regarding the detection of the region of interest, a first medical image data set of medical image data that actually resulted in a false negative detection result by this first detector is prepared, and the first detector can be automatically re-learned using these first medical image data sets. Thereby, the diagnostic support system 10 or the information processing apparatus 18 can automatically perform learning specialized for the detection of false negative medical images occurring in this facility.

[0052] Therefore, according to the diagnostic support system 10, the information processing apparatus 18, or the learning data generation apparatus 11, regarding the diagnostic support system 10 introduced in this facility, it is possible to successively and efficiently improve the detection accuracy regarding the detection of the region of interest in a direction where the false negative detection results of the medical images occurring in this facility are reduced. Therefore, when detecting the region of interest using medical images, it is possible to efficiently improve the detection accuracy.

[0053] An embodiment of the information processing apparatus 18 of the present invention will be described. As shown in FIG. 3, as a hardware configuration, the information processing apparatus 18 of the present embodiment is a computer in which an input device 17 as an input device, a display 16 as an output device, a control unit 31, a communication unit 32, and a storage unit 33 are electrically interconnected via a data bus 34.

[0054] The input device 17 is an input device such as a keyboard, a mouse, or a touch panel of the display 16. The display 16 is an output device and may be a speaker or the like. The display 16 displays various operation screens according to operations of the input device 17 such as a mouse and a keyboard. The operation screen is provided with an operation function by a GUI (Graphical User Interface). The computer constituting the information processing device 18 receives an input of an operation instruction from the input device 17 through the operation screen.

[0055] The control unit 31 includes a CPU (Central Processing Unit) 41 which is a processor, a RAM (Random Access Memory) 42, and a ROM (Read Only Memory) 43 and the like. The CPU 41 loads a program stored in the storage unit 33 or the like into the RAM 42 or the ROM 43 and executes processing according to the program, thereby comprehensively controlling each part of the computer. The communication unit 32 is a network interface that performs transmission control of various information via the network 35. Note that the RAM 42 or the ROM 43 may have the function of the storage unit 33.

[0056] The storage unit 33 is an example of a memory, and is, for example, a hard disk drive built in the computer constituting the information processing device 18, a solid state drive, or a disk array in which a plurality of hard disk drives are connected through a cable or a network. The storage unit 33 stores a control program, various application programs, various data used for these programs, and display data of various operation screens associated with these programs.

[0057] The storage unit 33 of the present embodiment stores various data such as a program 44 for the learning data generation unit, data 45 for the learning data generation unit, a program 46 for the learning unit, and data 47 for the learning unit.

[0058] The program 44 for the learning data generation unit or the data 45 for the learning data generation unit is a program or data for implementing various functions of the learning data generation unit 21 (see FIG. 3). The functions of the learning data generation unit 21 are realized by the program 44 for the learning data generation unit and the data 45 for the learning data generation unit. Further, the data 45 for the learning data generation unit stores the first medical image data set (see FIG. 7) generated by the learning data generation unit 21, which will be described later, and the data temporarily generated by the program 44 for the learning data generation unit, etc.

[0059] The program 46 for the learning unit or the data 47 for the learning unit is a program or data for implementing various functions of the learning unit 22 (see FIG. 2), respectively. The functions of the learning unit 22 are realized by the program 46 for the learning unit and the data 47 for the learning unit. Further, the data 47 for the learning unit stores the data temporarily generated by the learning unit 22, etc.

[0060] The computer constituting the information processing apparatus 18 can be a general-purpose server apparatus, a PC (Personal Computer), etc., in addition to a dedicatedly designed apparatus.

[0061] As shown in FIG. 4, the information processing apparatus 18 of the present embodiment includes, as a software configuration, a learning data generation unit 21 and a learning unit 22 (see FIG. 2). The learning data generation unit 21 includes a medical image data acquisition unit 51, a first detection unit 52, a display control unit 53, a specific information reception unit 54, a second detection unit 55, and a first medical image data set generation unit 56. The learning unit 22 includes a first medical image data set acquisition unit 57 and a first detector learning unit 58.

[0062] The information processing apparatus 18 of the present embodiment is a processor device, and a program related to medical image data processing is stored in a storage unit 33 which is a program memory in the information processing apparatus 18. In the information processing apparatus 18, a control unit 31 constituted by a processor or the like causes a program in the program memory to operate, whereby the functions of a learning data generation unit 21, namely, a medical image data acquisition unit 51, a first detection unit 52, a display control unit 53, a specific information reception unit 54, a second detection unit 55, a first medical image data set generation unit 56, and a learning unit 22, namely, a first medical image data set acquisition unit 57 and a first detector learning unit 58 are realized.

[0063] The learning data generation unit 21, based on the acquired medical image data, detects a region of interest by a first detector, then re-detects the region of interest based on specific medical image data, and generates a first medical image data set in which the re-detection result of the re-detected region of interest is associated with the medical image data. The learning unit 22 performs learning of the detector using the generated first medical image data set.

[0064] As shown in FIG. 5, the medical image data acquisition unit 51 included in the learning data generation unit 21 acquires medical image data in which an inspection target in an endoscopy or the like is imaged from a device capable of outputting medical image data such as an endoscope system 14. In the present embodiment, endoscopic image data 71 captured during an inspection from the endoscope system 14 is acquired in real time as medical images. Therefore, for the description of the present embodiment, hereinafter, the case where endoscopic image data 71 is used as medical image data will be described.

[0065] The first detection unit 52 includes a first detector 61. The first detector 61 detects a region of interest included in the inspection target shown in the acquired endoscopic image data 71 based on the endoscopic image data 71. Specifically, the first detector 61 is a learning model constructed using a machine learning algorithm, and when feature amounts based on the endoscopic image data 71 are input to the first detector 61, it is a learning model capable of outputting the presence or absence of the region of interest in the endoscopic image data 71 as an objective variable. The first detector 61 is pre-learned using an initial image data set for the first detector 61 consisting of endoscopic image data using a machine learning algorithm so that it can output the presence or absence of the region of interest in the endoscopic image data 71 as an objective variable, and parameters and the like are adjusted.

[0066] As the machine learning algorithm used for the first detector 61, various algorithms can be used as long as they are algorithms used for supervised learning, but it is preferable to use an algorithm that is said to output good inference results in image recognition as an objective variable. For example, it is preferable to use a multi-layer neural network or a convolutional neural network, and it is preferable to use a technique called so-called deep learning.

[0067] The detection result of the region of interest includes the location, size or area, shape, or number, etc. of the region of interest detected in the endoscopic image data 71, and also includes the content that the location or size, etc. of the region of interest was 0, that is, the region of interest was not detected. Based on the detection result of the region of interest by the first detector 61, the endoscopic image data 71 is classified into non-detection endoscopic image data 72 of the region of interest in which the region of interest 73 was not detected and detection endoscopic image data 74 of the region of interest in which the region of interest 73 was detected. Note that multiple regions of interest may be detected in one endoscopic image, but the non-detection endoscopic image data 72 of the region of interest is the endoscopic image data 71 in which no region of interest was detected. The detection endoscopic image data 74 of the region of interest is associated with the detection result regarding the region of interest and can be used as diagnostic support information for a doctor 76 to diagnose the inspection target.

[0068] The display control unit 53 controls the display on the display 16 of the endoscopic image data 71 indicating that no attention area has been detected, that is, the endoscopic image data 72 without detection of the attention area. The endoscopic image data 72 without detection of the attention area may include undetected attention areas 75 that were not detected by the first detector 61 but may be determined to be attention areas when confirmed by a doctor or the like. Note that the display control unit 53 may at least display the endoscopic image data 72 without detection of the attention area and may control the display of endoscopic image data 71 other than the endoscopic image data 72 without detection of the attention area, or may control the display of the endoscopic image data 74 with detection of the attention area.

[0069] The doctor 76, who is the user, determines from the endoscopic image data 72 without detection of the attention area displayed on the display 16 whether the undetected attention area 75 is included or not. Among the endoscopic image data 72 without detection of the attention area, the endoscopic image data 71 determined by the doctor 76 to include the undetected attention area 75 is the false-negative endoscopic image data 77 where the detection result of the first detector 61 is a false negative. The doctor 76 determines the presence or absence of the undetected attention area 75 from the endoscopic image data 72 without detection of the attention area. If the undetected attention area 75 is included, the doctor 76 selects, as the false-negative endoscopic image data 77, the endoscopic image data 71 that is the false-negative endoscopic image data 77 using the input device 17 such as the touch panel of the display 16.

[0070] Based on the selection by the doctor 76 using the input device 17, the specific information reception unit 54 receives information from the doctor 76 for specifying the endoscopic image data 71 that is the false-negative endoscopic image data 77 including the undetected attention area 75 and evaluated as having a false-negative detection result.

[0071] As shown in FIG. 6, the second detection unit 55 specifies and acquires the false-negative endoscopic image data 77 from the endoscopic image data 71 acquired by the medical image data acquisition unit 51 based on the information specifying the false-negative endoscopic image data 77 from the specific information reception unit 54.

[0072] Based on the false negative endoscopic image data 77, the second detection unit 55 re-detects the region of interest 73 included in the inspection target shown in the false negative endoscopic image data 77. The second detection unit 55 includes a second detector 62. Specifically, the second detector 62 is a learning model constructed using a machine learning algorithm, and when feature quantities based on the false negative endoscopic image data 77 are input to the second detector 62, it can output, as the re-detection result 79, whether or not the region of interest 73 exists in the false negative endoscopic image data 77, with the existence or non-existence as the target variable.

[0073] As the feature quantities based on the false negative endoscopic image data 77, the feature quantities when using the false negative endoscopic image data 77, which is the same type of endoscopic image data 71 as that used for the first detector 61, may be used, or the feature quantities of the adjusted endoscopic image 78 generated by performing image adjustment processing on the false negative endoscopic image data 77 may be used. Also, the false negative endoscopic image data 77 may be used alone, or a plurality of endoscopic image data 71 including the false negative endoscopic image data 77 and other endoscopic image data 71 may be used.

[0074] As a method of the image adjustment processing, image adjustment processing for improving the detection accuracy of the learning model in the machine learning technology can be performed. For example, methods such as using the adjusted endoscopic image 78 created by changing the resolution can be mentioned. Also, the same applies to the method of using a plurality of false negative endoscopic image data 77, and methods performed for improving the detection accuracy of the learning model in the machine learning technology can be adopted. For example, methods such as using the adjusted endoscopic image 78 obtained by duplicating the false negative endoscopic image data 77 and making their respective resolutions different from each other, or using the endoscopic image data 71 of the frames before and after the false negative endoscopic image data 77 is obtained as the adjusted endoscopic image 78 can be mentioned.

[0075] As shown in FIG. 7, in the present embodiment, the second detection unit 55 includes an image adjustment unit 63. The image adjustment unit 63 performs an image adjustment process for increasing the resolution of the false negative endoscopic image data 77, and generates adjusted endoscopic image 78. It is preferable that the second detector 62 re-detects the region of interest based on the endoscopic image data 71 having a higher resolution when compared with the first detector 61. By inputting endoscopic image data with a high resolution to the second detection unit 55, the detection sensitivity of a smaller region of interest can be improved.

[0076] As a method for the image adjustment process of increasing the resolution of the false negative endoscopic image data 77, various methods conventionally performed for increasing the number of pixels of the image data can be adopted. After performing the method of increasing the number of pixels, image processing such as sharpness processing may be performed.

[0077] As shown in FIG. 6, the second detector 62 re-detects the region of interest 73 in the adjusted endoscopic image 78 by inputting the feature amount using the adjusted endoscopic image 78 in which the image adjustment process has been performed. The second detector 62 is a detector that can re-detect the region of interest 73 that the first detector 61 could not correctly detect. Therefore, the second detector 62 is a detector different from the first detector 61. The second detector 62 is learned by an initial image data set for the second detector 62 composed of endoscopic image data using a machine learning algorithm, and the parameters and the like are adjusted so that the region of interest that the first detector 61 could not correctly detect can be re-detected.

[0078] The machine learning algorithm used for the second detector 62 may be the same as or different from that of the first detector 61, and various algorithms can be used as long as they are algorithms used for supervised learning. However, it is preferable to use an algorithm that outputs a good inference result in image recognition as the target variable. For example, it is preferable to use a multi-layer neural network or a convolutional neural network, and it is preferable to use a technique called so-called deep learning.

[0079] The redetection result 79 of the region of interest includes the location, size or area, shape, or number, etc. of the region of interest 73 redetected in the false-negative endoscopic image data 77, and also includes a redetection result 79 indicating that the region of interest 73 was not redetected. Region of interest 7 The redetection result 79 of 7 can be used as an annotation for the false-negative endoscopic image data 77. Also, the false-negative endoscopic image data 77 is associated with the redetection result 79 regarding the region of interest, and the redetection result 79 itself can be utilized as diagnostic support information for a doctor 76 to diagnose the inspection target.

[0080] The first medical image dataset generation unit 56 generates a first medical image dataset 81 including the false-negative endoscopic image data 77 input to the second detector 62 and the redetection result 79 of the region of interest 73 associated with this false-negative endoscopic image data 77.

[0081] In this specification, the association when referring to the redetection result 79 of the region of interest 73 associated with the false-negative endoscopic image data 77 means that when the false-negative endoscopic image data 77 is specified, the redetection result 79 detected based on this false-negative endoscopic image data 77 can be specified. For the association, for example, correspondence information in which the false-negative endoscopic image data 77 and the redetection result 79 detected based on this false-negative endoscopic image data 77 correspond can be used. With this correspondence information, when the false-negative endoscopic image data 77 is specified, the redetection result 79 of this false-negative endoscopic image data 77 is specified. In this embodiment, correspondence information in which the false-negative endoscopic image data 77 and the redetection result 79 detected based on the false-negative endoscopic image data 77 correspond is created and used. The correspondence information may be one in which the feature amount of the false-negative endoscopic image data 77 and the redetection result 79 are associated.

[0082] The first medical image dataset 81 can be used as learning data for training the first detector 61. The first medical image dataset 81 is sent to the learning unit 22.

[0083] As shown in FIG. 8, in the learning unit 22, the first medical image dataset acquisition unit 57 acquires the first medical image dataset 81 generated by the first medical image dataset generation unit 56. The first detector learning unit 58 performs learning of the first detector 52 using the first medical image dataset 81 as learning data. The first medical image dataset 81, which is the learning data, includes false negative endoscopic image data 77 to be input to the first detector 61 during learning and a redetection result 79 for the false negative endoscopic image data 77 which is the annotation.

[0084] The learning is supervised learning, and the teacher data (annotation) is the redetection result 79 of the region of interest 73 in the false negative endoscopic image data 77. The first detector learning unit 58 inputs the false negative endoscopic image data 77 to the first detector 61 and compares the output detection result 91 with the redetection result 79 which is the annotation. Then, the parameters of the first detector 61 are updated so that the detection result 91 becomes the same as the redetection result 79.

[0085] The first detector learning unit 58 can adopt various methods performed by machine learning techniques during learning. For example, when the algorithm of the first detector 61 is a neural network, it is preferable to perform learning such that some of the network parameters are fixed and some are updated. Thereby, it becomes possible to prevent so-called catastrophic forgetting, and it becomes possible to cope with new endoscopic image data 71 while maintaining the basic performance of the first detector 61 before learning it with the first medical image dataset 81. Also, for the same reason, it is also preferable to lower the learning rate, which is a hyperparameter. R ate).

[0086] As described above, according to the information processing apparatus 18, for the false negative endoscopic image data 77 in which the detection result of the first detector 61 is a false negative, re-detection is performed by the second detector, and a first medical image data set with the re-detection result 79 as an annotation is generated. Using the first medical image data set as learning data, learning of the first detector 61 is performed. Therefore, in the first detector 61 that has performed learning using the re-detection result 79, in particular, the accuracy of detecting the region of interest based on the endoscopic image data 71 is improved in a direction where false negative detection results are reduced.

[0087] Moreover, according to the information processing apparatus 18, since it can be automatically performed without manual intervention except when the user selects the false negative endoscopic image data 77, in the information processing apparatus 18 using the first detector 61, it is possible to efficiently improve the detection accuracy of the region of interest 73 based on the endoscopic image data 71.

[0088] In particular, in the first detector 61 included in a specific information processing apparatus 18 already introduced in the facility, in order to improve the accuracy regarding the detection of the region of interest, a first medical image data set 81 of the endoscopic image data 71 that actually resulted in a false negative detection result by this first detector 61 is prepared, and the first detector 61 can be automatically re-learned using these first medical image data sets 81. Thereby, the information processing apparatus 18 can automatically perform learning specialized for the detection of false negative endoscopic image data 71 occurring in this facility.

[0089] Therefore, according to the information processing apparatus 18, for the diagnostic support system 10 introduced in this facility, it is possible to successively improve the detection accuracy regarding the detection of the region of interest in a direction where false negative detection results of the endoscopic image data 71 occurring in this facility are reduced.

[0090] Next, the first detector 61 and the second detector 62 will be further described. The second detector 62 is a detector different from the first detector 61, and is a detector capable of re-detecting the region of interest for the endoscopic image data 71 in which the region of interest is not detected by the first detector 61 and a false negative detection result is obtained. In the second detector 62, the region of interest is re-detected by a process with higher accuracy than the process in the first detector 61.

[0091] Generally, in a learning model for detecting a region of interest, the detection accuracy and the detection speed, that is, the small amount of calculation, are in a trade-off relationship, and the second detector 62 performs a process with a larger amount of calculation than the first detector 61. The process in the first detector 61 preferably suppresses the amount of calculation so that the region of interest can be detected in real time during an endoscopic examination. However, the process in the second detector device 62 can be processed in the background after or during the endoscopic examination, so even if the calculation time is long, there is little possibility of becoming a problem as a whole.

[0092] Therefore, it is preferable that the operation load of the second detector 62 is higher than the operation load of the first detector 61. A high operation load means a large amount of calculation in the control unit 31 constituted by a processor or the like. In the second detector 62, since a process with a larger amount of calculation and higher accuracy than the process in the first detector 61 is performed, it becomes possible to easily re-detect the region of interest that was not detected by the first detector 61.

[0093] Further, the second detector 62 is constructed using a machine learning algorithm, and it is preferable that the number of parameters of the second detector 62 is larger than the number of parameters of the first detector 61. Therefore, it is preferable that the control unit 31 sets the number of parameters of the first detector 61 to be less than the number of parameters of the second detector 62. As the machine learning algorithm, as described above, it is preferably a multi-layer neural network or a convolutional neural network that is used for supervised learning and outputs a good inference result in image recognition as the target variable, and it is preferable to use a method called so-called deep learning. For example, in these algorithms, a learning model with a large number of parameters can perform high-precision image recognition. Therefore, in the second detector 62, it becomes possible to easily re-detect the target area that was not detected by the first detector 61.

[0094] Also, since it is a model for detecting the target area, an algorithm used for object detection can be used as the learning model of the second detector 62. In particular, in object detection, it is preferable to use an algorithm called a Two-Stage system that can accurately detect an object. The Two-Stage system algorithm is an algorithm that performs detection in two stages, and is an algorithm that serially executes a stage of detecting candidates for the area of the subject and a stage of identifying the category of the area candidates. The Two-Stage system algorithm generally has a higher computational load and operates more slowly compared to, for example, a Single-Stage system algorithm that performs area extraction and category identification in the same stage. However, as described above, since it is preferable that the computational load of the second detector 62 is higher than that of the first detector 61, the second detector 62 preferably uses a Two-Stage system algorithm, whereby the target area can be re-detected with higher accuracy.

[0095] Regarding the endoscopic image data 71 input to the second detector 62 and the first detector 61, the second detector 62 preferably re-detects the region of interest based on each of a plurality of endoscopic image data 71, and integrates a plurality of re-detection results based on each of these plurality of endoscopic image data 71 to obtain the re-detection result by the second detector 62. By using a plurality of re-detection results 79 obtained by inputting a plurality of endoscopic image data 71, it is possible to accurately detect the region of interest of the endoscopic image data 71. Therefore, in the second detector 62, it becomes possible to easily re-detect the region of interest that was not detected by the first detector 61.

[0096] As shown in FIG. 9, by inputting each of the false-negative endoscopic image data 77a, the false-negative endoscopic image data 77b, and the false-negative endoscopic image data 77c as a plurality of endoscopic image data 71 to the second detector 62, a re-detection result 79a, a re-detection result 79b, and a re-detection result 79c are obtained. These plurality of re-detection results are integrated to obtain the re-detection result 79 by the second detector 62.

[0097] As a method for integrating the re-detection results 79a, the re-detection result 79b, and the re-detection result 79c, which are a plurality of re-detection results 79, it is possible to adopt a conventionally used method such as various methods for integrating image data, with the re-detection results 79a, the re-detection result 79b, and the re-detection result 79c being used as image data. For example, in each of the re-detection results 79a, the re-detection result 79b, and the re-detection result 79c, by performing an average or a majority vote on the pixel values at the same position, an integrated re-detection result 79 can be obtained. At this time, in the image data of the re-detection results 79a, the re-detection result 79b, and the re-detection result 79c, alignment may be performed before integration.

[0098] Also, regarding the endoscopic image data 71 input to the second detector 62 and the first detector 61, the second detector 62 is composed of a plurality, and each of the second detectors 62 re-detects a region of interest based on at least one of the plurality of endoscopic image data 71, and integrates the plurality of re-detection results, which may be used as the re-detection result by the second detector 62. By using the plurality of re-detection results 79 obtained by each of the plurality of second detectors 62, it is possible to accurately detect the region of interest in the endoscopic image data 71. Therefore, in the second detector 62, it becomes possible to easily re-detect the region of interest that was not detected by the first detector 61.

[0099] As shown in FIG. 10, as the second detector 62, each of the second detector 62a, the second detector 62b, and the second detector 62c re-detects a region of interest based on at least one of the plurality of endoscopic image data 71, and integrates the plurality of re-detection results, which may be used as the re-detection result by the second detector 62. Specifically, the second detector 62a outputs a re-detection result 79a by re-detecting a region of interest based on the false negative endoscopic image data 77a, the second detector 62b outputs a re-detection result 79b by re-detecting a region of interest based on the false negative endoscopic image data 77b, and the second detector 62c outputs a re-detection result 79c by re-detecting a region of interest based on the false negative endoscopic image data 77c.

[0100] The method of integrating the plurality of re-detection results 79 is the same as described above. The integrated re-detection result 79d obtained by averaging or majority voting the pixel values at each position of the re-detection result 79a, the re-detection result 79b, and the re-detection result 79c is used as the re-detection result 79 by the second detector 62.

[0101] Note that, as the plurality of endoscopic image data 71 input to the second detector 62, it is preferable that each of the plurality of endoscopic image data 71 is different from each other. By using the plurality of re-detection results 79 obtained by inputting the plurality of endoscopic image data 71 that are different from each other, it is possible to accurately detect the region of interest in the endoscopic image data 71. Therefore, in the second detector 62, it becomes possible to easily re-detect the region of interest that was not detected by the first detector 61. Such a plurality of endoscopic image data 71 can specifically be the following endoscopic image data 71.

[0102] The plurality of endoscopic image data 71 preferably includes endoscopic image data 71 having different resolutions from each other. For that purpose, as described above, the image adjustment unit 63 (see FIG. 6) performs an image adjustment process so as to generate a plurality of false negative endoscopic image data 77 having different resolutions based on the false negative endoscopic image data 77 which is the endoscopic image data 71.

[0103] As shown in FIG. 11, the image adjustment unit 63 acquires the false negative endoscopic image data 77 and generates false negative endoscopic image data 77a, false negative endoscopic image data 77ab, and false negative endoscopic image data 77c having different resolutions from each other. In FIG. 11, the differences in the sizes of the false negative endoscopic image data 77a, the false negative endoscopic image data 77 b , and the false negative endoscopic image data 77c represent the differences in resolution. The resolutions of the false negative endoscopic image data 77, shown in ascending order, are the false negative endoscopic image data 77a, the false negative endoscopic image data 77b, and the false negative endoscopic image data 77c.

[0104] As described above, the second detector 62 outputs a re-detection result 79a by inputting the false negative endoscopic image data 77a, outputs a re-detection result 79b by inputting the false negative endoscopic image data 77b, and outputs a re-detection result 79c by inputting the false negative endoscopic image data 77c (see FIG. 9). By integrating these re-detection results 79a, re-detection result 79b, and re-detection result 79c, a re-detection result 79 is generated.

[0105] Note that since these re-detection results 79 are image data with different resolutions, when integrating them, they are changed to have the same resolution as the original false-negative endoscopic image data 77 before changing the resolution by the image adjustment process. Therefore, the resolutions of the re-detection result 79a, the re-detection result 79b, and the re-detection result 79c are changed to be the same as those of the original false-negative endoscopic image image 77's After making the resolutions the same, the average of the re-detection results of these re-detection result 79a, the re-detection result 79b, and the re-detection result 79c is set as the re-detection result 79. Thereby, the re-detection result 79 based on various types of false-negative endoscopic image data 77 can be obtained, and by integrating these re-detection results 79, the detection accuracy is improved. Therefore, in the second detector 62, it becomes possible to easily re-detect the region of interest that was not detected by the first detector 61.

[0106] Also, the image adjustment unit 63 (see FIG. 6) may perform an image adjustment process so as to generate a plurality of false-negative endoscopic image data 77 on which different image conversion processes are performed based on the false-negative endoscopic image data 77 that is the endoscopic image data 71.

[0107] As shown in FIG. 12, the image adjustment unit 63 acquires the false-negative endoscopic image data 77 and generates false-negative endoscopic image data 77a, false-negative endoscopic image data 77ab, and false-negative endoscopic image data 77c on which different image conversion processes are performed. In FIG. 12, the differences in the shading of the false-negative endoscopic image data 77a, the false-negative endoscopic image data 77ab, and the false-negative endoscopic image data 77c represent the differences in the performed image conversion processes.

[0108] As the image conversion process, various image conversion processes can be adopted, but it is preferably a process for making the endoscopic image data 71 easier to view. This is because by inputting the endoscopic image data 71 with poor shooting conditions to the second detector 62 after performing a process for making it easier to view, it is possible to improve the detection accuracy. Examples of the process for making the endoscopic image data 71 easier to view include a process for changing the color tone, a structure enhancement process, or contrast flattening.

[0109] Also, as the image conversion process, it is also preferable to adopt an image conversion process that is said to improve the detection accuracy in image recognition processing in machine learning. As such image conversion processes, various methods performed for data augmentation of learning data can be adopted.

[0110] In particular, it is preferable to adopt a method such as TTA (Test Time Augmentation), which is said to have excellent detection accuracy. In TTA, a plurality of endoscopic image data 71 are input into the second detector 62 to obtain a plurality of detection results, and the result obtained by averaging or majority voting of these detection results is used as the detection result.

[0111] The image adjustment unit 63 generates false negative endoscopic image data 77a, false negative endoscopic image data 77b, and false negative endoscopic image data 77c by performing mutually different image conversion processes in the above-described image conversion process. Note that different image conversion processes include image conversion processes performed with different parameters in the same image conversion method, and also include image conversion processes performed with different methods.

[0112] As described above, the image adjustment unit 63 generates false negative endoscopic image data 77a, false negative endoscopic image data 77ab, and false negative endoscopic image data 77c that have undergone different image conversion conversion processes, and the second detector 62 outputs detection results using these different false negative endoscopic image data 77 respectively. Thereby, a re-detection result 79 based on various types of false negative endoscopic image data 77 can be obtained, and by integrating these re-detection results 79, the detection accuracy is improved. Therefore, in the second detector 62, it becomes possible to easily re-detect the region of interest that was not detected by the first detector 61.

[0113] Further, it is preferable that the plurality of endoscopic image data 71 include endoscopic image data 71 with different shooting times from each other. As the endoscopic image data 71 with different shooting times from each other, in the second detector 62, since the false negative endoscopic image data 77 is input and the detection result is output, for example, before or after the time when the endoscopic system acquires the false negative endoscopic image data 77, or endoscopic image data 71 acquired within a preset range before and after can be adopted and used.

[0114] The endoscopic system acquires one piece of endoscopic image data 71 per frame. In this specification, "frame" means the unit of shooting the inspection target by the endoscopic system. As shown in FIG. 13, by the endoscopic system 14, during a certain period of the inspection, in chronological order from the earlier one, endoscopic image data 71a, endoscopic image data 71b, endoscopic image data 71c, endoscopic image data 71d, and endoscopic image data 71 e are acquired. Among these endoscopic image data 71, when the endoscopic image data 71c is the false negative endoscopic image data 77 including the non-detection target region 75, the second detector 62 targets the endoscopic image data 71b and the endoscopic image data 71d, which are the frames before and after the endoscopic image data 71c, for target region detection in the same manner as the endoscopic image data 71c. Then, the detection result obtained by integrating the plurality of detection results by the endoscopic image data 71b, the endoscopic image data 71c, and the endoscopic image data 71d is used as the detection result of the target region of the false negative endoscopic image data 71c. The integration can be performed in the same manner as above.

[0115] In addition to adopting one frame of endoscopic image data 71 immediately before and immediately after the false negative endoscopic image data 77, it is also possible to adopt a specific plurality of frames immediately before and immediately after the time when the false negative endoscopic image data 77 is acquired, or to adopt frames separated by a specific number before and after the time when the false negative endoscopic image data 77 is acquired.

[0116] In the second detection unit 55, during shooting intervalA plurality of endoscopic image data 71 that are different from each other are input into the second detector 62, and by integrating the plurality of re-detection results 79 output therefrom, it is possible to detect a region of interest that is difficult to confirm with a single endoscopic image data 71 and becomes easier to understand by checking the time-series endoscopic image data 71.

[0117] As described above, the image adjustment unit 63 generates false-negative endoscopic image data 77a, false-negative endoscopic image data 77ab, and false-negative endoscopic image data 77c that have undergone different image conversion processing, and the second detector 62 outputs detection results using these different false-negative endoscopic image data 77 respectively. Thereby, re-detection results 79 based on various types of false-negative endoscopic image data 77 can be obtained, and by integrating these re-detection results 79, the detection accuracy is improved. Therefore, in the second detector 62, it becomes possible to easily re-detect a region of interest that was not detected by the first detector 61.

[0118] As described above, the second detector 62 is different from the first detector 61, and preferably, the false-negative rate of the re-detection results of the second detector 62 is lower than the false-negative rate of the first detector 61. The false-negative rate means the ratio of the endoscopic image data 71 that has become a false-negative detection result among all the re-detection results or all the detection results for the input endoscopic image data 71. The second detector 62 is a detector that can re-detect a region of interest for the endoscopic image data 71 for which the region of interest was not detected by the first detector 61 and a false-negative detection result was obtained. When the same endoscopic image data 71 as the endoscopic image data 71 that results in a false-negative result in the first detector 61 is input into the second detector 62, it is highly likely that a false-negative result will not occur. Therefore, in this case, in the second detector 62, it is possible to easily re-detect a region of interest that was not detected by the first detector 61.

[0119] Next, the learning using the first medical image dataset 81 for the first detector 61 will be further described. When the first detector 61 is learned using the first medical image dataset 81, it is preferable that in the first detector 61, the false negative rate in the detection result is reduced compared to before learning using the first medical image dataset 81. Therefore, when performing the learning using the first medical image dataset 81 for the first detector 61, it is preferably performed as follows.

[0120] For the first detector 61, it is preferable to perform learning using the first medical image dataset 81 and the initial medical image dataset. By learning both the first medical image dataset 81 and the initial medical image dataset, in the first detector 61, without degrading the detection accuracy of the region of interest constructed by learning using the initial medical image dataset, it is possible to improve the detection of the region of interest that has failed to be detected by learning using the first medical image dataset so as to succeed.

[0121] Also, it is preferable to perform learning on the first detector 61 after weighting each of the first medical image dataset 81 and the initial medical image dataset. As a weighting method, for example, the weight of the first medical image dataset 81 can be set to be larger than that of the initial medical image dataset to perform learning. The learning data included in the first medical image dataset 81 is learning data that includes the region of interest that the first detector 61 could not detect as an annotation. Therefore, by increasing the weight of learning using these learning data, it becomes possible to detect the region of interest that could not be detected in the region of interest detection in the first detector 61, leading to the overcoming of weaknesses.

[0122] On the one hand, learning can be performed by setting the weight of the first medical image dataset 81 to be smaller than that of the initial medical image dataset. Since the training data included in the first medical image dataset 81 uses the re-detection result 79 by the second detector 62 as an annotation, it may not be accurate ground truth data. Therefore, by reducing the weight of learning using these training data, it is possible to improve the detection accuracy while preventing the possibility of a decrease in detection accuracy in the region of interest detection in the first detector 61.

[0123] Also, it is preferable to perform learning on the first detector 61 after weighting each of the first medical image datasets 81 included in the plurality of first medical image datasets 81. The first medical image dataset 81 includes false negative endoscopic image data 77 and the re-detection result 79 by the second detector 62 for the false negative endoscopic image data 77. For example, when there are a plurality of second detectors 62, the re-detection result 79 is obtained by integrating a plurality of re-detection results 79. Here, paying attention to the variance of the plurality of re-detection results 79 before integration, for the first medical image dataset 81 consisting of the plurality of re-detection results 79 with a small variance, increase the weight. Those with a small variance of the plurality of re-detection results 79 are those with similar re-detection results 79 in each of the plurality of second detectors 62. Therefore, it is highly likely that they are training data with high accuracy of the re-detection result 79. Thus, for the training data with high accuracy, by performing learning of the first detector 61 after increasing the weight, it is possible to improve the detection accuracy of the first detector 61.

[0124] It is also preferable to learn using the first medical image dataset 81 as a soft target. By causing the first detector 61 to learn, using as a soft label the region of interest included in the re-detection result 79 by the second detector 62, it is possible to efficiently learn the first detector 61 with a small number of the first medical image datasets 81. In the layer structure of the first detector 61, it is also preferable to learn the first detector 61 by adjusting parameters so that, up to an intermediate process, it is the same as the first detector 61 before learning using the first medical image dataset 81.

[0125] Note that the false negative endoscopic image data 77 included in the first medical image dataset 81 is preferably acquired at a specific facility. The specific facility is preferably the facility that acquired the endoscopic image data 71 used for detecting the region of interest by the first detector 61, and it is preferable that the first medical image dataset 81 is acquired at this facility. Thereby, for example, in the first detector 61 included in the diagnostic support system introduced into a specific facility, learning specialized for the endoscopic image data 71 acquired at this facility becomes possible, and detection of the region of interest for the endoscopic image data 71 acquired at this facility can be made more accurate, which is preferable.

[0126] Next, the timing etc. of detection by the first detector 61 or re-detection by the second detector 62 will be further described. According to the information processing apparatus 18, during an examination for acquiring the endoscopic image data 71, it is preferable that the first detector 61 detects the region of interest based on the acquired endoscopic image data 71. Thereafter, during the same examination, it is preferable to receive information for specifying the endoscopic image data 71 evaluated as false negative by the doctor 76.

[0127] The endoscopic image data 71 is obtained by being captured by an endoscope included in the endoscope system during an endoscopic examination (see Fig. 5). During the endoscopic examination, the endoscopic image data 71 obtained by being captured by the endoscope is displayed as a video on the display 16. The first detector 61 acquires the obtained endoscopic image data 71 and detects a region of interest. The display 16 is controlled by the control unit 31 to display at least the endoscopic image data 71 for which the detection result is that no region of interest was detected.

[0128] Regarding the display of the endoscopic image data 71 for which the detection result is that no region of interest was detected, it may be displayed in a manner that allows the doctor 76 to recognize that no region of interest was detected in the endoscopic image data 71 being displayed. As shown in Fig. 14, for example, when the endoscopic image data 71 is displayed on the display 16 as the endoscopic image 101, as the detection result by the first detector 61, it is indicated by a specific - colored frame 102 showing that no region of interest was detected. On the display 16, the endoscopic image 101 is displayed in the endoscopic image area 103, and the detection result 91 is displayed in the detection result area 104. The detection result 91 may be displayed in the endoscopic image area 103 by being superimposed on the endoscopic image 101. In the case of Fig. 14, since the endoscopic image 101 includes the region of interest 73 and the detection result 91 or the frame 102 shows that there is no region of interest, it is a case where the false - negative endoscopic image data 77 is displayed on the display 16.

[0129] When the doctor 76 looks at the endoscopic image 101 and the detection result 91 displayed on the display 16 and determines that a region of interest 73 exists in the examination subject, the doctor issues information identifying that this endoscopic image data 71 is the false - negative endoscopic image data 77. Examples of the method include, for example, pressing a specific scope button provided on the endoscope when this endoscopic image data 71 is being displayed on the display 16, indicating it by voice using voice recognition, or pressing a foot switch provided in the endoscope system, and the like.

[0130] By the method as described above, the doctor 76 can issue information for identifying the endoscopic image data 71 that was evaluated as false negative during the examination. Thereby, the labor of identifying the endoscopic image data 71 that was evaluated as false negative after the examination can be saved. During the examination, when the doctor 76 diagnoses that the examination target includes the target area, the doctor often presses the freeze button provided on the endoscope to obtain a still image. Therefore, the freeze button may also serve as the scope button for identifying the endoscopic image data 71 that was evaluated as false negative. By referring to the acquisition time of the still image and the detection result 91 of the first detector 61 at that time, it is possible to automatically determine whether the endoscopic image data 71 is the one evaluated as false negative or not.

[0131] The specific information reception unit 54 (see FIG. 4) receives the information of the endoscopic image data 71 specified by the doctor 76. As described above, since the evaluation that the test result is false negative can be easily performed during the examination, it does not take time to evaluate the false negative endoscopic image data 77, and the false negative endoscopic image data 77 can be efficiently identified.

[0132] During the examination, the second detector 62 preferably re-detects the target area based on the false negative endoscopic image data 77. The second detector 62 detects the target area in the background based on the false negative endoscopic image data 77. When the target area is detected, by displaying the detection result on the display 16, the doctor 76 can obtain the re-detection result by the second detector 62 for the endoscopic image data 71 that the doctor himself / herself evaluated as the false negative endoscopic image data 71.

[0133] Also, after the examination to acquire the endoscopic image data 71, a region of interest may be detected based on the endoscopic image data 71 acquired by the first detector 61. Then, for example, when creating an examination report, information for identifying the endoscopic image data 71 evaluated as a false negative by the doctor 76 may be received. When creating an examination report, the doctor 76 often selects an endoscopic image to be included in the examination report from the still images acquired during the examination. The still images acquired by the doctor 76 during the examination often show the lesion or the like clearly and with high image quality without blurring or distortion. Such a still image, the false negative endoscopic image data 77 in which the first detector 61 was unable to detect the region of interest, is a region of interest that is difficult to detect, such that even with high image quality, the first detection unit 52 is unable to detect the region of interest. Therefore, it is valuable as learning data. Thus, evaluating the false negative endoscopic image data 77 by the doctor 76 after the examination is effective for generating a good first medical image dataset 81.

[0134] Also, when performing the detection of the region of interest by the first detector 61 after the examination, there is no need to accelerate the detection speed of the first detector 61 to be close to real time. Also, since the doctor 76 creates an examination report after the examination, the doctor 76 can perform the creation of the examination report and the evaluation of the false negative endoscopic image data 77 simultaneously, and thus can efficiently identify the false negative endoscopic image data 77. Also, since the doctor 76 can obtain the detection results for the endoscopic image data 71 evaluated as the false negative endoscopic image data 71 by himself / herself, these results can be referred to, leading to the creation of a more accurate examination report.

[0135] Also, the doctor 76 may evaluate the false-negative endoscopic image data 77 both during and after the examination. For example, when the doctor 76 evaluates that there is even a slight possibility of false-negative endoscopic image data 77 during the examination as described above, the doctor may roughly record it. Next, after the examination, by referring to this record, the evaluation that it is false-negative endoscopic image data 77 is confirmed. This record may be a record of the time when the endoscopic image data 71 with the possibility of false-negative endoscopic image data 77 is acquired during the examination, or the record may be tagged to the endoscopic image data 71 itself. Thereby, even when it is not possible to evaluate the false-negative endoscopic image data 77 over time during the examination, the false-negative endoscopic image data 77 can be evaluated more appropriately after the examination without the trouble of searching for the acquired endoscopic image data 71. By evaluating the appropriate false-negative endoscopic image data 77, a useful first medical image data set 81 can be generated.

[0136] Note that the display 1 6 may be controlled to display the endoscopic image data 71 and the detection result of the region of interest by the first detector 61 based on the endoscopic image data 71. On the display 16, the endoscopic image data 71 and the detection result of the region of interest by the first detector 61 may be displayed with the detection result superimposed on the endoscopic image data 71, or may be displayed in different regions of the display 16.

[0137] As shown in FIG. 15, FIG. 16, or FIG. 17, the endoscopic image 101 based on the endoscopic image data 71 can be displayed in the endoscopic image area 103, and the detection result 91 of the region of interest by the first detector 61 based on the endoscopic image data 71 can be displayed in the detection result area 104 in different areas of the display 16. In the case shown in FIG. 15, the endoscopic image data 71 includes the region of interest 73, and the detection result 91 of the region of interest by the first detector 61 based on the endoscopic image data 71 is displayed by the detected region of interest 111 or the frame 102. In this case, the region of interest 73 is included in the endoscopic image data 71, and the detected region of interest 111 is also displayed in the detection result 91 by the first detector 61. Therefore, in this case, it is the case where the detection result of the first detector 61 is appropriate.

[0138] In the case shown in FIG. 16, the endoscopic image data 71 does not include the region of interest 73, and the detection result 91 of the region of interest by the first detector 61 based on the endoscopic image data 71 shows that the region of interest 73 is not detected. Also in this case, it is the case where the detection result of the first detector 61 is appropriate.

[0139] In the case shown in FIG. 17, the endoscopic image data 71 does not include the region of interest 73, but the detection result 91 of the region of interest by the first detector 61 based on the endoscopic image data 71 is displayed by the detected region of interest 111 or the frame 102 as detecting the region of interest 73. In this case, it is the case where it is displayed on the display 16 for the false positive endoscopic image data.

[0140] By displaying the detection result of the area of interest on the display 16 regardless of the content of the detection result, the doctor 76 can make his / her diagnosis more appropriate by looking at the displayed endoscopic image data 71 and its detection result. Also, in the case of Fig. 17, it can be evaluated that the detection result by the first detector 61 is a false positive detection result. The endoscopic image data 71 evaluated as a false positive can be used as learning data for the first detector 61. The endoscopic image data 71 evaluated as such a false positive can be used as learning data by uniformly attaching an annotation that the entire endoscopic image data 71 is a non-target area. As such an annotation, for example, an annotation such as "background" can be given to the entire false positive endoscopic image data 131.

[0141] As shown in Fig. 18, when using the false positive detection result as learning data, the information processing device 18 includes a second medical image dataset generation unit 121 in the learning data generation unit 21 and a second medical image dataset acquisition unit 122 in the learning unit 22.

[0142] The second medical image dataset generation unit 121 generates a second medical image dataset including the false positive endoscopic image data 131, which is the endoscopic image data evaluated by the doctor 76 as having a false positive detection result by the first detector 61, and an annotation that the false positive endoscopic image data 131 is a non-target area.

[0143] As a specific method, it can be the same as in the case of false negative. The specific information reception unit 54 (see Fig. 4) receives information for specifying the endoscopic image data 71 evaluated by the doctor 76 as having a false positive detection result. The second medical image dataset generation unit 121 generates a second medical image dataset by uniformly attaching an annotation such as "background" to the false positive endoscopic image data 131. The second medical image dataset is acquired by the second medical image dataset acquisition unit 122 and used for the learning of the first detector 61.

[0144] The first detector 61 performs learning using the second medical image data set including the false positive endoscopic image data. For example, in the first detector 61 adjusted in the direction of reducing false negative detection results, when the false positive detection result increases, etc., since it is possible to reduce the false positive detection result, the detection accuracy of the entire first detector 61 can be improved.

[0145] As described above, according to the information processing device 18, when detecting the attention area using the endoscopic image data 71, it is possible to efficiently improve the detection accuracy.

[0146] Next, the learning data generation device 141 will be further described. As shown in FIG. 19, the learning data generation device 141 is a device in which the learning data generation unit 21 is a single device, and the parts with the same reference numerals in the learning data generation unit 21 are the same as those described in the description of the learning data generation unit 21. Hereinafter, the differences from the learning data generation unit 21 described above will be described.

[0147] The learning data generation device 141 includes a storage unit 142. The storage unit 142 is preset in the learning data generation device 141 and stores the generated second medical image data set. The stored second medical image data set is used as learning data for the learning of the detector.

[0148] The learning data generation device 141 can automatically store learning data effective for improving the detection accuracy of the detector. The stored learning data can be used in various ways. For example, it can be conveniently used for the movement of the stored learning data or the processing of the learning data itself.

[0149] Next, the diagnostic support system 10 will be further described. The diagnostic support system 10 (see FIG. 1) includes the information processing device 18 and the detection device 13 described above. The parts with the same reference numerals in the information processing device 18 are the same as those described in the description of the information processing device 18. Hereinafter, the differences from the information information processing device 18 described above will be described.

[0150] As shown in FIG. 20, the diagnostic support system 10 includes a third detector generation unit 151 in the learning unit 22 of the information processing device 18. The third detector generation unit 151 acquires the first detector 61 that the first detector learning unit 58 has learned and uses it as the third detector 152.

[0151] As shown in FIG. 21, the detection device 13 includes the third detector 152. The detection device 13 uses the third detector 152 to detect a region of interest included in the inspection target shown in the endoscopic image data 71 based on the endoscopic image data 71, thereby generating diagnostic support information regarding the inspection target. Control such as displaying the generated diagnostic support information on the display 16 is performed.

[0152] On the display 16, the endoscopic image data 71 acquired by the endoscopic system and the diagnostic support information that is the detection result by the third detector 152 based on the endoscopic image data 71 are displayed (see FIGS. 14 to 17). The doctor 76 can diagnose the inspection target with reference to the endoscopic image data 71 and the diagnostic support information displayed on the display 16.

[0153] Note that after generating the third detector 152, the third detector generation unit 151 preferably replaces the third detector 152 with the first detector 61. After the third detector 152 is replaced with the first detector 61, it functions as the first detector 61. In this case, the first detector 61 is the third detector 152 constructed in the past. That is, the first detector 61 is automatically and continuously updated by the third detector 152. Therefore, in the diagnostic support system 10, since the information processing device 18 is provided, the third detector 152 is automatically and continuously improved in detection accuracy. Therefore, the diagnostic support system 10 is a system that can automatically and continuously detect a region of interest based on the endoscopic image data 71 by the third detector 152 with improved detection accuracy.

[0154] As described above, according to the diagnostic support system 10, the information processing device 18, the learning data generation device 11, or the like, a first medical image data set for automatically training the first detector is generated from the medical image data that is a false negative selected by the user. Then, even when the inference of the first detector is incorrect, a first medical image data set with more accurate annotations can be generated by the first detector that first detects the region of interest in the medical image data and a second detector different from the first detector. Further, by making the detection accuracy of the second detector higher than that of the first detector, an inference with high accuracy can be performed, and a more accurate first medical image data set can be created. Further, since the first medical image data set is particularly effective learning data for the first detector, the accuracy of the first detector can be more surely improved by retraining the first detector using the first medical image data set. Further, since the first medical image data set is automatically saved, it is possible to automatically save and store high-quality learning data. Further, since the detection accuracy for medical image data of the diagnostic support system 10 is automatically improved, even for a detector that has been constructed once, the detection accuracy can be improved without taking much effort, and it is possible to automatically and continuously improve the detection accuracy.

[0155] Note that there may be a plurality of displays 16, and a small portable terminal device such as a tablet (not shown) may be included. At the time of display, the screen layout and the like can be preset according to the device to be displayed.

[0156] In the above embodiment, the hardware structure of a processing unit such as a learning data generation unit 21, a learning unit 22 included in the information processing apparatus 18 which is a processor apparatus, or a control unit 31 or the like that executes various processes in a detection apparatus 13 or the like included in the diagnostic support system 10 is various processors as shown below. The various processors include a CPU (Central Processing Unit) which is a general-purpose processor that executes software (program) and functions as various processing units, a programmable logic device (PLD) such as an FPGA (Field Programmable Gate Array) which is a processor whose circuit configuration can be changed after manufacture, and an application specific electric circuit which is a processor having a circuit configuration specifically designed to execute various processes.

[0157] One processing unit may be constituted by one of these various processors, or may be constituted by a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA). Further, a plurality of processing units may be constituted by one processor. As an example of constituting a plurality of processing units by one processor, firstly, as represented by a computer such as a client or a server, one processor is constituted by a combination of one or more CPUs and software, and this processor functions as a plurality of processing units. Secondly, as represented by a System On Chip (SoC) or the like, there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. Thus, as a hardware structure, the various processing units are constituted by using one or more of the above various processors.

[0158] Furthermore, more specifically, the hardware structure of these various processors is an electric circuit in a form in which circuit elements such as semiconductor elements are combined.

Description of Reference Numerals

[0159] 10 Diagnostic support system 11, 141 Learning data generation device 12 Learning device 13 Detection device 14 Endoscope system 15 PACS 16 Display 17 Input device 18 Information processing device 21 Learning data generation section 22 Learning section 31 Control section 32 Communication section 33 Memory section 34 Data bus 35 Network 41 CPU 42 RAM 43 ROM 44 Program for learning data generation section 45 Data for learning data generation section 46 Program for learning section 47 Data for learning section 51 Medical image data acquisition section 52 First detection section 53 Display control section 54 Specific information reception section 55 Second detection section 56 First medical image dataset generation section 57 First medical image dataset acquisition section 58 First detector learning section 61 First detector 62 Second detector 62a Second detection device 62b Second detection device 62c Second detection device 63 Image adjustment section 71 Endoscope image data 72 Endoscope image data with no detected region of interest 73 Region of interest 74 Endoscopic image data of the area of interest 75 Non-detected area of interest 76 Physician 77 False negative endoscopic image data 78 Adjusted endoscopic image 79 Re-detection result 81 First medical image dataset 91 Detection result 101 Endoscopic image 102 Frame 103 Endoscopic image area 104 Detection result area 111 Detected area of interest 121 Second medical image dataset generation unit 122 Second medical image dataset acquisition unit 131 False positive endoscopic image data 142 Storage unit 151 Third detector generation unit 152 Third detector

Claims

1. Comprising a processor, The processor: Obtains medical image data in which an inspection target appears, Based on the medical image data, a first detector detects a region of interest included in the inspection target shown in the medical image data, Performs control to display, on a display, the medical image data for which a detection result indicates that the region of interest has not been detected, Receives information for specifying the medical image data for which the detection result has been evaluated by a user as being a false negative, Based on the medical image data evaluated as being a false negative, a second detector re-detects the region of interest included in the inspection target shown in the medical image data, Generates a first medical image data set including the medical image data and the re-detection result of the region of interest in the medical image data, Performs learning on the first detector using the first medical image data set, An information processing apparatus in which the computational load of the second detector is higher than the computational load of the first detector.

2. Comprising a processor, The processor: Obtains medical image data in which an inspection target appears, Based on the medical image data, a first detector detects a region of interest included in the inspection target shown in the medical image data, Performs control to display, on a display, the medical image data for which a detection result indicates that the region of interest has not been detected, Receives information for specifying the medical image data for which the detection result has been evaluated by a user as being a false negative, Based on the medical image data evaluated as being a false negative, a second detector re-detects the region of interest included in the inspection target shown in the medical image data, Generates a first medical image data set including the medical image data and the re-detection result of the region of interest in the medical image data, Performs learning on the first detector using the first medical image data set, The second detector is constructed using a machine learning algorithm, An information processing apparatus in which the number of parameters of the second detector is larger than the number of parameters of the first detector.

3. The information processing apparatus according to claim 1 or 2, wherein the second detector re-detects the region of interest based on medical image data having a higher resolution when compared with the first detector.

4. Comprising a processor, The processor: Obtains medical image data in which an inspection target appears, Based on the medical image data, a first detector detects a region of interest included in the inspection target shown in the medical image data, Perform control to display the medical image data, which is the detection result that the attention area was not detected, on the display. Receive information for identifying the medical image data evaluated by the user as a false negative in the detection result. Based on the medical image data evaluated as a false negative by the second detector, redetect the attention area included in the inspection target shown in the medical image data. Generate a first medical image data set including the medical image data and the redetection result of the attention area of the medical image data. Perform learning on the first detector using the first medical image data set. The second detector redetects the attention area based on each of the plurality of medical image data. The processor integrates the results of the plurality of redetections based on each of the plurality of medical image data and uses the result as the redetection result by the second detector. The second detector consists of a plurality. Each of the second detectors is an information processing device that redetects the attention area based on at least one of the plurality of medical image data.

5. The information processing device according to claim 4, wherein the plurality of medical image data includes medical image data having different resolutions from each other.

6. The information processing device according to claim 4 or 5, wherein the plurality of medical image data includes medical image data on which different image conversion processes have been performed.

7. The information processing device according to any one of claims 4 to 6, wherein the plurality of medical image data includes medical image data having different shooting times from each other.

8. The information processing device according to any one of claims 1 to 7, wherein the false negative rate of the redetection result of the second detector is lower than the false negative rate of the detection result of the first detector.

9. The first detector is pre-constructed by learning using an initial medical image data set for a machine learning algorithm. The information processing device according to any one of claims 1 to 8, wherein the processor performs learning on the first detector using the first medical image data set and the initial medical image data set.

10. The information processing device according to claim 9, wherein the processor performs weighting on each of the initial medical image data set and the first medical image data set and then performs learning on the first detector.

11. The information processing apparatus according to any one of claims 1 to 10, wherein the processor performs learning on the first detector after weighting each of the first medical image data sets included in the plurality of first medical image data sets.

12. The information processing apparatus according to any one of claims 1 to 11, wherein the medical image data included in the first medical image data set is acquired at a specific facility.

13. During the examination for acquiring the medical image data, the processor detects the region of interest by the first detector based on the medical image data, The information processing apparatus according to any one of claims 1 to 12, wherein information for specifying the medical image data evaluated as a false negative during the examination is received.

14. The information processing apparatus according to claim 13, wherein the processor re-detects the region of interest by the second detector based on the medical image data during the examination.

15. The information processing apparatus according to any one of claims 1 to 14, wherein the processor controls to display the medical image data and the detection result of the region of interest based on the medical image data on the display.

16. The processor receives information for specifying the medical image data evaluated as a false positive by a user, generates a second medical image data set including the medical image data evaluated as a false positive and an annotation that the medical image data is a non-region of interest, and performs learning on the first detector using the second medical image data set.

17. Comprising a processor, The processor, acquires medical image data in which an examination subject is imaged, detects, by a first detector, a region of interest included in the examination subject imaged in the medical image data based on the medical image data, controls to display, on a display, the medical image data for which the detection result is that the region of interest was not detected, receives information for specifying the medical image data evaluated as a false negative by a user, re-detects, by a second detector, the region of interest included in the examination subject imaged in the medical image data based on the medical image data evaluated as a false negative, and generates a medical image data set including the medical image data and the re-detection result of the region of interest associated with the medical image data. Save the medical image dataset in a preset storage unit. A learning data generation device in which the computational load of the second detector is higher than that of the first detector.

18. Equipped with a processor. The processor Acquires medical image data in which the inspection target appears. Using a first detector, detects a region of interest included in the inspection target that appears in the medical image data based on the medical image data. Controls the display to display the medical image data for which the detection result is that the region of interest has not been detected. Receives information for specifying the medical image data evaluated by the user as having a false negative detection result. Using a second detector, redetects the region of interest included in the inspection target that appears in the medical image data based on the medical image data evaluated as having a false negative. Generates a medical image dataset including the medical image data and the redetection result of the region of interest associated with the medical image data. Generates a third detector by performing learning on the first detector using the medical image dataset. Using the third detector, generates diagnostic support information regarding the inspection target by detecting the region of interest included in the inspection target that appears in the medical image data based on the medical image data. A diagnostic support system in which the computational load of the second detector is higher than that of the first detector.

19. The diagnostic support system according to claim 18, wherein the first detector is the third detector constructed in the past.

Images