Machine Learning Device

Abstract

A machine learning device includes: a processor that processes data samples; and a storage device that stores processing results. The processor generates multiple classifiers based on multiple learning databases. The multiple learning databases each store multiple learning data samples. The processor generates evaluation results of the recognition performance of each of the multiple classifiers and, based on the evaluation results, determines one of the multiple learning databases and a classifier generated based on the one learning database as the learning database and classifier to be used.

Description

[0001] Reference-based import

[0002] This application claims priority from Japanese Patent Application No. 2019-181165, filed on October 1, 2019, the contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to machine learning and can be applied to, for example, image processing technology using machine learning for detecting specific objects (eg, cancer cells, bubbles on a liquid surface, etc.) included in a captured image. Background Art

[0004] In recent years, research on image recognition technologies using machine learning and other techniques has been conducted. Deep learning and other techniques have been used to improve the accuracy of detecting objects within images. For example, Patent Document 1 proposes a technique for developing a classifier that detects objects within images. In Patent Document 1, multiple sets of training image data are set to perform machine learning and calculate the parameters of a neural network.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2016-143351 Summary of the Invention

[0008] Problems to be solved by the invention

[0009] However, even if the training images are divided into multiple image groups and retrained to obtain parameters, as in Patent Document 1, the image groups may sometimes contain images that do not contribute to improving the recognition accuracy of the classifier, and the recognition accuracy of the classifier may not necessarily improve. In addition, Patent Document 1 fails to create a training image database that can continuously improve the recognition accuracy of the classifier.

[0010] Means for solving problems

[0011] A machine learning device according to one embodiment of the present invention includes: a processor configured to process data samples; and a storage device configured to store the results of the processing. The processor generates multiple classifiers based on multiple learning databases, each of which stores multiple learning data samples, generates evaluation results of the recognition performance of each of the multiple classifiers, and, based on the evaluation results, determines one of the multiple learning databases and the classifier generated based on the one learning database as the learning database and classifier to be used.

[0012] Other features related to the present invention will become apparent from the description and drawings of this specification. Furthermore, the present invention is achieved through the elements and combinations of various elements, as well as the detailed description below and the accompanying technical solutions. The descriptions in this specification are merely typical examples and should not be understood as limiting the technical solutions or application examples of the present invention in any way.

[0013] Effects of the Invention

[0014] According to one aspect of the present invention, it is possible to create and use an appropriate learning database and a classifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram showing the functions of the machine learning device according to the first embodiment.

[0016] Figure 2A This is a diagram showing a hardware configuration example of the machine learning device according to the first embodiment.

[0017] Figure 2B This is a diagram showing a configuration example of a learning unit according to the first embodiment.

[0018] Figure 3 This is a diagram for explaining an example of the operation of the learning unit according to the first embodiment.

[0019] Figure 4 This is a diagram for explaining an example of the operation of the learning unit according to the first embodiment.

[0020] Figure 5 This is a diagram for explaining an example of the operation of the learning unit according to the first embodiment.

[0021] Figure 6 This is a diagram for explaining an example of the operation of the learning unit according to the first embodiment.

[0022] Figure 7 This is a diagram for explaining an example of the operation of the learning unit according to the first embodiment.

[0023] Figure 8 This is a diagram for explaining an example of the operation of the drawing unit according to the first embodiment.

[0024] Figure 9 This is a flowchart for explaining the operation of the learning unit according to the first embodiment.

[0025] Figure 10 This is a flowchart for explaining the overall operation of the machine learning device according to the first embodiment.

[0026] Figure 11This is a diagram for explaining an example of the update status display of the drawing unit according to the first embodiment.

[0027] Figure 12 This is a block diagram showing the functions of the machine learning device according to the second embodiment.

[0028] Figure 13 This is a flowchart for explaining the overall operation of the machine learning device according to the second embodiment.

[0029] Figure 14 This is a diagram showing a schematic configuration of a remote diagnosis support system equipped with an image diagnosis support device including a machine learning device according to a third embodiment.

[0030] Figure 15 This is a diagram showing a schematic configuration of a network entrusted service providing system equipped with an image diagnosis supporting device including a machine learning device according to a fourth embodiment. DETAILED DESCRIPTION

[0031] In one embodiment, machine learning is performed using images in a learning image database to create multiple recognizers. The multiple recognizers are evaluated to obtain evaluation results, and the multiple evaluation results are used to control whether updates to the recognizers and the learning image database are permitted. Thus, a machine learning device and method for creating a learning image database is provided. The learning image database comprises recognizers capable of highly accurately identifying objects (e.g., tissues and cells) within images, and images that contribute to improving the recognition accuracy of the recognizers.

[0032] The following describes embodiments of the present invention with reference to the accompanying drawings. Functionally identical elements are sometimes designated by the same reference numerals in the accompanying drawings. While the accompanying drawings illustrate specific embodiments and implementation examples based on the principles of the present invention, these are provided for understanding the present invention and are not intended to limit the present invention in any way.

[0033] In the present embodiment, those skilled in the art have described in sufficient detail for the purpose of implementing the present invention, but it should be understood that other installations and methods are also possible, and that changes in structure and construction, and replacement of various elements can be made without departing from the scope and spirit of the technical concept of the present invention. Therefore, the following description should not be interpreted as being limited to this.

[0034] Furthermore, as described later, the embodiments of the present invention may be implemented as software running on a general-purpose computer, or as dedicated hardware, or as a combination of software and hardware. Below, the various processes in the embodiments of the present invention are described with "each processing unit (e.g., a learning unit, etc.) as the subject (action subject). However, since the processes determined by the execution of the program by a processor (e.g., a CPU) are performed while using a memory and a communication port (communication control device), the description may also be based on the processor as the subject.

[0035] (1) First embodiment

[0036] <Functional Structure of Machine Learning Device>

[0037] Figure 1 This is a block diagram showing the functional structure of a machine learning device according to a first embodiment. The machine learning device 1 includes an input unit 10, a learning unit 11, an adaptability evaluation unit 12, an update determination unit 13, a rendering unit 14, a control unit 91, a learning image database (DB) (confirmation) 100, a learning image DB (confirmation + pre-confirmation) 101, evaluation images 102, and a memory 90. The machine learning device 1 can be implemented in an image acquisition device such as an image diagnosis support device, or, as described later (in the third and fourth embodiments), in a server connected to the image acquisition device via a network.

[0038] The input unit 10 , learning unit 11 , adaptability evaluation unit 12 , update determination unit 13 , and drawing unit 14 in the machine learning device 1 may be implemented by a program or a processor executing the program, or may be implemented by a hardware module.

[0039] Image data is input to the input unit 10. For example, the input unit 10 may also obtain the image data from a built-in image acquisition device (in Figure 1 The input image is encoded still image data in JPG, Jpeg2000, PNG, BMP format, etc., captured at predetermined time intervals by a camera or other imaging unit (not shown in the figure).

[0040] The input unit 10 may also extract still image data of frames at predetermined intervals from dynamic image data in formats such as MotionJPEG, MPEG, H.264, or HD / SDI, and obtain these images as input images. The input unit 10 may obtain input images from a camera unit via a bus, network, or the like. The input unit 10 may also obtain images already stored on a removable recording medium as input images. Images input from the input unit 10 are stored as learning images (before finalization) in the learning image DB (finalization + before finalization) 101.

[0041] The learning image database (determined) 100 stores multiple combinations of learning images, determined images, and correct labels. The correct label associated with each image is determined. Learning images (before determination) consist of multiple combinations of learning images, images before determination, and correct labels. The correct label associated with each image can be changed.

[0042] The learning unit 11 performs machine learning to identify an image of a specific object within an input image as that specific object, for example, to identify an image of normal tissue or cells as normal tissue or cells, and to identify an image of abnormal tissue or cells within an input image as abnormal tissue or cells. The learning unit 11 creates a classifier CA (with parameters required for recognition (filter coefficients, offset values, etc.)) based on the learning image DB (determined) 100. The learning unit 11 creates a classifier CB based on a learning image DB (determined + before determination) 101, which is obtained by adding the learning image (before determination) input from the input unit 10 to the learning image DB (determined) 100.

[0043] The adaptability evaluation unit 12 calculates the recognition results and recognition values ​​of the classifiers CA and CB using the evaluation image 102. The update determination unit 13 uses the recognition results and recognition values ​​of the classifiers CA and CB obtained by the adaptability evaluation unit 12 to control whether the classifier CA and the learning image DB (determination) 100 can be updated. In addition, the update determination unit 13 uses the Ave obtained by the adaptability evaluation unit 12 to determine whether the update is possible. dr1 、Ave dr2 、each M drN , the update of the classifier CA determined by the update determination unit 13, the number of updates of the learning image DB (determined), and the updated Ave dr1 Information such as the progress of the data is stored in the memory 90.

[0044] The drawing unit 14 converts Ave, which is obtained by the adaptability evaluation unit 12 and will be described later, into dr1 、Ave dr2 、each M drN , the update of the classifier CA determined by the update determination unit 13, the number of updates of the learning image DB (determined), and the updated Ave dr1 The information such as the progress of the data is output to an output device such as a display or a printer.

[0045] The control unit 91 is implemented by, for example, a processor that executes a program, and is connected to each component in the machine learning device 1. Each component of the machine learning device 1 operates as described above, either autonomously or in response to an instruction from the control unit 91.

[0046] Thus, in the machine learning device 1 of this embodiment, the learning unit 11 performs machine learning to generate a classifier CA based on the learning image DB (determined) 100, and creates a classifier CB based on the learning image DB (determined + before determination) 101 obtained by adding the learning image (before determination) input from the input unit 10 to the learning image DB (determined) 100. The adaptability evaluation unit 12 calculates the recognition results and recognition values ​​of the classifiers CA and CB using the evaluation image 102. The update determination unit 13 uses the recognition results and recognition values ​​of the classifiers CA and CB obtained by the adaptability evaluation unit 12 to control whether to update the classifier CA and the learning image DB (determined) 100.

[0047] <Hardware Structure of Machine Learning Device>

[0048] Figure 2A This diagram shows an example of the hardware configuration of a machine learning device 1 according to the first embodiment. The machine learning device 1 includes a CPU (processor) 201 that executes various programs, a memory 202 (main storage device) that stores various programs, and an auxiliary storage device 203 (equivalent to memory 90) that stores various data. The machine learning device 1 also includes an output device 204 for outputting recognition results, a recognizer, and an update permission / disapproval decision for the learning image DB (determination) 100; an input device 205 for inputting user instructions, images, and the like; and a communication device 206 for communicating with other devices. These components within the machine learning device 1 are interconnected via a bus 207.

[0049] The CPU 201 reads and executes various programs from the memory 202 as needed. The memory 202 stores the input unit 10, the learning unit 11, the adaptability evaluation unit 12, the update determination unit 13, and the drawing unit 14 as programs.

[0050] The auxiliary storage device 203 stores learning images (before finalization), parameters of classifiers CA and CB generated by the learning unit 11, recognition results and recognition values ​​generated by the adaptability evaluation unit 12, and update results determined by the update determination unit 13. The auxiliary storage device 203 also stores learning image DB (finalization) 100, learning image DB (finalization + before finalization) 101, and position information used for drawing detection frames generated by the drawing unit 14. The memory 202, the auxiliary storage device 203, or a combination thereof constitutes a storage device.

[0051] The output device 204 is configured to include devices such as a display, a printer, and a speaker. For example, the output device 204 is a display device that displays data generated by the rendering unit 14 on a screen. The input device 205 is configured to include devices such as a keyboard, a mouse, and a microphone. User instructions (including decisions regarding input of a learning image (before finalization)) are input to the machine learning device 1 via the input device 205.

[0052] Communication device 206 is not essential to machine learning apparatus 1. If a communication device is included in a personal computer or the like connected to the image acquisition device, machine learning apparatus 1 may not include communication device 206. Communication device 206 receives data (including images) transmitted from another device (e.g., a server) connected via a network and stores the data in auxiliary storage device 203.

[0053] The machine learning device of this embodiment performs machine learning using images from a learning image database to create multiple recognizers, and then evaluates the multiple recognizers to obtain evaluation results. The machine learning device determines the evaluation results and controls whether to update the recognizers and the learning image database. This allows the creation of a learning image database composed of recognizers that can more accurately recognize objects (such as tissues and cells) within images, as well as images that contribute to the continued improvement of the recognizers' recognition accuracy.

[0054] <Structure and Movement of Each Part>

[0055] The structure and operation of each element will be described in detail below.

[0056] (i) Learning Department 11

[0057] Figure 2B FIG. 1 shows a configuration example of the learning unit 11 . The learning unit 11 includes a feature extraction unit 111 , a local recognition unit 112 , and a global recognition unit 113 .

[0058] (ii) Feature Extraction Unit 111

[0059] The feature extraction unit 111 obtains the feature quantity of the input image. Figure 3 An example of obtaining feature quantities is shown. Figure 3 CNN stands for Convolutional Neural Network. For example, the feature extraction unit 111 uses a feature extractor FEA that performs the calculation of Formula 1 to obtain a feature value FAi of an object (eg, tissue or cell) in the input image Ai from the input image Ai.

[0060] [Mathematical formula 1]

[0061]

[0062] The learning unit 11 uses machine learning to determine the filter coefficients wj to identify each object image as the object (e.g., to identify normal tissue or normal cells as normal tissue or normal cells, or to identify abnormal tissue or abnormal cells as abnormal tissue or abnormal cells). pj represents the pixel value, bi represents the offset value, m represents the number of filter coefficients, and h represents the nonlinear function.

[0063] like Figure 4 As shown, the feature extraction unit 111 uses Formula 1 to calculate the calculation results of each filter 42 from the upper left to the lower right of the target image (for example, a pathological tissue image) 41. This calculation results in the feature quantity fi of any filter i. For example, the matrix of feature quantities fi calculated by the feature extractor A is set as the feature quantity FAi of the input image Ai. The method for creating the feature extractor FEA will be described later.

[0064] (i-ii) Local identification unit 112

[0065] like Figure 5 As shown, the local identification unit 112 uses the feature quantity FAi of the feature extractor FEA obtained by the feature extraction unit 111 and the nonlinear function NF (e.g., a sigmoid function) to calculate the object similarity value (e.g., lesion similarity) for each local area according to Formula 2. Based on the calculated value, the local identification unit 112 determines whether the object in the input image Ai is an object to be detected (e.g., a normal cell or an abnormal cell).

[0066] [Mathematical formula 2]

[0067]

[0068] In Formula 2, LS is a local identification value consisting of a three-dimensional array of level, height, and width, FAi is a feature value consisting of a three-dimensional array of the feature number, height, and width obtained by the feature extraction unit 111, W is a filter used to calculate a local identification value consisting of a four-dimensional array of level, feature number, height, and width, and B is an offset value used to calculate a local identification value consisting of a one-dimensional array of levels. c represents the index of the level, y represents the vertical index of the feature, x represents the horizontal index of the feature, fy represents the vertical index of the filter, fx represents the horizontal index of the filter, and j represents the index of the filter.

[0069] In Formula 2, the local recognition value is calculated using convolution processing, but the calculation method of the local recognition value is not limited to this. For example, the local recognition value can be obtained by applying convolution processing or nonlinear functions multiple times, or the feature value at each coordinate can be input into other recognition methods such as random forest and support vector machines to calculate the local recognition value.

[0070] (i-iii) Overall recognition unit 113

[0071] like Figure 6As shown, the global recognition unit 113 calculates a basic recognition value BS using the local recognition value LS calculated by the local recognition unit 112 and a nonlinear function (e.g., a sigmoid function). The global recognition unit 113 uses the basic recognition value BS as a calculation result R representing the object similarity (e.g., lesion similarity) of each object image in the input image, and determines whether the object in the input image Ai is an object to be detected (e.g., a normal cell or an abnormal cell, etc.) (S1).

[0072] The global recognition value GS of Formula 3 and the basic recognition value BS are calculated using Formula 4.

[0073] [Mathematical formula 3]

[0074]

[0075] [Formula 4]

[0076]

[0077] In Formula 3, GS represents the global recognition value, which is a one-dimensional array of levels; FAi represents the feature quantity, which is a three-dimensional array of the feature quantity number, height, and width obtained by the feature extraction unit 111; W represents the filter used to calculate the global recognition value, which is a four-dimensional array of the level, feature quantity number, height, and width. B represents the offset value used to calculate the global recognition value, which is a one-dimensional array of levels; c represents the level index; y represents the vertical index of the feature quantity; x represents the horizontal index of the feature quantity; fy represents the vertical index of the filter; fx represents the horizontal index of the filter; and j represents the filter index.

[0078] In addition, Label in Formula 4 represents the teacher label (correct answer label) for each image, which is a one-dimensional array of levels. The learning unit 11, described later, calculates the coefficients of the updated filter W and the offset value B in Formula 3 through machine learning. NLL represents a loss function, such as negative log likelihood.

[0079] Formula 3 uses convolution processing and horizontal and vertical averaging to calculate the global recognition value, but the method for calculating the global recognition value is not limited to this. For example, horizontal and vertical averaging can be performed after applying convolution processing or nonlinear functions multiple times. Alternatively, the horizontal and vertical averages of the values ​​obtained by inputting the feature values ​​at each coordinate into other recognition methods such as Random Forest or Support Vector Machine (SVM) can be calculated. Furthermore, the method is not limited to horizontal and vertical averaging; summation can also be used.

[0080] The learning unit 11 uses existing machine learning techniques to learn the feature values ​​of each object so that the overall recognition unit 113 can use the local recognition values ​​to recognize each object in the input image as a respective object (for example, normal tissue and normal cells as normal tissue and normal cells, and abnormal tissue and abnormal cells as abnormal tissue and abnormal cells), and calculate the coefficients of the filter W and the offset value B. For example, a Convolutional Neural Network can be used as a machine learning technique.

[0081] like Figure 7 As shown, the learning unit 11 uses an input image Ai (e.g., a pathological image) through prior machine learning and calculates a feature value FAi of the input image Ai using Formula 1. Next, the learning unit 11 calculates a local classification value LS based on the feature value FAi using Formula 2. Using the basic classification value BS calculated based on the local classification value LS, the learning unit 11 calculates the parameters of Formulas 1 to 3 to identify the image of each target object as the target object (e.g., identifying abnormal tissue or abnormal cells as abnormal tissue or abnormal cells, and identifying normal tissue or normal cells as normal tissue or normal cells).

[0082] The learning unit 11 uses multiple learning images of the learning image DB (determined) 100, and repeatedly performs processing by the feature extraction unit 111, the local recognition unit 112 and the overall recognition unit 113 to obtain the parameters shown in Formulas 1, 2 and 3 (filter coefficient wj, coefficient of filter W, offset value bi and B, etc.).

[0083] The learning unit 11 creates a classifier CA, which is composed of a feature extractor that calculates the input image's feature quantity, a local classifier that calculates local recognition values, and a global classifier that calculates global recognition values. Similarly, the learning unit 11 uses the multiple learning images in the learning image DB (determined + before determination) 101 to calculate the parameters (such as the filter coefficients wj, the coefficients of filter W, the offset values ​​bi and B) shown in Formulas 1, 2, and 3, and creates a classifier CB. The learning unit 11 stores the calculated parameters (such as the filter coefficients wj, the coefficients of filter W, the offset values ​​bi and B) in the memory 90.

[0084] For example, the learning unit 11 adjusts the balance between the number of images of each recognition type in the learning image DB (confirmed) 100 and the learning image DB (confirmed + before confirmation) 101 to create classifiers CA and CB. For example, the learning unit 11 adjusts the number of images in each of the learning image DB (confirmed) 100 and the learning image DB (confirmed + before confirmation) 101 so that the difference in the number of images between the recognition types is less than a predetermined threshold. This enables more appropriate evaluation.

[0085] (ii) Adaptability evaluation unit 12

[0086] The adaptability evaluation unit 12 uses the evaluation image 102 and calculates the Ave of the classifier CA according to Formula 5 for the classifiers CA and CB created by the learning unit 11. drX (Ave dr1 ) and Ave of the recognizer CB drX (Ave dr2 ), and thus, the evaluation of these identifiers is performed. In Formula 5, N represents the number of types to be identified. Each M drN Indicates the detection rate of the object. For example, N = 2, M dr1 represents the benign tumor detection rate (average), M dr2 The adaptability evaluation unit 12 can use the same drX Different values, for example, M drN The maximum value of .

[0087] [Formula 5]

[0088] Ave drX =(M dr1 +…+M drN ) / N…Formula 5

[0089] (iii) Update determination unit 13

[0090] The update determination unit 13 updates the plurality of Ave obtained by the adaptability evaluation unit 12. drX Comparison is made and the update of the recognition unit CA and the learning image DB (determination) 100 is controlled. For example, in Ave dr2 >Ave dr1 and Ave dr2 > TH1 (for example, TH1 = 0.7) (K1), the update determination unit 13 updates the classifier CA and the training image DB (determined) 100. Specifically, the update determination unit 13 updates the contents of the training image DB (determined) 100 with the contents of the training image DB (determined + before determination) 101, and updates the classifier CA with the contents of the classifier CB learned using the training image DB (determined + before determination) 101.

[0091] In addition, in Ave dr2 ≤Ave dr1 and Ave dr2> TH1 (K2), the update determination unit 13 does not update the identifier CA and the learning image DB (determined), and randomly replaces the order of the images in the learning image DB (before determination). The learning unit 11 performs re-learning using the learning image DB (before determination) with the replaced order. In the case of neither K1 nor K2, the update determination unit 13 changes the correct labels of several images in the learning image DB (before determination) for re-learning based on the user's designation via the input device or automatically. For example, the update determination unit 13 may also determine whether the correct labels have been changed for a small batch of images.

[0092] The update determination unit 13 updates the Ave obtained by the adaptability evaluation unit 12. dr1 、Ave dr2 、each M drN , the update of the classifier CA determined by the update determination unit 13, the update number of the learning image DB (determined) 100, and the updated Ave dr1 The progress of the data is recorded in the memory 90 or in a log file.

[0093] (iv) Drawing section 14

[0094] As an example, the drawing unit 14 Figure 11 The GUI (Graphical User Interface) shown in FIG. 1 is used to display the Ave obtained by the adaptability evaluation unit 12. dr1 、Ave dr2 、each M drN , the identifier CA determined by the update determination unit 13, the number of updates of the learning image DB (determined), and the updated Ave dr1 About Figure 11 For example, Ave dr1 Displayed as 0.71, Ave dr2 It is displayed as 0.81, and M dr1 Displayed as 0.80, M dr2 It is displayed as 0.82 and the number of updates is displayed as 10.

[0095] In addition, the drawing unit 14 displays the recognition results of each recognizer of the unknown image input from the input unit 10. When a specific portion in the image is determined to be an object to be detected (for example, abnormal tissue, abnormal cells), as shown in FIG. Figure 8 As shown, the drawing unit 14 may draw a detection frame 82 within the input target image 81 to indicate the portion of the object to be detected (e.g., a portion suspected of having abnormal tissue or abnormal cells). On the other hand, if the target image 81 is determined to be normal tissue or normal cells, the drawing unit 14 may not draw the detection frame 82 on the input target image 81 and may display the input target image 81 as it is.

[0096] In addition, if Figure 8 As shown, the rendering unit 14 displays the result of the determined object similarity (e.g., tumor) 83. The rendering unit 14 is not an essential component of the machine learning device 1. If the image diagnosis support device includes the rendering unit, the machine learning device 1 may not have the rendering unit 14.

[0097] <Processing Process of Machine Learning Device>

[0098] Figure 9 This is a flowchart for explaining the operation of the learning unit 11 of the machine learning device 1 according to the first embodiment. Hereinafter, the learning unit 11 is described as the main operating unit, but the CPU 201 may be the main operating unit, and the CPU 201 may execute each processing unit as a program.

[0099] (i) Step 901

[0100] The input unit 10 receives a learning input image Ai and outputs the input image Ai to the learning unit 11 .

[0101] (ii) Step 902

[0102] The learning unit 11 uses machine learning to obtain the feature quantity of the object (e.g., tissue, cell, etc.) in the input image Ai using a filter according to the above formula 1, and creates a feature extractor FEA. The learning unit 11 obtains the filter coefficient wj and the offset value bi for the feature quantity FAi.

[0103] (iii) Step 903

[0104] The learning unit 11 uses machine learning to calculate the local recognition value LS based on the feature value FAi according to Formula 2, calculates the object similarity (for example, lesion similarity, etc.) according to the local area, and calculates the various parameters of Formula 2 used to calculate the local recognition value (coefficient of the filter W, offset value B, etc.) to determine whether the object in the input image Ai is an object to be detected (for example, a normal cell or an abnormal cell, etc.).

[0105] (iv) Step 904

[0106] The learning unit 11 uses the basic recognition value BS obtained based on the local recognition value LS through machine learning to obtain the parameters of Formula 3 (coefficients of the filter W, offset value B, etc.) to determine the image of each target object as the target object (for example, abnormal tissue and abnormal cells are determined as abnormal tissue and abnormal cells, and normal tissue and normal cells are determined as normal tissue and normal cells).

[0107] (v) Step 905

[0108] The learning unit 11 stores the parameters of Formula 1, Formula 2, and Formula 3 (filter coefficient wj, coefficient of filter W, offset values ​​bi, B, etc.) in the memory 90 .

[0109] Figure 10 This is a flowchart for explaining the operation of the machine learning device 1 according to this embodiment. The following description uses the processing units (input unit 10, learning unit 11, etc.) as the main operating units. However, the CPU 201 can also be used as the main operating unit, with the CPU 201 executing the processing units as programs.

[0110] (i) Step 1001

[0111] The input unit 10 outputs the input image Ai of the learning image DB (finalization+before finalization) to the learning unit 11 .

[0112] (ii) Step 1002

[0113] The learning unit 11 reads the parameters of Formula 1, Formula 2, and Formula 3 related to the classifier CA from the memory 90. The learning unit 11 also performs machine learning using the learning image DB (determined + before determination) 101 to calculate the parameters of Formula 1, Formula 2, and Formula 3 related to the classifier CB.

[0114] (iii) Step 1003

[0115] The adaptability evaluation unit 12 calculates the Ave of the classifier CA according to Formula 5 using the parameters of the classifiers CA and CB and the evaluation image. dr1 and Ave of the identifier CB dr2 .

[0116] (iv) Step 1004

[0117] The update determination unit 13 calculates Ave dr1 With Ave dr2 Compare. dr2 >Ave dr1 In the case of Ave, the update determination unit 13 transfers to step 1006. On the other hand, when the calculation result is Ave dr2 ≤Ave dr1 In the case of , the update determination unit 13 transfers to step 1006.

[0118] (v) Step 1005

[0119] On Ave dr2 >TH1, the update determination unit 13 proceeds to step 1007. On the other hand, in the case of Ave dr2 If ≤ TH1, the update determination unit 13 proceeds to step 1008 .

[0120] (vi) Step 1006

[0121] On Ave dr2 >TH1, the update determination unit 13 proceeds to step 1008. On the other hand, in the case of Ave dr2 If ≤ TH1, the update determination unit 13 proceeds to step 1009 .

[0122] (vii) Step 1007

[0123] The update determination unit 13 updates the classifier CA and the learning image DB (determination) 100 .

[0124] (viii) Step 1008

[0125] The update determination unit 13 changes the order of images in the learning image DB (determined+before-determined).

[0126] (ix) Step 1009

[0127] The update determination unit 13 replaces the correct answer label in the learning image DB (before finalization).

[0128] (x) Step 1010

[0129] The update determination unit 13 checks whether the update determination is completed for all images in the learning image DB (determined + before determination), and if so, moves to step 1011. On the other hand, if not, the update determination unit 13 returns to step 1002 and repeats steps 1002 to 1009.

[0130] (xi) Step 1011

[0131] The update determination unit 13 stores the information of the identifier CA in the memory 90 (equivalent to the auxiliary storage device 203 ).

[0132] As described above, even in the case where the learning image database contains images that do not contribute to improving the recognition accuracy of the identifier, the machine learning device automatically determines images that contribute to improving the recognition accuracy of the identifier, and controls whether the identifier and the learning image database can be updated based on the determination result. More specifically, the machine learning device uses the images of multiple learning image databases to perform machine learning to produce multiple identifiers, and evaluates the multiple identifiers produced to obtain evaluation results. The machine learning device determines the multiple evaluation results to control whether the identifier and the learning image database can be updated, and determines the learning image database and identifier to be used. In this way, a learning image database can be produced, which is composed of identifiers that can recognize objects (for example, tissues and cells, etc.) in an image with high accuracy and images that contribute to improving the continuous recognition accuracy of the identifier.

[0133] Furthermore, even if the input training images include images that do not contribute to improving the recognition accuracy of the recognizer, these images can be excluded to create a training image database. Furthermore, even if the input training images do not contribute to improving the recognition accuracy of the recognizer at that point in time, images that contribute to improving the recognition accuracy of the recognizer can be used by retraining the training images in a different order.

[0134] (2) Second embodiment

[0135] Hereinafter, a second embodiment will be described. Figure 12 The machine learning device 1 of the second embodiment shown includes multiple Figure 1 The same structural elements are included in the first embodiment, but instead of the learning image DB (determined + before determination) 101, it includes a learning image DB (before determination) 201 and an update determination unit 23. Figure 1 Different structures are described.

[0136] The machine learning device 1 of this embodiment uses each image in the learning image database to perform machine learning to create multiple identifiers, and then evaluates the multiple identifiers to obtain evaluation results. The machine learning device 1 determines the multiple evaluation results to control whether the identifiers and the learning image database can be updated or created. This makes it possible to create a learning image database suitable for each facility or each period, for example, which is composed of identifiers that can accurately identify objects (e.g., tissues and cells) within images and images that help improve the identifier's continued recognition accuracy.

[0137] <Structure and Movement of Each Part>

[0138] The following is about Figure 1 The structure and operation of each element are explained in detail.

[0139] (i) Learning image DB (before determination) 201

[0140] The learning image DB (before finalization) 201 stores images input from the input unit 10 and does not store other images.

[0141] (ii) Update determination unit 23

[0142] The learning unit 11 creates a classifier CA based on the learning image DB (determined) 100 and creates a classifier CB based on the learning image DB (before determination) 201. The adaptability evaluation unit 12 uses the evaluation image to obtain the Ave of the classifier CA. dr1 and Ave of the identifier CB dr2, thereby evaluating the discriminator. The update determination unit 23 updates the plurality of Ave obtained by the adaptability evaluation unit 12. drX The comparison is performed to control whether or not the classifiers CA and CB, the learning image DB (determined) 100, and the learning image DB (before determination) 201 are updated or created.

[0143] That is, for all images of the evaluation image, in Ave dr2 >Ave dr1 In this case, a classifier CB created from, for example, a learning image DB (before determination) 201 collected at another facility or at another time is more suitable for recognizing the evaluation image than a classifier CA created from a previously collected learning image DB (determined) 100. Therefore, the update determination unit 23 stores the classifier CB and the learning image DB (before determination) separately from the classifier CA and the learning image DB (determined) and sets them together with the evaluation image.

[0144] On Ave dr2 ≤Ave dr1 In this case, the classifier CA created based on the previously collected learning image DB (determined) is more suitable for recognizing the evaluation image than the classifier CB created based on the learning image DB (determined) collected at another facility or at another time. Therefore, the update determination unit 23 stores the classifier CA and the learning image DB (determined) together with the evaluation image.

[0145] <Hardware Structure of Machine Learning Device>

[0146] The hardware configuration example of the machine learning device 1 of this embodiment has the same configuration as that of FIG. 2 , but differs from the machine learning device 1 of the first embodiment in that an update determination unit 23 is included in the memory 202 .

[0147] The auxiliary storage device 203 of the machine learning device 1 stores the calculation result Ave obtained by the adaptability evaluation unit 12. drX , the identifiers CA and CB determined by the update determination unit 23, the learning image DB (determined) 100, the learning image DB (before determination) 201 and the evaluation image, and the parameters of Formula 1, Formula 2 and Formula 3 generated by the learning unit 11, etc.

[0148] Figure 13 This is a flowchart for explaining the operation of the machine learning device 1 according to this embodiment. The following description uses the processing units (input unit 10, learning unit 11, etc.) as the main operating units. However, the CPU 201 can also be used as the main operating unit, with the CPU 201 executing the processing units as programs.

[0149] (i) Step 1301

[0150] The input unit 10 outputs the input image Ai of the learning image DB (before finalization) 201 to the learning unit 11 .

[0151] (ii) Step 1302

[0152] The learning unit 11 reads the parameters of Formulas 1, 2, and 3 related to the classifier CA from the memory 90. It also performs machine learning using the learning image DB (before determination) 201 to calculate the parameters of Formulas 1, 2, and 3 related to the classifier CB.

[0153] (iii) Step 1303

[0154] The adaptability evaluation unit 12 calculates the Ave of the classifier CA according to Formula 5 using the parameters of the classifiers CA and CB and the evaluation image. dr1 and Ave of the identifier CB dr2 .

[0155] (iv) Step 1304

[0156] The update determination unit 13 calculates Ave dr1 With Ave dr2 Compare. dr2 >Ave dr1 In the case of Ave, the update determination unit 13 transfers to step 1305. On the other hand, in dr2 ≤Ave dr1 In the case of , the update determination unit 13 transfers to step 1306.

[0157] (v) Step 1305

[0158] The update determination unit 13 updates the classifier CB, the learning image DB (before determination) 201, the evaluation image, and the calculation result (Ave dr2 、Ave dr1 ) are stored in groups in the memory 90 (equivalent to the auxiliary storage device 203).

[0159] (vi) Step 1306

[0160] The update determination unit 13 updates the classifier CA, the learning image DB (determined) 100, the evaluation image, and the calculation result (Ave dr2 、Ave dr1 ) are stored in groups in the memory 90 (equivalent to the auxiliary storage device 203).

[0161] (vii) Step 1307

[0162] The update determination unit 13 checks whether the update determination is completed for all images in the learning image DB (before finalization) 201. If completed, the update determination unit 13 ends the process. On the other hand, if not completed, the update determination unit 13 returns to step 1302 and repeats steps 1302 to 1306.

[0163] The second embodiment uses machine learning to create multiple identifiers using images from multiple learning image databases. These identifiers are then evaluated to determine evaluation results. The second embodiment uses these evaluation results to control whether the identifiers and learning image databases can be updated or created, thereby determining the learning image database and identifier to use. This allows for the creation of identifiers and learning image databases that can accurately identify objects (e.g., tissues and cells) within images of each facility.

[0164] Furthermore, by creating data by grouping the classifier, the learning image DB, the evaluation image, and the calculation results, the performance of a classifier created using data from other facilities (such as hospitals) can be determined by comparing the calculation results.

[0165] Furthermore, by changing the evaluation image to image data of an arbitrary facility, it is possible to determine which facility identifier can accurately recognize an object within the image of the facility.

[0166] The machine learning device 1 may hold a plurality of learning image DBs (determined) 100 and execute the above-described process between each of the plurality of learning image DBs (determined) 100 and the learning image DB (before determination) 201. This can obtain a more appropriate learning image DB and classifier.

[0167] (3) Third embodiment

[0168] Figure 14 14 is a functional block diagram showing the configuration of a remote diagnosis support system 1400 according to the third embodiment. The remote diagnosis support system 1400 includes a server (computer) 1403 and an image acquisition device 1405 .

[0169] The image acquisition device 1405 is a device such as a personal computer equipped with a virtual slide device and a camera, and includes an imaging unit 1401 for capturing a new image and a display unit 1404 for displaying the determination results transmitted from the server 1403. Although not shown, the image acquisition device 1405 also includes a communication device for transmitting image data to the server 1403 and receiving data transmitted from the server 1403.

[0170] Server 1403 includes an image diagnosis support device 5 that performs image processing on image data transmitted from image acquisition device 1405 using the machine learning device 1 of the first or second embodiment, and a storage unit 1402 that stores recognition results output from image diagnosis support device 5. Although not shown, server 1403 also includes a communication device that receives image data transmitted from image acquisition device 1405 and transmits determination result data to image acquisition device 1405.

[0171] The image diagnosis support device 5 uses the identifier (current identifier) ​​obtained by the machine learning device 1 to identify the presence of objects to be detected (e.g., abnormal tissue or abnormal cells (e.g., cancer)) within the image generated by the imaging unit 1401 (e.g., tissue or cells). The display unit 1404 displays the identification results transmitted from the server 1403 on the display screen of the image acquisition device 1405.

[0172] As the image acquisition device 1405 , a regenerative medicine device including an imaging unit, an iPS cell culture device, or an MRI or ultrasonic imaging device may be used.

[0173] As described above, the third embodiment provides a remote diagnosis support system. Specifically, the remote diagnosis support system uses the identifier parameters calculated by the machine learning device 1 to accurately classify objects (e.g., tissue, cells, etc.) within images transmitted from a different facility or the like as objects to be detected (abnormal tissue, abnormal cells, etc.). The classification results are then transmitted to the different facility or the like, and displayed on a display unit of an imaging device located at that facility or the like.

[0174] (4) Fourth embodiment

[0175] Figure 15 This is a functional block diagram showing the configuration of a network managed service providing system 1500 according to the fourth embodiment. The network managed service providing system 1500 includes a server (computer) 1503 and an image acquisition device 1505 .

[0176] The image acquisition device 1505 is a device such as a personal computer equipped with a virtual slide device and a camera. The image acquisition device 1505 includes an imaging unit 1501 for capturing new images, a storage unit 1504 for storing the identifier (current identifier) ​​transmitted from the server 1503, and the image diagnosis support device 5.

[0177] The image diagnosis auxiliary device 5 reads the identifier transmitted from the server 1503, and uses the identifier obtained by the machine learning device 1 of the first embodiment or the second embodiment to determine whether the object (for example, tissue, cell, etc.) in the image newly captured by the imaging unit 1501 is an object to be detected (for example, abnormal tissue, abnormal cell, etc.).

[0178] Although not shown in the figure, the image acquisition device 1505 includes a communication device that transmits image data to the server 1503 and receives data transmitted from the server 1503 .

[0179] The server 1503 includes the image diagnosis support device 5 and a storage unit 1502 for storing the identifier output from the machine learning device 1 of the image diagnosis support device 5. The image diagnosis support device 5 generates an identifier using the machine learning device 1 of the first embodiment or the second embodiment for the image data transmitted from the image acquisition device 1505, and performs recognition processing using the generated identifier.

[0180] Although not shown, the server 1503 includes a communication device that receives image data transmitted from the image acquisition device 1505 and transmits a recognizer to the image acquisition device 1505 .

[0181] Furthermore, the machine learning device 1 within the image diagnosis support device 5 performs machine learning on objects (e.g., tissues, cells, etc.) within images captured by the imaging unit 1501 to determine which objects should be detected (e.g., normal tissues, cells are normal tissues, cells, abnormal tissues, cells are abnormal tissues, cells, etc.), thereby creating a discriminator. The discriminator calculates the feature values ​​of objects (e.g., tissues, cells, etc.) within images of various facilities, etc. The storage unit 1504 stores the discriminator transmitted from the server 1503.

[0182] The image diagnosis assisting device 5 within the image acquisition device 1505 reads the identifier from the storage unit 1504, and uses the identifier to classify whether the object (for example, tissue, cell, etc.) in the image newly captured by the imaging unit 1501 of the image acquisition device 1505 is an object to be detected (for example, abnormal tissue, abnormal cell, etc.), and displays the classification result on the display screen of the output device (display device) 204 of the image diagnosis assisting device 5.

[0183] As the image acquisition device 1505 , a regenerative medicine device including an imaging unit, an iPS cell culture device, or an MRI or ultrasonic imaging device may be used.

[0184] As described above, according to the fourth embodiment, a network entrusted service providing system can be provided. Specifically, the network entrusted service providing system performs machine learning on objects (e.g., tissues, cells, etc.) in images transmitted from facilities at different locations, etc., to classify them as objects that should be detected (e.g., normal tissues, cells are normal tissues, cells, abnormal tissues, cells are abnormal tissues, cells, etc.), and creates an identifier. The network entrusted service providing system transmits the identifier to the facilities at different locations, etc., and reads the identifier through an image acquisition device at the facility, etc. The identifier in the image acquisition device classifies the objects (e.g., tissues, cells, etc.) in the new image as objects that should be detected (e.g., abnormal tissues, abnormal cells, etc.).

[0185] The above-described embodiments can be modified as follows. While the learning unit 11 uses filters to determine feature quantities through machine learning, other feature quantities such as HOG can also be used. The learning unit 11 can also use a loss function such as quadratic error or Hinge loss instead of negative log likelihood. The learning unit 11 can generate a classifier using any method different from the methods described in the above embodiments.

[0186] The above embodiment describes the updating or generation of the recognizer and the learning image database, but by changing the dimension of the input data of Formulas 1 to 3 from 2 dimensions to other dimensions, the updating or generation of the recognizer and the learning database in the above embodiment can also be applied to data samples different from images, such as sound data samples, sensor data samples, and text data samples.

[0187] The present invention can also be implemented by the program code of the software that realizes the functions of the embodiment. In this case, a storage medium recording the program code is provided to a system or device, and the computer (or CPU, MPU) of the system or device reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described, and the program code itself and the storage medium storing the program code constitute the present invention. As a storage medium for supplying such a program code, for example, a floppy disk, CD-ROM, DVD-ROM, hard disk, optical disk, optical magneto-magnetic disk, CD-R, magnetic tape, non-volatile memory card, ROM, etc. are used.

[0188] Alternatively, an OS (operating system) or the like running on a computer may perform some or all of the actual processing according to the instructions of the program code, thereby realizing the functions of the embodiments described above through such processing. Furthermore, after the program code is read from a storage medium and written to a memory on a computer, the computer's CPU or the like may perform some or all of the actual processing according to the instructions of the program code, thereby realizing the functions of the embodiments described above through such processing.

[0189] Furthermore, the program code of the software that implements the functions of the implementation method can also be distributed via a network, whereby the program code is stored in a storage unit such as a hard disk or memory of the system or device or a storage medium such as a CD-RW or CD-R. When in use, the program code stored in the storage unit or the storage medium is read out and executed by a computer (or CPU, MPU) of the system or device.

[0190] Finally, the processes and techniques described herein are not inherently tied to any specific apparatus and can be implemented using any suitable combination of components. Furthermore, various types of general-purpose equipment can be used in accordance with the methods described herein. It is sometimes beneficial to construct specialized apparatuses to perform the steps of the methods described herein. Furthermore, various inventions can be formed by appropriately combining the various structural elements disclosed in the embodiments.

[0191] For example, several structural elements may be deleted from all the structural elements shown in the embodiments. Furthermore, structural elements in different embodiments may be appropriately combined. The present invention has been described in conjunction with specific examples, but these examples are not intended to be limiting in all respects but rather to be illustrative. Those skilled in the art will appreciate that there are multiple combinations of hardware, software, and firmware suitable for implementing the present invention. For example, the described software can be installed using a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, Java, etc.

[0192] Furthermore, in the above-described embodiment, the control lines and information lines are shown as those necessary for explanation, and not all control lines and information lines on the product are necessarily shown. All structures may be connected to each other.

[0193] Furthermore, other embodiments of the present invention will become apparent to those having ordinary knowledge in the art from a review of the description and embodiments of the present invention disclosed herein. The various modes and / or components of the described embodiments may be used alone or in any combination.

Claims

1. A machine learning device, characterized in that Include: a processor that processes the data samples; and a storage device for storing the results of said processing, The processor creates a plurality of identifiers based on a plurality of learning databases, wherein the plurality of learning databases respectively store a plurality of learning data samples. The processor generates an evaluation result of the recognition performance of each of the plurality of identifiers using the evaluation samples. The processor determines one of the plurality of learning databases and a classifier generated based on the one learning database as the learning database and classifier to be used based on the evaluation result. The multiple learning databases include a first learning database and a second learning database, The second learning database stores the images stored in the first learning database and the new input image. The processor determines whether the first learning database can be updated through the second learning database and whether the identifier generated according to the second learning database can be used based on the comparison results of the first identifier generated according to the first learning database and the second identifier generated according to the second learning database.

2. The machine learning device according to claim 1, wherein The data sample is an image, The plurality of learning databases are respectively learning image databases.

3. The machine learning device according to claim 1, wherein The processor determines whether to replace the order of the new input image based on a comparison result of the recognition result of the first recognizer and the recognition result of the second recognizer, When the processor determines that the order is to be replaced, the processor creates a new second identifier based on the second learning database after the order of the new input images is changed. The processor determines whether the first learning database can be updated by the second learning database and whether the identifier generated based on the second learning database can be used based on a comparison result of the recognition results of the first identifier and the new second identifier.

4. The machine learning device according to claim 1, wherein In the first learning database and the second learning database, the balance of the number of images of the recognition type is adjusted.

5. An image diagnosis assisting device, characterized in that: Include: a processor that processes images; and a storage device for storing the results of said processing, The processor makes a plurality of recognizers based on a plurality of learning image databases, The processor generates an evaluation result of the recognition performance of each of the plurality of recognizers using the evaluation image. The processor determines one of the plurality of learning image databases and a classifier generated based on the one learning image database as the learning image database and classifier to be used based on the evaluation result. The plurality of learning image databases include a first learning image database and a second learning image database, The second learning image database stores images stored in the first learning image database and new input images. The processor determines whether the first learning image database can be updated using the second learning image database and whether the identifier generated based on the second learning image database can be used based on a comparison result of recognition results of a first identifier generated based on the first learning image database and a second identifier generated based on the second learning image database, The processor displays a recognition result of a new input image generated by the recognizer based on the one learning image database.

6. A machine learning method for a machine learning device to produce an identifier, characterized in that: The machine learning device comprises: a processor that processes the data samples; and a storage device for storing the results of said processing, In the machine learning method, The processor creates a plurality of identifiers based on a plurality of learning databases, wherein the plurality of learning databases respectively store a plurality of learning data samples. The processor generates an evaluation result of the recognition performance of each of the plurality of identifiers using the evaluation samples. The processor determines one of the plurality of learning databases and a classifier generated based on the one learning database as the learning database and classifier to be used based on the evaluation result. The multiple learning databases include a first learning database and a second learning database, The second learning database stores the images stored in the first learning database and the new input image. In the machine learning method, the processor determines whether the first learning database can be updated through the second learning database and whether the identifier generated according to the second learning database can be used based on the comparison results of the first identifier generated according to the first learning database and the second identifier generated according to the second learning database.

7. The machine learning method according to claim 6, wherein: The data sample is an image, The plurality of learning databases are respectively learning image databases.

8. An image diagnosis assistance method based on an image diagnosis assistance device, characterized in that: The image diagnosis auxiliary device comprises: a processor that processes images; and a storage device for storing the results of said processing, In the image diagnosis assisting method, The processor makes a plurality of recognizers based on a plurality of learning image databases, The processor generates an evaluation result of the recognition performance of each of the plurality of recognizers using the evaluation image. The processor determines one of the plurality of learning image databases and a classifier generated based on the one learning image database as the learning image database and the classifier to be used based on the evaluation result. The plurality of learning image databases include a first learning image database and a second learning image database, The second learning image database stores images stored in the first learning image database and new input images. In the image diagnosis assisting method, The processor determines whether the first learning image database can be updated using the second learning image database and whether the identifier generated based on the second learning image database can be used based on a comparison result of recognition results of a first identifier generated based on the first learning image database and a second identifier generated based on the second learning image database, The processor displays a recognition result of a new input image generated by the recognizer based on the one learning image database.

Images