Computer, learning method of recognizer, and analysis system

Abstract

The computer of the present application includes a storage unit that stores learning data, a learning unit that performs learning processing for generating an identifier using the learning data, and a generation unit that generates the learning data, the generation unit calculating a feature amount vector of the identifier processing using the learning data stored in the storage unit, and analyzing a distribution of the learning data in a feature amount space based on the feature amount vector, thereby determining a boundary of a change in an identification result of the identifier in the feature amount space, and generating new learning data using the learning data existing in the vicinity of the boundary.

Description

[0001] This application claims priority to Japanese Patent Application No. 2020-18575, filed on February 6, 2020, the contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to a technique for generating a recognizer that can identify arbitrary events. Background Technology

[0003] Automated analytical systems equipped with immunoassay devices are used to analyze the components of samples such as blood and urine, and to measure the color and luminescence states of the reaction solution produced by reacting the sample with reagents. The samples, reagents, and reaction solutions used for analysis are stored in containers such as test tubes, and samples are collected from the containers using a dispensing probe or other collection unit.

[0004] When air bubbles are present on the liquid surface during sample or reagent collection, there is a problem that the dispensing probe may misdetect the air bubbles as liquid, thus preventing the collection of the required amount of sample or reagent for analysis. To solve this problem, the technology described in Patent Document 1 is known.

[0005] Patent document 1 describes "an apparatus for determining the state of a sample containing an analytical object stored in a container, the apparatus acquiring an image of the sample, using the image of the sample to analyze the position and size of the object to be analyzed relative to a detection range set in the image, and determining the state of the sample based on the analysis result".

[0006] In recent years, products equipped with recognizers generated by performing machine learning have begun to appear on the market. The recognizer is generated by using machine learning, which uses a set of learning data (learning dataset) consisting of input data (input signal) and training data (training signal) fed to the recognizer.

[0007] Recognizers (models) such as neural networks have complex structures, making their behavior difficult for humans to understand. Therefore, to address misidentification in real-world scenarios, new containers and prototypes, or the customization and tuning of each facility, a common approach is to construct a new learning dataset and re-perform machine learning to regenerate the recognizer. In this specification, this is referred to as relearning.

[0008] Relearning can predict an improvement in recognition accuracy for input data that is currently unmanageable. However, since relearning alters the characteristics of the recognizer, it may produce incorrect recognition results for input data that was correctly identified before relearning. In particular, when evaluating the recognizer using evaluation data—consisting of input data and training data, and rigorously determined for product implementation—deterioration in recognition accuracy for that evaluation data is undesirable. Furthermore, the training data constituting the evaluation data is also known as the correct recognition data.

[0009] The technology described in Patent Documents 2 and 3 is known in this regard.

[0010] Patent document 2 describes "an information processing apparatus 10 for inference using parameters, comprising: a data acquisition unit 31 for acquiring input data; a basic parameter storage unit 41 for storing parameters before additional learning; a difference parameter storage unit 40 for storing the difference between the parameters used for inference and the basic parameters, i.e., a first difference parameter; an additional learning unit 42 for calculating the difference between the parameters after additional learning for the basic parameters and the basic parameters as a second difference parameter; an update unit 43 for updating the first difference parameter stored in the difference parameter storage unit 40 based on the first difference parameter and the second difference parameter; and an inference unit 34 for performing inference on the input data using model parameters generated based on the basic parameters and the difference parameters updated by the update unit."

[0011] Furthermore, Patent Document 3 describes a method for learning to modify the combination of a neural network to modify the parameter values ​​of a presumed object by means of a neural network. The method is to modify the combination of the neural network in a way that is predetermined according to each region represented by a portion of a plurality of input parameters. The method is to determine which region the neural network belongs to based on the portion of the parameter values ​​received by the region determination unit, in the same way as when the path modification unit learns the combination of the neural network based on the region determined by the region determination unit.

[0012] Existing technical documents

[0013] Patent documents

[0014] Patent Document 1: Japanese Patent Application Publication No. 2019-027927

[0015] Patent Document 2: Japanese Patent Application Publication No. 2017-138808

[0016] Patent Document 3: Japanese Patent Application Publication No. 11-85719 Summary of the Invention

[0017] The problem that the invention aims to solve

[0018] However, while the technology described in Patent Document 2 can reproduce the characteristics of the identifier at the time of manufacture, the identifier uses parameters obtained by adding the basic parameters and the differential parameters when it is actually performing identification, so it is impossible to avoid changes in the identification results.

[0019] Furthermore, while the technology described in Patent Document 3 is applicable when the input signal has a low dimensionality, it is impractical to create a massive number of recognizers for multi-dimensional signals such as images. Additionally, in machine learning, obtaining general feature sets and recognition performance requires more training data, making segmenting the training data by each region less preferable.

[0020] In this invention, a method for generating a recognizer that efficiently and effectively improves the recognition accuracy of the recognizer is proposed.

[0021] Methods for solving problems

[0022] The following is a representative example of the invention disclosed in this application. Specifically, a computer has a computing device, a storage device connected to the computing device, and an interface connected to the computing device. The computer generates a recognizer that identifies arbitrary events. The computer includes: a storage unit that stores learning data consisting of first input data and first training data; a learning unit that uses the learning data stored in the storage unit to perform learning processing for generating the recognizer; and a generation unit that generates the learning data. The generation unit uses the first input data from the learning data stored in the storage unit to calculate a feature vector processed by the recognizer, and based on the feature vector of the learning data, analyzes the distribution of the learning data in a feature space formed by the feature vector, thereby determining the boundary of the recognition result variation of the recognizer in the feature space. Using the learning data existing near the boundary, i.e., the feature vector representing the learning data, it generates first pseudo-input data, generates new learning data consisting of the first pseudo-input data and the first training data representing the learning data, and stores it in the storage unit.

[0023] Invention Effects

[0024] According to the present invention, it is possible to generate a recognizer that is highly efficient and effectively improves recognition accuracy. Other issues, structures, and effects beyond those described above will become clear through the following description of embodiments. Attached Figure Description

[0025] Figure 1 This is a diagram illustrating an example of the structure of the automatic analysis system of Embodiment 1.

[0026] Figure 2 This is a diagram illustrating an example of the functional blocks of the machine learning device of Embodiment 1.

[0027] Figure 3 This is a diagram illustrating an example of the structure of the recognizer generated by the machine learning device of Embodiment 1.

[0028] Figure 4A This is a diagram illustrating the change in the recognition result of the recognition unit before and after relearning in Example 1.

[0029] Figure 4B This is a diagram illustrating the change in the recognition result of the recognition unit before and after relearning in Example 1.

[0030] Figure 5 This is a flowchart illustrating an example of the processing performed by the pseudo-sample generation unit in Example 1.

[0031] Figure 6A This is a diagram illustrating an example of the distribution and distribution density of the learning data in the feature space calculated by the pseudo-sample generation unit in Example 1.

[0032] Figure 6B This is a diagram illustrating an example of the distribution and distribution density of the learning data in the feature space calculated by the pseudo-sample generation unit in Example 1.

[0033] Figure 7 This is a diagram illustrating an example of the calculation method for the generated object vector of the pseudo-sample generation unit in Embodiment 1.

[0034] Figure 8 This is a flowchart illustrating an example of the processing performed by the pseudo-sample generation unit in Example 2.

[0035] Figure 9 This is a diagram showing an example of the generated object vector calculated by the pseudo-sample generation unit in Example 2.

[0036] Figure 10 This is a flowchart illustrating an example of the processing performed by the pseudo-sample generation unit in Example 3. Detailed Implementation

[0037] Hereinafter, embodiments of the present invention will be described using the accompanying drawings. Furthermore, in the following description and drawings, constituent elements having the same function are labeled with the same reference numerals, and repeated descriptions are omitted. Additionally, the terms "first," "second," "third," etc., used in this specification are added for the purpose of identifying constituent elements and do not necessarily limit the number or order. Furthermore, for ease of understanding of the invention, the positions, sizes, shapes, and extents of the structures shown in the drawings sometimes do not represent actual positions, sizes, shapes, and extents. Therefore, the present invention is not limited to the positions, sizes, shapes, and extents disclosed in the drawings.

[0038] Example 1

[0039] Figure 1 This is a diagram illustrating an example of the structure of the automatic analysis system 100 of Embodiment 1. The automatic analysis system 100 includes a machine learning device 101 and an automatic analysis device 102. The automatic analysis system 100 is connected to a user interface 131 operated by a user.

[0040] User interface 131 consists of input devices such as a mouse and keyboard, and output devices such as a monitor and printer. The input devices and output devices can also be different devices.

[0041] First, the structure of the automatic analysis device 102 will be explained.

[0042] The automatic analysis device 102 includes an imaging unit 124, an identification unit 125, a control unit 121, an acquisition unit 122, and an analysis unit 123. The hardware components are interconnected via a bus (not shown).

[0043] The imaging unit 124 is a device for photographing the container 132 and the sample 133 contained in the container 132, such as a camera. The container 132 is a test tube, etc., and the sample 133 is a sample such as blood or urine, a reagent that reacts with the sample, or a reaction solution obtained by reacting the sample and the reagent. The imaging unit 124 is located on the opening side of the container 132, that is, above the container 132, and photographs the container 132 and the sample 133 from above. The image photographed by the imaging unit 124 from above the container 132 is called the top image.

[0044] The image above can be a static image such as BMP, PNG, or JPEG, or a frame image extracted from a dynamic image such as MPEG or H.264 at fixed intervals.

[0045] The recognition unit 125 identifies the state of the sample and reagents by inputting the image above as input data into the recognizer. For example, the recognition unit 125 identifies whether there are bubbles and foreign objects on the sample surface that may hinder sample collection. The recognizer is generated using machine learning methods such as Neural Network and SVM (Support Vector Machine).

[0046] The control unit 121 is a device that controls the operation of the acquisition unit 122 based on the recognition result output from the recognition unit 125, such as a CPU (Central Processing Unit).

[0047] The acquisition unit 122 is a device that acquires data from the container 132 to the acquisition object 133, such as a dispensing probe.

[0048] The analysis unit 123 is a device for performing analysis on the sample 133, such as an immunoassay analyzer. The analysis results of the analysis unit 123 are displayed on a monitor (not shown).

[0049] Next, the structure of the machine learning device 101 will be described.

[0050] The machine learning device 101 includes an interface unit 111, a processing unit 112, a memory 113, and a bus 114. The interface unit 111, the processing unit 112, and the memory 113 exchange information with each other via the bus 114.

[0051] The interface unit 111 is a device for connecting to external devices. The machine learning device 101 is connected to the automatic analysis device 102 and the user interface 131 via the interface unit 111. The interface unit 111 receives images from the imaging unit 124 above, and also receives signals input by the operator through the user interface 131.

[0052] The computing unit 112 is a device that performs various processes of the machine learning device 101, such as a CPU and an FPGA (Field-Programmable Gate Array). Figure 2 The functions performed by the arithmetic unit 112 will be explained.

[0053] The memory 113 is a device that stores the program executed by the arithmetic unit 112, various information used by the program (parameters and coefficients), and processing results. It can be an HDD (Hard Disk Drive), RAM (Random Access Memory), ROM (Read Only Memory), or flash memory, etc. In addition, the memory 113 includes the working area used by the program.

[0054] Furthermore, the machine learning device 101 can also be installed in a different system than the automatic analysis system 100. In this case, the machine learning device 101 sends and receives information such as images and calculation results with the automatic analysis device 102 via a communication unit such as a network.

[0055] In addition, Figure 1 In this configuration, the interface unit 111 is directly connected to the imaging unit 124 and the recognition unit 125 of the automatic analysis device 102. However, it may also be a configuration in which the automatic analysis device 102 has a memory and transmits and receives images and processing results from the calculation unit 112 via the memory.

[0056] Furthermore, in this specification, as an example of the method of utilizing the machine learning device 101, an automatic analysis system 100 including an automatic analysis device 102 will be described. However, as long as the system performs image recognition, image detection, and segmentation using machine learning, it is not limited to the automatic analysis device 102 and can utilize the machine learning device 101.

[0057] Figure 2 This is a diagram illustrating an example of the functional blocks of the machine learning device 101 in Embodiment 1. Figure 3 This is a diagram illustrating an example of the structure of the recognizer generated by the machine learning device 101 of Embodiment 1.

[0058] The machine learning device 101 includes an input unit 201, a storage unit 202, a learning unit 203, a pseudo-sample generation unit 204, a recognizer evaluation unit 205, and an output unit 206. In Embodiment 1, the input unit 201, storage unit 202, learning unit 203, pseudo-sample generation unit 204, recognizer evaluation unit 205, and output unit 206 are implemented by a computation unit 112 that executes a program. Furthermore, each of the above-mentioned functional units can also be implemented using dedicated hardware.

[0059] The input unit 201 receives the overhead image from the automatic analysis device 102 and stores it in the storage unit 202. Additionally, the input unit 201 outputs the received overhead image to the user interface 131 via the output unit 206, and receives training signals for the overhead image from the user. The data associated with the training signals for the overhead image is saved as learning data.

[0060] The acquisition and processing of training signals can be performed when the image above is received, or multiple information images can be processed together after the image above is saved in the storage unit 202 and a fixed time has elapsed, or it can be performed when a user request is received.

[0061] The output unit 206 outputs the image above and the learning results, etc.

[0062] Storage unit 202 stores various types of information. Specifically, storage unit 202 stores information (parameters and coefficient sets) of the recognizer used by the recognizer unit 125 of the automatic analysis device 102, the learning dataset used when the recognizer is generated through machine learning, the evaluation dataset used to evaluate the recognizer, the evaluation results of the recognizer using the evaluation data, and pseudo samples generated by the pseudo sample generation unit 204. Furthermore, sometimes a portion of the learning data is used as evaluation data. Additionally, storage unit 202 stores the upper image received by the input unit 201, and stores the training signal input to the upper image in association with that upper image. Furthermore, the evaluation results of the recognizer are stored for each generation of the learning process.

[0063] Here, the evaluation result of the recognizer refers to the recognition result such as the presence or probability of bubbles in the evaluation data, or the recognition accuracy of the evaluation data.

[0064] Explanation of Study Department 203.

[0065] The learning unit 203 generates information, such as a set of coefficients of a neural network, for the recognizer used by the recognizer unit 125 of the automatic analysis device 102 by performing machine learning. In this embodiment, machine learning that generates a neural network consisting of three fully combined layers as the recognizer will be described as an example.

[0066] Figure 3 The network 300 shown represents a neural network consisting of three fully combined layers: an input layer 301, a hidden layer 302, and an output layer 303. Each layer contains more than one unit.

[0067] The input layer 301 is a layer that receives input signals, such as the brightness of each pixel of the image above, which is input to each unit.

[0068] Hidden layer 302 is the layer that obtains features from the input signal. For example, the units of hidden layer 302 substitute the input signal Xi, weight Wji, and bias bj received from each unit of input layer 301 into equation (1) to calculate the feature Yj.

[0069] [Formula 1]

[0070]

[0071] Here, N represents the number of pixels in the image above. The function f is the activation function, such as the Sigmoid function, tanh function, and ReLU function.

[0072] Output layer 303 is the layer that obtains the final output. For example, by substituting the feature quantity Yj, weight Wkj and bias bk of hidden layer 302 into equation (2), the final output Zk is calculated.

[0073] [Formula 2]

[0074]

[0075] Here, H represents the number of units in hidden layer 302. Additionally, softmax is a function for calculating probabilities, defined as in equation (3).

[0076] [Formula 3]

[0077]

[0078] Here, M represents the number of classes in the recognition result.

[0079] In neural networks, a loss function is typically calculated to represent the difference between the final output Zk and the training signal T. The loss function is then used to find the minimum weights Wji and Wkj based on gradient descent. Negative log-likelihood and hinge loss, among other loss functions, are commonly used.

[0080] In this specification, the process of updating the coefficient sets of weights Wji and Wkj based on gradient descent is referred to as learning. Furthermore, the vector composed of the feature quantities Yj output from each unit of hidden layer 302 is called the feature vector.

[0081] The storage unit 202 stores information about the coefficient set calculated through the learning described above, which serves as the recognizer.

[0082] Furthermore, as an example, Network 300 uses a neural network consisting of three fully concatenated layers, but the number of hidden layers can be increased, or a convolutional neural network can be used instead of fully concatenated layers. In this case, the feature quantity refers to the output signal of the hidden layer before the output layer (the final hidden layer).

[0083] Furthermore, the learning unit 203 in Embodiment 1 performs relearning, which uses a learning dataset pre-stored in the storage unit 202, a newly input upper image from the input unit 201, a training signal associated with the upper image, and learning data generated by the pseudo-sample generation unit 204 (described later). As a result, a coefficient set with higher recognition accuracy can be obtained compared to the case where a coefficient set (recognizer) stored in the storage unit 202 before performing relearning is used.

[0084] In machine learning, as a method to improve the robustness of a recognizer, the number of patterns in the learning data is increased, which involves appending the learning data. Appending the learning data refers to generating learning data based on perturbations such as position, rotation angle, and brightness, as well as generating learning data using generators.

[0085] Adding training data can improve the robustness of the recognizer. However, the number of patterns of combinations of perturbations such as position, rotation angle, and brightness is enormous, making it impractical to use all combinations of training data. Furthermore, it is difficult to determine which training data is effective in improving the recognizer's accuracy. Additionally, relearning using the added training data presents the following challenges. Figure 4A as well as Figure 4B This is a diagram illustrating the change in the recognition result of the recognition unit 125 before and after relearning in Embodiment 1.

[0086] Here, we will use a bubble detector as an example. This detector outputs the probabilities of "bubbles present" and "bubbles absent".

[0087] Figure 4A as well as Figure 4B This represents a graph depicting the learning and evaluation data in the feature space 400. In the feature space 400, the learning and evaluation data are represented as feature vectors. In the following description, input data, without distinguishing between learning and evaluation data, is referred to as a sample. Figure 4A as well as Figure 4B In this model, the training signals for the recognizer are represented by the shapes of the samples (circles and squares). Furthermore, learning data is represented in white, and evaluation data in black.

[0088] The recognition surface 401 represents the boundary of the recognition results of each class in the feature space 400. Samples located on the upper side of the recognition surface 401 are judged as "having bubbles", and samples located on the lower side of the recognition surface 401 are judged as "not having bubbles".

[0089] From the perspective of the probabilities of "with bubbles" and "without bubbles" output by the recognizer, the probabilities of both sides of the sample located on the recognition surface 401 are the same. The sample located on the upper side of the recognition surface 401 has a higher probability of "with bubbles", while the sample located on the lower side of the recognition surface 401 has a higher probability of "without bubbles".

[0090] Figure 4A This is an example of the feature space 400 and the recognition surface 401 before relearning.

[0091] Learning data 402 and learning data 403 are the data used for training the recognizer. Learning data 402 represents the training data when the training signal is "with bubbles", and learning data 403 represents the training data when the training signal is "without bubbles".

[0092] The learning unit 203 updates the recognizer by outputting the correct recognition result for the learning data 402 and 403. As a result of learning using the learning data 402 and 403, the recognition surface 401 is formed.

[0093] Evaluation data 404 and 405 are used to evaluate the recognizer and are not referenced during learning. The recognizer's accuracy relative to the evaluation data and the recognition results are considered as the recognizer's performance. Evaluation data 404 represents evaluation data where the training signal is "with bubbles," and evaluation data 405 represents evaluation data where the training signal is "without bubbles."

[0094] exist Figure 4AIn the above, evaluation data 404 is located on the upper side of recognition surface 401, and evaluation data 405 is located on the lower side of recognition surface 401, indicating that the recognizer correctly outputs the recognition result.

[0095] Figure 4B This is an example of the relearned feature space 400 and recognition surface 401.

[0096] Through relearning, the recognition of face 401 is improved. Figure 4A The recognition surface 401 changes. The recognizer is able to correctly recognize the learning data 402 and 403 as before relearning. However, the evaluation data 404, represented by circles, is located below the recognition surface 401, and the recognition result is "no bubbles". Thus, because the recognition surface 401 changes through relearning, misidentifications that did not occur before relearning may sometimes happen.

[0097] The reasons for the changes in the recognition surface 401 as described above include various factors such as the structure of the learning data, the learning order, the structure of the recognizer, the initial value of the recognizer's weights, the learning convergence method, and the number of learning iterations.

[0098] One method to maintain a similar recognition surface 401 before and after relearning is transfer learning. In conventional learning, the initial values ​​for the weights are random numbers, but transfer learning uses the coefficient set of the recognizer before relearning as the initial values. Thus, by using the recognition surface 401 before relearning as the starting point, it is easier to maintain the recognition surface 401. However, although the main purpose of relearning is to correct the recognition surface relative to sample groups that cannot be correctly identified by the recognizer before relearning, in the case of transfer learning, learning starts from a state where incorrect identification is not possible. Therefore, compared to learning from random numbers, it is sometimes difficult to improve the recognition accuracy against misidentifications.

[0099] To address the aforementioned issues, the pseudo-sample generation unit 204, described later, generates learning data for relearning based on the distribution of learning data in the feature space 400.

[0100] In Example 1, pseudo-samples (input data) are generated for regions with high distribution density of learning data in the feature space 400 to reproduce the recognition surface 401 before relearning. This suppresses changes in the recognition surface 401 in regions with dense learning data, and improves recognition accuracy through relearning for samples that the recognizer could not correctly identify before relearning (i.e., regions with insufficient learning).

[0101] As for the timing of relearning execution, it is possible to consider cases where a fixed number of new top images and training data sets are stored in the storage unit 202, or cases where a fixed number of misidentified top images are stored. Alternatively, relearning can also be performed when a relearning start instruction is received from a user or system administrator via the input unit 201 using the user interface 131 or a communication device.

[0102] Next, the pseudo-sample generation unit 204 will be explained.

[0103] Figure 5 This is a flowchart illustrating an example of the processing performed by the pseudo-sample generation unit 204 in Embodiment 1. Figure 6A and Figure 6B This is a diagram illustrating an example of the distribution and distribution density of learning data in the feature space calculated by the pseudo-sample generation unit 204 of Embodiment 1. Figure 7 This is a diagram illustrating an example of the calculation method for the generated object vector of the pseudo-sample generation unit 204 in Embodiment 1.

[0104] The pseudo-sample generation unit 204 calculates the distribution density of the learning data in the feature space (step S501). Here, feature quantity refers to... Figure 3 The output of hidden layer 302 has a feature space of H dimensions with H features as axes. Specifically, the following processing is performed.

[0105] The pseudo-sample generation unit 204 obtains the feature vector of the learning data by inputting the learning data (input data) into the network 300 and records it in the working area.

[0106] The pseudo-sample generation unit 204 calculates the distribution density of the learning data using a set of feature vectors. One example of a method for calculating the distribution density is to use an H-dimensional kernel that extracts a portion of the feature space, while simultaneously scanning the feature space and counting the number of data points contained within the kernel.

[0107] Furthermore, the pseudo-sample generation unit 204 determines the boundary surface based on the feature vector set and the training data of the learning data. Methods for determining the boundary surface include, for example, using the least squares method.

[0108] Furthermore, when the dimension of the feature vector is large, the computation time for determining the boundary surface increases, and the required computing resources also increase. Therefore, the dimension of the feature vector can be reduced based on computation time and computing resource requirements. For example, there are methods to reduce the dimension of the feature vector by performing principal component analysis.

[0109] exist Figure 6A and Figure 6BFor the sake of simplicity, we assume H = 2. Figure 6A The learning data distribution 601 shown represents the distribution of learning data in the feature space with feature quantities Y0 and Y1 as axes. For learning data distribution 601, with a kernel size of 11×11 and a kernel sliding width of 5×5, the result of calculating the distribution density of the learning data is... Figure 6B The distribution density shown is 602. The lower the density within the kernel, the whiter it appears; the higher the density, the blacker it appears.

[0110] The above is an explanation of the processing in step S501.

[0111] Next, the pseudo-sample generation unit 204 extracts representative learning data located near the recognition surface based on the distribution density of the learning data (step S502). Specifically, the following processing is performed.

[0112] The pseudo-sample generation unit 204 identifies regions with high distribution density in the learning data. One method for this identification is to pre-set a threshold and select regions with a distribution density above that threshold.

[0113] The pseudo-sample generation unit 204 determines representative points for the determined regions. Representative points may be, for example, the maxima or centroids of the distribution density within each determined region.

[0114] The pseudo-sample generation unit 204 extracts the learning data closest to the location of the representative point in the feature space. In the following description, the input data of the extracted learning data is referred to as the neighborhood sample.

[0115] The nearby sample group is the sample group that has a significant impact on the formation position of the recognition surface. In this embodiment, pseudo-samples are generated in a way that maintains the positional relationship between nearby samples, so that the recognition surface after relearning can easily reproduce the recognition surface before relearning.

[0116] The above is an explanation of the processing in step S502.

[0117] Next, the pseudo-sample generation unit 204 calculates a feature vector for generating pseudo-samples (step S503). In the following description, this feature vector will be referred to as the generation object vector. Specifically, the following processing is performed.

[0118] The pseudo-sample generation unit 204 determines, for each neighboring sample associated with the "no-bubble" training data, the neighboring sample with the shortest distance to each neighboring sample associated with the "no-bubble" training data and associated with the "bubble" training data. In the following description, the neighboring sample associated with the "no-bubble" training data is referred to as the first neighboring sample, and the neighboring sample associated with the "bubble" training data is referred to as the second neighboring sample.

[0119] The pseudo-sample generation unit 204 calculates the generated object vector FA' and the generated object vector FB' by substituting the feature vector FA of the first nearby sample and the feature vector FB of the determined second nearby sample into equations (4) and (5).

[0120] [Formula 4]

[0121] FA′=FA+a(FB-FA)…(4)

[0122] [Formula 5]

[0123] FB′=FB-b(FB-FA)…(5)

[0124] Here, (FB-FA) represents the position of the feature vector FB in the feature space with the feature vector FA as the origin. Coefficients a and b determine the magnitude of the perturbation to the generated object vectors FA' and FB', and are set according to the distance from the recognition surface. Specific setting examples will be described later.

[0125] use Figure 7 The relationship between nearby samples and generated object vectors is explained in detail.

[0126] Nearby sample 701 represents the first nearby sample, and nearby sample 702 represents the second nearby sample. The feature vector of nearby sample 701 is FA, and the feature vector of nearby sample 702 is FB. Nearby sample 701 is located at coordinates that are far from the recognition surface 703 at a distance DA, and nearby sample 702 is located at coordinates that are far from the recognition surface 703 at a distance DB. Generated object vectors FA'704 and FA”706 are generated object vectors based on the first nearby sample 701, and generated object vectors FB'705 and FB”707 are generated object vectors based on the second nearby sample 702.

[0127] The generated object vector FA'704 is a vector calculated based on equation (4), and the generated object vector FB'705 is a vector calculated based on equation (5). Here, an example of the calculation of the coefficient a is shown in equation (6).

[0128] [Formula 6]

[0129] a=DA*r…(6)

[0130] According to equation (6), the coefficient 'a' is a value proportional to the distance DA. 'r' is a proportionality constant related to both the distance DA and 'a', taking a real value between 0.0 and 1.0. Figure 7In the equation (4), r = 0.3. Therefore, the generated object vector FA'704 calculated by equation (4) is located at the coordinates after the feature vector FA has been moved by DA*0.3 from the first nearby sample 701 toward the second nearby sample 702.

[0131] Equation (6) describes the coefficient a, but by replacing the distance DA with the distance DB and the coefficient a with the coefficient b, it can also be applied to the coefficient b. Therefore, the generated object vector FB'705 calculated by equation (5) is located at the coordinates after the feature vector FB has been moved by DB*0.3 from the second nearby sample 702 toward the first nearby sample 701.

[0132] By configuring the generated object vectors over a wide range in proportion to the distance from the recognition surface 703, it is possible to suppress the relearned recognition surface from getting too close to nearby samples, thus enabling the reproduction of the recognition surface that maintains its positional relationship with nearby samples.

[0133] In addition, to maintain the central position of the distribution, the generating object vectors FA'706 and FB'707 can be generated as shown in equations (7) and (8). It should be noted that the signs of the coefficients are reversed compared to the calculation formulas (4) and (5) for generating object vectors FA'704 and FB'705.

[0134] [Formula 7]

[0135] FA″=FA-a(FB-FA)…(7)

[0136] [Formula 8]

[0137] FB″=FB+b(FB-FA)…(8)

[0138] By adding learning data to both sides of the first nearby sample 701 and the second nearby sample 702, which serve as the distribution center, it is possible to suppress the changes in the position of the nearby sample 701 and the nearby sample 702 in the feature space.

[0139] Furthermore, the above example illustrates the generation of a pair of generated object vectors, but multiple generated object vectors can also be generated. Equation (9) represents an example of the calculation formula for the coefficient 'a' in the case of generating multiple generated object vectors.

[0140] [Formula 9]

[0141] a∈[0,a′]…(9)

[0142] a' represents the maximum value of the coefficient a. For example, by setting the value calculated by equation (6) as a', multiple generated object vectors can be generated while the coefficient a is randomly varied between 0 and a'. In addition, the example of changing the position of the generated object vector according to the distance was given above, but the number of generated object vectors can also be changed according to the distance.

[0143] The above is an explanation of the processing in step S503.

[0144] Next, the pseudo-sample generation unit 204 generates an upper image (pseudo-sample) as input data based on the generated object vector, and also generates learning data including the upper image (step S504). The generated learning data is stored in the storage unit 202. Specifically, the following processing is performed.

[0145] The pseudo-sample generation unit 204 copies nearby samples that serve as the computational source for generating object vectors and uses them as appended samples I.

[0146] The pseudo-sample generation unit 204 updates the additional sample I based on equation (10).

[0147] [Formula 10]

[0148]

[0149] Here, I t I represents time t. t+1 Let I represent the additional sample at time t+1. Z represents the generated object vector calculated in step S503, and F... t This indicates that additional sample I will be added. t The feature values ​​obtained by inputting into network 300. I is updated by repeatedly performing the operation of equation (10). t It can generate images that output object vectors.

[0150] The pseudo-sample generation unit 204 generates learning data by associating the generated image (pseudo-sample) with the training signal. Furthermore, the training signal associated with the generated image uses the training signal of a nearby sample that serves as the copy source for the appended sample I.

[0151] Furthermore, the above is one example of a generation method; pseudo-samples can also be generated using generators such as GAN (Generative Adversarial Networks) and VAE (Variational Autoencoder). With the generator set to g, the image (input data) can be generated in the same way as described above using the update formula shown in Equation (11).

[0152] [Formula 11]

[0153]

[0154] Here, R t It is the input vector of the generator at time t, and the generator g will use R t As input, generate generated data g(R) with the same dimension as the appended sample I. t Additionally, F(g(R) t )) indicates that data g(R) will be generated. t The feature vector obtained by inputting it into the recognizer.

[0155] The learning data group generated through the above process is added to the learning dataset, thereby maintaining the positional relationship between nearby samples and reproducing the same recognition surface as before relearning.

[0156] Furthermore, the pseudo-sample generation unit 204 generates learning data only for regions with dense learning data. Therefore, it can reproduce the recognition surface only for regions with dense learning data, i.e., regions with high reliability of the recognition result. On the other hand, for regions with sparse learning data, recognition accuracy is improved through relearning.

[0157] Next, the recognition evaluation unit 205 will be explained.

[0158] The recognizer evaluation unit 205 compares the recognition results of the recognizer before relearning and the recognizer after relearning, and determines the recognizer to be applied to the recognizer unit 125 of the automatic analysis device 102 based on the comparison results.

[0159] Here, an example of the evaluation method is described. The recognizer evaluation unit 205 pre-stores evaluation results for the evaluation data of the recognizer before relearning in the storage unit 202. The recognizer evaluation unit 205 uses the newly generated recognizer from the learning unit 203 to obtain the evaluation results for the evaluation data stored in the storage unit 202. The recognizer evaluation unit 205 verifies the difference in recognition accuracy or misidentified samples between the recognizer before and after relearning, and determines the recognizer to be applied to the recognition unit 125.

[0160] For example, the recognizer evaluation unit 205 has the following method: in order to verify that no new misidentification has occurred through relearning, it confirms that the set of misidentified samples in the relearned recognizer is included in the set of misidentified samples in the recognizer before relearning, and adopts the relearned recognizer if no new misidentification has occurred. An example of the evaluation formula is shown in Equation (12).

[0161] [Formula 12]

[0162] |M β -M α |≤ThM …(12)

[0163] M β M represents the set of misidentifications by the relearned recognizer. α Let || represent the set of misidentifications by the recognizer before relearning, || represent the number of elements in the set, and negative indicates the operation of finding the difference set. M This represents the permissible number relative to the number of misidentifications.

[0164] For example, if equation (12) is satisfied, the recognizer evaluation unit 205 uses the relearned recognizer; if equation (12) is not satisfied, the recognizer evaluation unit 205 uses the unlearned recognizer. In the case of Th... M Setting it to 0 ensures that no new misidentifications occur, thus preventing the recognition accuracy from decreasing due to relearning or recognition updates.

[0165] Alternatively, a method can be used to confirm whether the recognition accuracy of the relearned recognizer is higher than that of the recognizer before relearning. The evaluation formula is shown in Equation (13).

[0166] [Formula 13]

[0167] A β -A α ≥Th A …(13)

[0168] A α And A β Th represents the recognition accuracy relative to the evaluation data before and after relearning. A This represents the threshold relative to the recognition accuracy.

[0169] If equation (13) is satisfied, the recognizer evaluation unit 205 uses the relearned recognizer; if equation (13) is not satisfied, the recognizer evaluation unit 205 uses the unlearned recognizer.

[0170] Alternatively, the evaluation data can be divided into multiple groups, with the evaluation method switched for each group. For example, the following method could be used: easily identifiable evaluation data or data with a significant impact on the analysis can be divided into a first group, while difficult-to-identify evaluation data or data whose impact is minimal even if identification fails can be divided into a second group. The first group is then used to verify that no new misidentifications occurred, and the second group is used to verify that the identification accuracy has not decreased, thus verifying whether both conditions are met. Another method could be to pre-set weights for identification accuracy and the number of misidentifications for each group, and then use the sum of the values ​​obtained by multiplying the identification accuracy and the number of misidentifications by their respective weights for a comprehensive evaluation.

[0171] The recognizer evaluation unit 205 outputs the evaluation result via the output unit 206. For example, the recognizer evaluation unit 205 outputs information indicating the change in misidentification of the recognizer before and after relearning as the evaluation result. In addition, the recognizer evaluation unit 205 outputs the information of the adopted recognizer as the evaluation result.

[0172] According to Embodiment 1, the machine learning device 101 generates new learning data for reproducing recognition surfaces in regions with high distribution density of learning data in the feature space. Furthermore, by performing relearning using a learning dataset supplemented with new learning data, the machine learning device 101 can suppress variations in recognition results for regions with dense learning data and efficiently improve recognition accuracy for regions with sparse learning data. In other words, it can efficiently and effectively improve the recognition accuracy of the recognizer.

[0173] Example 2

[0174] In Embodiment 2, in addition to the learning data generated based on the boundary surface, the machine learning device 101 also adds learning data to regions with low distribution density of learning data in the feature space.

[0175] Since the distribution of learning data in the feature space directly affects the generation of the recognition surface, the machine learning device 101 in Embodiment 2 generates learning data in regions with low distribution density of learning data in the feature space. The distribution density of learning data in the input space formed by the learning data (input data) itself tends to be significantly different from the distribution density of learning data in the feature space formed by the features processed by the recognizer. Therefore, even if additional learning data is added based on the distribution density of learning data in the input space, the recognition accuracy of the recognizer may not necessarily improve.

[0176] Hereinafter, Example 2 will be described with a focus on the differences from Example 1.

[0177] The structure of the automatic analysis system 100 in Embodiment 2 is the same as that in Embodiment 1. The hardware structure and functional block structure of the machine learning device 101 in Embodiment 2 are the same as those in Embodiment 1. Furthermore, the hardware structure and functional block structure of the automatic analysis device 102 in Embodiment 2 are the same as those in Embodiment 1.

[0178] In Embodiment 2, the processing performed by the pseudo-sample generation unit 204 in the functional blocks of the machine learning device 101 is somewhat different. The other functional blocks are the same as in Embodiment 1.

[0179] Figure 8 This is a flowchart illustrating an example of the processing performed by the pseudo-sample generation unit 204 in Embodiment 2. Figure 9 This is a diagram showing an example of the generated object vector calculated by the pseudo-sample generation unit 204 in Embodiment 2.

[0180] The processing of steps S501 to S504 is the same as in Example 1, so the description is omitted.

[0181] After the processing in step S504 is completed, the pseudo-sample generation unit 204 calculates the generated object vectors for regions with low distribution density of the learning data (step S801). Specifically, the following processing is performed.

[0182] The pseudo-sample generation unit 204 identifies regions with low distribution density in the learning data. One method for this identification is to pre-set a threshold and select regions with distribution density less than the threshold.

[0183] The pseudo-sample generation unit 204 randomly selects multiple points from the determined region and calculates the selected points as the generated object vector. One method for determining the number of points to select is to divide the determined region into multiple grids and select points in such a way that each grid contains an equal amount of learning data.

[0184] exist Figure 9 The method for calculating the generated object vector is explained in the text. Figure 9 In the diagram, based on the learning data distribution 601, a region 901 is shown where Y0 is greater than or equal to 25 and less than 40, and Y1 is greater than or equal to 25 and less than 50. This region 901 represents a low distribution density of the learning data, with a distribution density of 0 to 2. Square markers represent the feature vectors of the learning data, and cross markers represent the generated object vectors. Figure 9 In this method, the generated object vector is calculated by including three training data points (the feature vector of the training data and the generated object vector) in a grid that divides the feature space with a fixed width. That is, the lower the density of the region, the more generated object vectors are calculated.

[0185] The above is an explanation of the processing in step S801.

[0186] Next, the pseudo-sample generation unit 204 generates an image (pseudo-sample) based on the generated object vector, and generates learning data containing that image (step S802). The generated learning data is stored in the storage unit 202. The processing of step S802 is the same as that of step S504, so the description is omitted. At this time, the training signal associated with the generated image uses the training signal containing the learning data in the determined region. In addition, the user can also input the training signal by referring to the generated image.

[0187] Through the processing in steps S801 and S802, learning data can be added in a focused manner to regions with low distribution density of learning data in the feature space.

[0188] According to Embodiment 2, the machine learning device 101 adds learning data, such as maintaining boundary surfaces, to regions with high distribution density of learning data in the feature space, and adds learning data to regions with low distribution density of learning data in the feature space. Furthermore, by performing relearning using the newly added learning dataset, the machine learning device 101 can efficiently improve the recognition accuracy of the recognizer.

[0189] Example 3

[0190] Example 3 adds evaluation data to regions with low distribution density of evaluation data in the feature quantity space.

[0191] In evaluating a recognizer generated through machine learning, the comprehensiveness of the evaluation data becomes important. However, as described in Example 2, even if the evaluation data is distributed comprehensively in the input space, the distribution of the evaluation data in the feature space is different. Therefore, in the feature space, there may be regions with sparse or dense distribution of evaluation data, or regions where no evaluation data exists. In this case, sufficient robustness cannot be guaranteed. Therefore, there is a possibility that the product (automatic analysis device 102) equipped with the recognizer may be misidentified after leaving the factory.

[0192] Therefore, in Example 3, the method described in Example 2 is applied to generate evaluation data for improving the evaluation quality of the recognizer.

[0193] Hereinafter, Example 3 will be described with a focus on the differences from Example 1 and Example 2.

[0194] The structure of the automatic analysis system 100 in Embodiment 3 is the same as that in Embodiment 1. The hardware structure and functional block structure of the machine learning device 101 in Embodiment 3 are the same as those in Embodiment 1. Furthermore, the hardware structure and functional block structure of the automatic analysis device 102 in Embodiment 3 are the same as those in Embodiment 1.

[0195] In Embodiment 3, the processing performed by the pseudo-sample generation unit 204 and the recognizer evaluation unit 205 in the functional blocks of the machine learning device 101 is partially different. The other functional blocks are the same as in Embodiment 1.

[0196] First, the pseudo-sample generation unit 204 will be explained. Figure 10 This is a flowchart illustrating an example of the processing performed by the pseudo-sample generation unit 204 in Embodiment 3.

[0197] The pseudo-sample generation unit 204 calculates the distribution density of the evaluation data in the feature space (step S1001). The processing in step S1001 is to replace the processing object of step S501 with the processing of evaluation data.

[0198] Next, the pseudo-sample generation unit 204 calculates the generation object vector for regions with low distribution density of evaluation data (step S1002). The processing in step S1002 is to replace the object processed in step S801 with the evaluation data.

[0199] Next, the pseudo-sample generation unit 204 generates an image (pseudo-sample) based on the generated object vector, and generates evaluation data containing the image (step S1003). The generated evaluation data is stored in the storage unit 202. The processing in step S1003 is the process of replacing the object processed in step S802 with the evaluation data.

[0200] By execution Figure 10 The processing described herein can focus on adding evaluation data to areas with low distribution density of evaluation data in the feature space.

[0201] Next, the recognizer evaluation unit 205 will be described. The recognizer evaluation unit 205 evaluates the recognizer using evaluation data pre-stored in the storage unit 202 and evaluation data generated by the pseudo-sample generation unit 204. The recognizer evaluation method can be a method that evaluates the overall recognition accuracy and false recognition of the evaluation data, or a method that evaluates the evaluation data as different evaluation datasets. The verification method for recognition accuracy and false recognition, and the evaluation method using multiple evaluation datasets, are the same as the methods described in Example 1.

[0202] According to Embodiment 3, the machine learning device 101 adds evaluation data to regions with low distribution density of evaluation data in the feature space. By using both existing and added evaluation data to evaluate the recognizer, the robustness of the recognizer can be evaluated more accurately.

[0203] Furthermore, the present invention is not limited to the embodiments described above, and includes various modifications. Additionally, for example, the embodiments described above are examples illustrating the structure in detail for ease of understanding of the present invention, and are not limited to having all the described structures. Furthermore, regarding a portion of the structure of each embodiment, other structures can be added to, deleted from, or replaced.

[0204] Furthermore, some or all of the aforementioned structures, functions, processing units, and processing modules can be implemented in hardware, for example, through integrated circuit design. Additionally, the present invention can also be implemented by software program code that implements the functions of the embodiments. In this case, a storage medium storing program code is provided to a computer, and the processor of the computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself implements the functions of the aforementioned embodiments, and the program code itself and the storage medium storing the program code constitute the present invention. Examples of storage media used for supplying such program code include floppy disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, optical discs, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0205] Furthermore, the program code that implements the functions described in this embodiment can be implemented using a wide range of program or scripting languages, such as assembler, C / C++, perl, shell, PHP, Python, and Java.

[0206] Furthermore, the program code of the software that implements the functions of the embodiments can also be distributed via a network, thereby storing the program code in a storage unit such as a computer hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer reads and executes the program code stored in the storage unit or the storage medium.

[0207] In the above embodiments, the control lines and information lines shown are those deemed necessary for the description, and not necessarily all control lines and information lines on the product are shown. All structures can also be interconnected.

Claims

1. A computer comprising a computing unit, a storage device connected to the computing unit, and an interface connected to the computing unit, wherein the computer generates a recognizer that recognizes arbitrary events, characterized in that, The computer has: The storage unit stores learning data consisting of the first input data and the first training data; The learning unit uses the learning data stored in the storage unit to perform learning processing for generating the recognizer; as well as The generation unit generates the learning data. The generation unit performs the following processing: Using the first input data of the learning data stored in the storage unit, the feature vector processed by the recognizer is calculated; Based on the feature vector of the learning data, the distribution of the learning data in the feature space formed by the feature vector is analyzed, thereby determining the boundary of the recognition result change of the recognizer in the feature space; Based on the distribution density of the learning data, the learning data existing near the boundary is extracted as representative learning data. For the region where the learning data is dense in the feature space, the first pseudo-input data is generated according to the feature vector located near the representative learning data in the feature space. as well as New learning data is generated, consisting of the first pseudo-input data and the first training data representing the learning data, and stored in the storage unit. The generation unit also performs the following processing: Based on the analysis results of the distribution of the learning data in the feature space, the sparse regions of the learning data are determined; Multiple points are randomly selected from the determined region, and the selected points are calculated as the generated object vector to generate the second pseudo-input data. The determined region is divided into multiple grids, and the number of multiple points is selected in such a way that the number of learning data contained in each grid is equal. as well as New learning data is generated, consisting of the second pseudo-input data and the first training data containing the learning data in the determined region, and stored in the storage unit.

2. The computer according to claim 1, characterized in that, The storage unit stores evaluation data consisting of the second input data and the second training data. The computer includes an evaluation unit that uses the evaluation data stored in the storage unit to evaluate the recognizer. The generation unit performs the following processing: The feature vector is calculated using the second input data of the evaluation data; Based on the feature vector of the evaluation data, the distribution of the evaluation data in the feature space is analyzed to determine the sparse region of the evaluation data. The second pseudo-input data is generated using the feature vector contained in the determined region; as well as New evaluation data is generated by the second pseudo-input data and the second training data containing the evaluation data in the determined region, and stored in the storage unit.

3. The computer according to claim 1, characterized in that, The storage unit stores evaluation data consisting of the second input data and the second training data. The computer includes an evaluation unit that evaluates the recognizer using the evaluation data stored in the storage unit. After generating the first recognizer, the learning unit generates a second recognizer by re-executing the learning process, provided that the learning data generated by the generation unit is stored in the storage unit. The evaluation department performs the following processing: By comparing the output values ​​obtained by inputting the second input data of the evaluation data into the first recognizer and the second recognizer with the second training data of the evaluation data, the recognition results of the first recognizer and the second recognizer relative to the evaluation data are obtained and stored in the storage unit. Based on the recognition results of the first and second recognizers relative to the evaluation data, analyze the changes in the recognition results relative to the evaluation data; as well as Based on the analysis of the changes in the recognition results relative to the evaluation data, it is determined whether to use the first recognizer or the second recognizer.

4. The computer according to claim 3, characterized in that, The computer includes an output unit that generates and outputs display information, the display information being used to indicate the analysis results of changes in the recognition results relative to the evaluation data.

5. A learning method for a computer-executed recognizer that identifies arbitrary events, characterized in that, The computer has a computing device, a storage device connected to the computing device, and an interface connected to the computing device. The storage device stores learning data consisting of the first input data and the first training data. The learning method of the recognizer includes: In the first step, the computing device performs learning processing for generating the recognizer using the learning data stored in the storage device; and In the second step, the computing device uses the learning data stored in the storage device to generate new learning data, and stores the new learning data in the storage device. The second step includes the following steps: The computing device uses the first input data of the learning data stored in the storage device to calculate the feature vector processed by the recognizer; The computing device analyzes the distribution of the learning data in the feature space formed by the feature vectors of the learning data, thereby determining the boundary of the recognition result variation of the recognizer in the feature space; based on the distribution density of the learning data, the computing device extracts the learning data existing near the boundary as representative learning data, and generates first pseudo-input data for the dense region of learning data in the feature space according to the feature vectors located near the representative learning data in the feature space; and The computing device generates new learning data consisting of the first pseudo-input data and the first training data representing the learning data. The second step also includes the following steps: The computing device determines the sparse regions of the learning data based on the analysis results of the distribution of the learning data in the feature space; The computing device randomly selects multiple points from the determined region, calculates the selected points as a generated object vector, and generates second pseudo-input data. The determined region is divided into multiple grids, and the number of points is selected such that each grid contains an equal amount of learning data. The computing device generates new learning data consisting of the second pseudo-input data and the first training data containing the learning data in the determined region.

6. The learning method for the recognizer according to claim 5, characterized in that, The storage device stores evaluation data consisting of the second input data and the second training data. The learning method of the recognizer includes the following steps: The computing device uses the second input data of the evaluation data to calculate the feature vector; The computing device analyzes the distribution density of the evaluation data in the feature space based on the feature vector of the evaluation data, thereby determining the sparse region of the evaluation data. The computing device uses the feature vector contained in the determined region to generate second pseudo-input data; The computing device generates new evaluation data consisting of the second pseudo-input data and the second training data containing the evaluation data in the determined region, and stores it in the storage device; as well as The computing device uses the evaluation data stored in the storage device to evaluate the recognizer.

7. The learning method for the recognizer according to claim 5, characterized in that, The storage device stores evaluation data consisting of the second input data and the second training data. The learning method of the recognizer includes the following steps: After generating the first recognizer, the computing device generates a second recognizer by re-executing the learning process, provided that the new learning data is stored in the storage device. The computing device obtains the recognition results of the first and second recognizers relative to the evaluation data by comparing the output value obtained by inputting the second input data of the evaluation data into the first and second recognizers with the second training data of the evaluation data. The computing device analyzes the changes in the recognition results relative to the evaluation data based on the recognition results of the first and second recognizers relative to the evaluation data. as well as The computing device determines which of the first and second recognizers to use based on the analysis of the changes in the recognition results relative to the evaluation data.

8. The learning method for the recognizer according to claim 7, characterized in that, The learning method includes the following steps: The computing device generates and outputs display information, which is used to indicate the analysis results of the changes in the recognition results relative to the evaluation data.

9. An analysis system, characterized in that, The analysis system has the following features: The computer according to any one of claims 1 to 4; and The analysis device performs analysis using a recognition device generated by the computer.

Images