Data adjustment system and data adjustment method

Abstract

A data adjustment system according to this disclosure includes an information processing device and a terminal device. The information processing device comprises: a measurement unit that measures the impact that learning data used in neural network learning had on the learning; and an adjustment unit that adjusts the learning set by eliminating data for which the measured impact was low, by acquiring from the terminal device or a database new data that corresponds to data for which the measured impact was high, and by adding the acquired new data.

Description

Technical Field

[0001] The present disclosure relates to a data adjustment system and Data adjustment method by the law and is related thereto.

Background Art

[0002] In various technical fields, information processing using machine learning (also simply referred to as "learning") has been utilized, and techniques for learning models such as neural networks have been provided. In such learning, since the performance of models such as neural networks to be learned is affected by the data used for learning, the data used for learning is important, and techniques related to the data used for learning have been provided (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the prior art, learning is performed using data in which missing values are supplemented from candidate values.

[0005] However, the prior art does not always enable learning using appropriate data. For example, in the prior art, when data that has no missing values but is not suitable for learning is used, that data is used as it is, and thus it may not be possible to learn a model such as a neural network having desired performance. As described above, the prior art considers whether there are missing values in the data used for learning, but does not consider whether the data itself used for learning is suitable for learning. Therefore, it is desired to make the data used for learning adjustable.

[0006] Therefore, in the present disclosure, a data adjustment system that can make the data used for learning adjustable is and proposed. the law

Means for Solving the Problems

[0007] In order to solve the above problems, a data adjustment system according to an aspect of the present disclosure includes an information processing device and a terminal device. The information processing device includes a measurement unit that measures the degree of influence that the learning data used for neural network learning has on the learning, an adjustment unit that excludes the data measured to have a low degree of influence and adjusts the learning data by acquiring new data that is new data corresponding to the data measured to have a high degree of influence from the terminal device or a database and adding the acquired new data.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the data adjustment system, data adjustment device, data adjustment method, terminal device, and information processing device according to the present application are not limited by this embodiment. Also, in each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] The present disclosure will be described according to the item order shown below. 1. Embodiment 1-1. Outline of data adjustment processing according to an embodiment of the present disclosure 1-1-1. Background and effects 1-1-2. Concept of data adjustment system 1-1-3. Influence function 1-1-4. Bayesian Deep Learning 1-1-5. Others (GAN, Grad-CAM, LIME, etc.) 1-2. Configuration of data adjustment system according to an embodiment 1-3. Configuration of data adjustment device according to an embodiment 1-3-1. Example of model (network) 1-4. Configuration of terminal device according to an embodiment 1-5. Information processing procedure according to an embodiment 1-5-1. Processing procedure related to data adjustment device 1-5-2. Procedure of Processing Related to Data Adjustment System 1-6. Example of Data Adjustment Based on Influence Degree 1-6-1. Specific Example of Adjustment 2. Other Embodiments 2-1. Other Configuration Examples 2-2. Others 3. Effects According to the Present Disclosure 4. Hardware Configuration

[0011] [1. Embodiment] [1-1. Outline of Data Adjustment Processing According to Embodiment of the Present Disclosure] FIG. 1 is a diagram showing an example of data adjustment processing according to an embodiment of the present disclosure. The data adjustment processing according to the embodiment of the present disclosure is realized by a data adjustment system 1 including a data adjustment device 100 and a terminal device 10. In FIG. 1, an outline of the data adjustment processing realized by the data adjustment system 1 is explained. FIG. 1 is a diagram showing an example of data adjustment processing according to an embodiment of the present disclosure.

[0012] The data adjustment device 100 is an information processing device that adjusts learning data by excluding predetermined data from the learning data used for learning a model by machine learning or by adding new data to the learning data. In FIG. 1, the case where the data adjustment device 100 executes a process of adjusting data of a data set used for learning a deep neural network (DNN) is shown as an example. The data adjustment device 100 executes a learning process of learning an identification model (hereinafter, also simply referred to as a “model”), which is a DNN for performing image recognition, using the data set. Hereinafter, the deep neural network (DNN) may be simply described as a neural network (NN). In FIG. 1, the case where the data adjustment device 100 learns a model used for smile detection is described as an example. Note that the use of the model learned by the data adjustment device 100 is not limited to smile detection, and the data adjustment device 100 learns models used for various uses such as object recognition and emotion detection according to the purpose and use of the model to be learned.

[0013] Also, in FIG. 1, as an example of the terminal device 10 that provides data to the data adjustment device 100 in response to the request of the data adjustment device 100, a data server having data corresponding to the request of the data adjustment device 100 and a camera that captures data (image) corresponding to the request of the data adjustment device 100 are shown. Note that the terminal device 10 is not limited to a data server or a camera, and may be various devices as long as it can provide the data requested by the data adjustment device 100 to the data adjustment device 100. For example, the terminal device 10 may be a moving body such as a UAV (Unmanned Aerial Vehicle) like a drone or a vehicle such as an automobile, or an image sensor (imager). Details regarding this point will be described later.

[0014] Hereinafter, the outline of the process shown in FIG. 1 will be described. First, in the example of FIG. 1, the data adjustment device 100 learns a model M1, which is a neural network used for smile detection, using a data set DS1 (step S1). For example, the data adjustment device 100 learns the model M1 using the data set DS1 stored in the data information storage unit 121 (see FIG. 5). In FIG. 1, each cell in the data set DS1 indicates data (image), showing that the data set DS1 contains a large number of data (images).

[0015] In the example of FIG. 1, the data adjustment device 100 designs the structure of a network (such as a neural network) corresponding to the model M1 stored in the model information storage unit 122 (see FIG. 6). The data adjustment device 100 designs the network structure (network structure) of the model M1 used for smile detection. For example, the data adjustment device 100 may generate the network structure of the model M1 used for smile detection based on information regarding the network structure corresponding to each use case previously stored in the storage unit 120 (see FIG. 4). For example, the data adjustment device 100 may acquire the network structure information of the model M1 used for smile detection from an external device.

[0016] For example, the data adjustment device 100 learns the model M1 using a dataset DS1 in which a correct label indicating the presence or absence of a smiling face is associated with each data (image). The data adjustment device 100 performs a learning process to minimize the set loss function (loss function) using the dataset DS1 and learns the model M1. If the influence degree of each data can be measured in the measurement process described later, the data adjustment device 100 may use various functions as the loss function. Note that the loss function in the Influence function will be described later.

[0017] For example, the data adjustment device 100 learns the model M1 by updating parameters such as weights and biases so that the output layer becomes the correct value for the input of the data. For example, in the error backpropagation method, for a neural network, using a loss function indicating how far the value of the output layer is from the correct state (correct label), and using the steepest descent method or the like, the weights and biases are updated so that the loss function is minimized. For example, the data adjustment device 100 gives an input value (data) to a neural network (model M1), the neural network (model M1) calculates a predicted value based on the input value, and compares the predicted value with the teacher data (correct label) to evaluate the error. Then, the data adjustment device 100 executes the learning and construction of the model M1 by sequentially correcting the values of the connection weights (synaptic coefficients) in the neural network (model M1) based on the obtained error. Note that the above is an example, and the data adjustment device 100 may perform the learning process of the model M1 by various methods.

[0018] Then, the data adjustment device 100 measures the degree of influence that each piece of data in the data set DS1 has on the learning of the model M1. The data adjustment device 100 measures the degree of influence that each piece of data in the data set DS1 has on the learning of the model M1 by using a method for measuring the degree of influence (measurement method MM1). Here, the degree of influence means that the larger the value, the higher the degree of contribution (degree of contribution) of that data to the learning of the model M1. The larger the value of the degree of influence, that is, the higher the degree of influence, indicates that it contributes to the improvement of the discrimination accuracy of the model M1. Thus, the higher the degree of influence, the more necessary that data is for the learning of the model M1. For example, the higher the degree of influence, the more beneficial that data is for the learning of the model M1.

[0019] Also, the degree of influence means that the smaller the value, the lower the degree of contribution (degree of contribution) of that data to the learning of the model M1. The smaller the value of the degree of influence, that is, the lower the degree of influence, indicates that it does not contribute to the improvement of the discrimination accuracy of the model M1. Thus, the lower the degree of influence, the more unnecessary that data is for the learning of the model M1. For example, the lower the degree of influence, the more harmful that data is for the learning of the model M1.

[0020] In FIG. 1, as an example of the measurement method MM1, a case where Influence function (Influence functions) is used is shown. The Influence function will be described later. Note that the measurement method MM1 used by the data adjustment device 100 for measuring the influence degree may be any method as long as a value indicating the influence degree of each data can be obtained, not limited to the Influence function. For example, when there is sufficient time and processing resources, the data adjustment device 100 may measure the influence degree of each data by removing the data one by one and performing re-learning processing. In this case, the data adjustment device 100 may measure the influence degree of data X by removing one data (data X) from the data set DS1 and performing re-learning processing. For example, the data adjustment device 100 may measure the influence degree of data X by taking the difference between the loss in learning when using the entire data set DS1 and the loss in learning when data X is excluded from the data set DS1. Note that the above is an example, and the data adjustment device 100 may measure the influence degree of each data by using other methods other than the Influence function and the above method.

[0021] In FIG. 1, the data adjustment device 100 measures the influence degree that the data DT14 in the data set DS1 has on the learning of the model M1 (step S2). As shown in the measurement result RS1, the data adjustment device 100 measures the influence degree that the data DT14 has on the learning of the model M1 as the influence degree IV14. Note that the influence degree IV14 is assumed to be a specific value (for example, 0.2, etc.).

[0022] Then, the data adjustment device 100 adjusts the data set DS1 based on the influence degree IV14 of the data DT14 (step S3). First, the data adjustment device 100 determines whether the data DT14 is necessary for the learning of the model M1 based on the influence degree IV14 of the data DT14. For example, the data adjustment device 100 uses the threshold value stored in the threshold value information storage unit 123 (see FIG. 7) to determine whether the data DT14 is necessary for the learning of the model M1.

[0023] For example, the data adjustment device 100 uses a threshold value (first threshold value TH1) that is used to determine data with a low impact degree, that is, a low contribution degree (also referred to as "first data") to determine whether the data DT14 is necessary for the learning of the model M1. The data adjustment device 100 compares the impact degree IV14 of the data DT14 with the first threshold value TH1, and if the impact degree IV14 is lower than the first threshold value TH1, it determines that the data DT14 is unnecessary for the learning of the model M1.

[0024] In FIG. 1, since the impact degree IV14 of the data DT14 is lower than the first threshold value TH1, the data adjustment device 100 determines that the data DT14 is unnecessary for the learning of the model M1. Therefore, as shown in the determination result DR1, the data adjustment device 100 determines that the contribution degree of the data DT14 to the learning of the model M1 is low, and decides to exclude the data DT14 from the data set DS1. Then, the data adjustment device 100 adjusts the data set DS1 by excluding the data DT14 from the data set DS1. Thereby, the data adjustment device 100 updates the data set DS1.

[0025] Also, in FIG. 1, the data adjustment device 100 measures the impact degree that the data DT33 in the data set DS1 has on the learning of the model M1 (step S4). The data adjustment device 100 measures the impact degree that the data DT33 has on the learning of the model M1 as the impact degree IV33 as shown in the measurement result RS2. Note that the impact degree IV33 is assumed to be a specific value (for example, 0.7, etc.).

[0026] Then, the data adjustment device 100 adjusts the data set DS1 based on the influence degree IV33 of the data DT33 (step S5). First, the data adjustment device 100 determines whether the data DT33 is necessary for the learning of the model M1 based on the influence degree IV33 of the data DT33. For example, the data adjustment device 100 uses the threshold value stored in the threshold value information storage unit 123 to determine whether the data DT33 is necessary for the learning of the model M1. In FIG. 1, since the influence degree IV33 of the data DT33 is equal to or greater than the first threshold value TH1, the data adjustment device 100 determines that the data DT33 is not unnecessary for the learning of the model M1.

[0027] Then, the data adjustment device 100 uses a threshold value (second threshold value TH2) used for determining data with a high influence degree, that is, a high contribution degree (also referred to as "second data") to determine whether the data DT33 is necessary for the learning of the model M1. Note that the second threshold value TH2 is a value larger than the first threshold value TH1. The data adjustment device 100 compares the influence degree IV33 of the data DT33 with the second threshold value TH2, and when the influence degree IV33 is higher than the second threshold value TH2, determines that the data DT33 is necessary for the learning of the model M1.

[0028] In FIG. 1, since the influence degree IV33 of the data DT33 is higher than the second threshold value TH2, the data adjustment device 100 determines that the data DT33 is necessary for the learning of the model M1. Therefore, as shown in the determination result DR2, the data adjustment device 100 determines that the contribution degree of the data DT33 to the learning of the model M1 is high, and decides to add the data corresponding to the data DT33 to the data set DS1. Then, the data adjustment device 100 adjusts the data set DS1 by adding the data corresponding to the data DT33 to the data set DS1. Thereby, the data adjustment device 100 updates the data set DS1.

[0029] In FIG. 1, the data adjustment device 100 requests the terminal device 10 for data corresponding to the data DT33 (step S6). The data adjustment device 100 transmits request information for requesting data corresponding to the data DT33 (also referred to as "new data") to the terminal device 10. The data adjustment device 100 requests the terminal device 10 for new data similar to the data DT33. The data adjustment device 100 requests the terminal device 10 for data similar to the data DT33 by transmitting information indicating the data DT33 to the terminal device 10. For example, the data adjustment device 100 requests the terminal device 10 for data similar to the data DT33 by transmitting the data DT33 to the terminal device 10.

[0030] The terminal device 10 that has received the request from the data adjustment device 100 collects data corresponding to the request information (step S7). The terminal device 10 collects data similar to the data DT33 as data to be provided to the data adjustment device 100 (also referred to as "data for provision").

[0031] For example, when the terminal device 10 is a data server, the terminal device 10 collects data corresponding to the request information by extracting data corresponding to the request information from the held data group. For example, the terminal device 10 collects data corresponding to the request information by extracting data similar to the data DT33 from the held database. For example, the terminal device 10 compares the data DT33 with each piece of data in the database and extracts, as data for provision, data whose similarity to the data DT33 is within a predetermined threshold. For example, the terminal device 10 may calculate the similarity between the data DT33 and each piece of data in the database using a model that outputs the similarity of images, and extract, as data for provision, data whose similarity to the data DT33 is within a predetermined threshold.

[0032] Also, for example, when the terminal device 10 is a camera, the terminal device 10 collects data corresponding to the request information by extracting the data corresponding to the request information from the captured data. For example, the terminal device 10 collects data corresponding to the request information by extracting data similar to the data DT33 from a plurality of captured images (data). For example, the terminal device 10 compares the data DT33 with each captured image and extracts, as the data for provision, the data whose similarity to the data DT33 is within a predetermined threshold value. Note that the terminal device 10 may be controlled to capture an image similar to the data DT33. In this case, the terminal device 10 captures an image similar to the data DT33 and collects the image as the data for provision.

[0033] Then, the terminal device 10 provides the data for provision to the data adjustment device 100 (step S8). The terminal device 10 transmits, as the data for provision, the data similar to the collected data DT33 to the data adjustment device 100.

[0034] The data adjustment device 100 that has acquired the data for provision from the terminal device 10 adds the acquired data for provision to the data set DS1 (step S9). Thereby, the data adjustment device 100 adds data similar to the data DT33 with a high contribution degree to the data set DS1.

[0035] In FIG. 1, the case where the data adjustment device 100 acquires new data added from the terminal device 10 and adds it to the data set DS1 is shown. However, the data adjustment device 100 may add the new data acquired by any means to the data set DS1.

[0036] For example, the data adjustment device 100 may acquire data (new data) corresponding to the data DT33 from the storage unit 120 and add the acquired data to the acquired data set DS1. In this case, the data adjustment device 100 acquires (extracts) data similar to the data DT33 from among the data not included in the data set DS1 from the storage unit 120, and adds the acquired (extracted) data to the data set DS1. In this way, the data adjustment device 100 may acquire data similar to the data with a high contribution degree (second data) in the data set DS1 from the storage unit 120 and add the data to the data set DS1.

[0037] Also, for example, the data adjustment device 100 may generate data corresponding to the data DT33 and add the generated data (new data) to the data set DS1. In this case, the data adjustment device 100 may generate data similar to the data DT33 and add the generated data to the data set DS1. For example, the data adjustment device 100 generates data similar to the data DT33 by appropriately using various techniques such as data expansion, and adds the generated data to the data set DS1. In this way, the data adjustment device 100 may generate data similar to the data with a high contribution degree (second data) in the data set DS1 and add the generated data to the data set DS1.

[0038] Note that in FIG. 1, only the processing for the two data DT14 and DT33 is shown for the sake of explanation, but the data adjustment device 100 executes the same processing for all the data in the data set DS1. For example, the data adjustment device 100 measures the influence degree for all the data in the data set DS1. Then, the data adjustment device 100 excludes data with a low contribution degree from the data set DS1. Also, the data adjustment device 100 adds data similar to the data with a high contribution degree to the data set DS1. Thereby, the data adjustment device 100 executes an adjustment process for adjusting the data set DS1.

[0039] Then, the data adjustment device 100 retrains the model M1 using the adjusted data set DS1 (step S10). For example, the data adjustment device 100 retrains the model M1 using the adjusted data set DS1 in which data with low contribution degrees such as data DT14 is excluded and data similar to data with high contribution degrees such as data DT33 is added.

[0040] As described above, the data adjustment device 100 executes an adjustment process for adjusting the data set DS1 by excluding data with low contribution degrees from the data set DS1 and adding data similar to data with high contribution degrees to the data set DS1. In this way, the data adjustment device 100 can adjust the data used for learning by excluding or adding data according to the contribution degree of each data to learning.

[0041] Also, when the data adjustment device 100 adds data corresponding to data with high contribution degrees, it requests the data from the terminal device 10. Then, the terminal device 10 that has received the request provides the data corresponding to the request to the data adjustment device 100 as data for provision. Thereby, the terminal device 10 can make the data used for learning adjustable.

[0042] As described above, the data adjustment system 1 can make the data used for learning adjustable by excluding data with low contribution degrees from the data set or adding data corresponding to data with high contribution degrees to the data set.

[0043] [1-1-1. Background and effects, etc.] Here, the background, effects, etc. of the above-described data adjustment system 1 will be explained. Through deep learning, predictions beyond human capabilities have been realized. However, the basis for the judgment of artificial intelligence cannot be known, and judgments are made by black boxes. To improve the accuracy of deep learning, there is a problem that a large amount of data is required. In recent years, research on elucidating the basis for such judgments has become active. The reason for the judgment of deep learning is to search for the cause from the result. With such a scientific approach, it may be possible to understand what data is necessary to improve the accuracy of deep learning.

[0044] Conventionally, although artificial intelligence has high performance, it has been called a black box. Deep learning has a structure that mimics the neurons of the human body, and a model is formed by optimizing a very large number of parameters. Due to its complexity, it is difficult to explain. In recent years, research on explainable artificial intelligence has become active, and various algorithms have been proposed. The research remains at the academic level, and the development into practical systems has been delayed.

[0045] By searching for the cause from the result judged by deep learning, the necessary data can be selected. By using technologies such as the selection of harmful and beneficial data, data insufficiency due to unlearning, the limit of estimation due to noise, and the detection of mislabeled data, the data in deep learning can be selected. It is a very laborious task for humans to adjust these operations. Therefore, a system that automatically readjusts learning data like the data adjustment system 1 finds the cause from misjudged data and automatically prepares an optimal dataset for relearning. In the data adjustment system 1, relearning is executed using the adjusted dataset, and by repeating these through a loop, further improvement in prediction accuracy can be achieved. An explanation of this point will be given below using Figure 2.

[0046] [1-1-2. Concept of Data Adjustment System] Figure 2 is a conceptual diagram of data adjustment processing according to an embodiment of the present disclosure. The process PS in Figure 2 shows an overall conceptual diagram of the automatic data adjustment process realized by the data adjustment system 1. Regarding the processes described below with the data adjustment system 1 as the main body of the process, any device included in the data adjustment system 1 such as the data adjustment device 100 or the terminal device 10 may perform the process.

[0047] First, the overall processing outline of the process PS in Figure 2 will be described. In the process PS by the data adjustment system 1, as shown in the learning LN in Figure 2, a process of learning the model NN, which is a neural network, using the dataset DS is performed. In the process PS by the data adjustment system 1, as shown in the input IN in Figure 2, test data such as the data TD is input to the learned model NN.

[0048] Then, in the process PS by the data adjustment system 1, as shown in the output OUT in Figure 2, an output (identification result) is obtained from the model NN in response to the input of the test data. And in the process PS by the data adjustment system 1, when the output (identification result) from the model NN is an error (misrecognition), learning is performed by feeding back that information.

[0049] Hereinafter, each process will be described individually. The data adjustment system 1 identifies the cause in the data where an error determination (misrecognition) has occurred. For example, as shown in the method MT1, the data adjustment system 1 classifies (assigns) whether the data is harmful or beneficial data by using the Influence function. The data adjustment system 1 repeats the calculation loop to minimize the loss function of deep learning by removing the harmful data.

[0050] For example, the Influence function can also select an optimal model. The accuracy of the data varies depending on the model. For example, the data adjustment system 1 may be configured to automatically select a model with a small distribution of harmful data.

[0051] For example, as shown in method MT2, the data adjustment system 1 can determine (know) the case where the accuracy is not achieved due to insufficient data by Bayesian DNN. In this case, the data adjustment system 1 can improve the accuracy by automatically supplementing the required data from the data lake and then retraining. Also, for example, as shown in method MT3, the data adjustment system 1 can also complement the data by generating data by GAN (Generative Adversarial Network). Details about Bayesian DNN and GAN will be described later.

[0052] Also, Bayesian DNN is a technology that can determine (know) the case where no further improvement in accuracy can be expected even with further learning due to noise or the like. After the accuracy of the data adjustment system 1 is improved to a certain extent by running the learning loop as described above, it can notify (inform) humans of the limit beyond which the accuracy cannot be further improved.

[0053] In the data adjustment system 1, the basis for judgment is such that in Grad-CAM (Gradient-weighted Class Activation Mapping), LIME (Local Interpretable Model-agnostic Explanations), etc., humans can know the reason for judgment by visualizing what is causing the problem. Details about Grad-CAM and LIME will be described later. Thus, the data adjustment system 1 is a self-growing learning system that integrates automatic data adjustment in deep learning learning, for example.

[0054] As described above, the data adjustment system 1 inputs test data into a network learned by deep learning. The data adjustment system 1 is a system that automatically adjusts data by identifying the cause in case of misjudgment. The data adjustment system 1 regenerates the network by re-learning with the adjusted data. The data adjustment system 1 conducts tests to clarify the causes of misjudgments that still remain. The data adjustment system 1 repeats a loop of automatically adjusting data and re-learning so as to improve accuracy. In these cause clarification techniques, the data adjustment system 1 uses Influence function, Bayesian DNN, etc. to determine beneficial / harmful data and identify unlearned / limiting factors. Thus, the data adjustment system 1 is characterized by integrating data and deep learning.

[0055] For example, by the data adjustment system 1, in deep learning that requires a large amount of data, high-quality data can be automatically selected to improve accuracy. The data adjustment system 1 can automatically select data for which accuracy should be improved by identifying scientific causes without humans adjusting the data intuitively. Since the data adjustment system 1 is a loop system, accuracy can be improved by leaving the calculation to a computer without human intervention.

[0056] [1-1-3.Influence function (Influence Function)] Next, each method in the data adjustment system 1 will be described. First, the Influence function will be described. The data adjustment system 1 quantitatively analyzes the influence that each piece of data in the data set exerts on the model (parameters) generated by the data using the Influence function. For example, the data adjustment system 1 formulates the influence that the presence or absence of a certain (learning) data has on the accuracy (output result) of the model using the Influence function (influence function). For example, the data adjustment system 1 measures the degree of influence that each piece of data has on learning without re-learning using the data set excluding each piece of data to be measured for influence. Hereinafter, the measurement of the degree of influence using the Influence function (influence function) will be described using mathematical formulas and the like.

[0057] The Influence function is also used, for example, as a method for explaining the black-box model of machine learning.

[0058] Note that the Influence function is disclosed, for example, in the following literature. ·Understanding Black-box Predictions via Influence Functions, Pang Wei Kho and Percy Liang <https: / / arxiv.org / abs / 1703.04730>

[0059] By using the Influence function, the data adjustment system 1 can calculate the contribution degree of the data to machine learning and measure (know) how much positive or negative influence a certain piece of data has. For example, the data adjustment system 1 calculates (measures) the degree of influence according to algorithms, data, etc. as shown below. Hereinafter, the case where an image is used as input data will be described as an example.

[0060] For example, consider the prediction problem in machine learning with an input x (image) and an output y (label). Each image is assigned a label, that is, the image and the correct label are associated. For example, assume there are n (n is an arbitrary natural number) sets of images and labels (dataset). Then each labeled image z (sometimes simply referred to as "image z") is as follows in Equation (1).

[0061]

Number

[0062] Here, if the loss at a certain point z (image z) with the model parameter θ ∈ Θ is denoted as L(z, θ), the empirical loss for all n data can be expressed as follows in Equation (2).

[0063]

Number

[0064] And since minimizing this empirical loss means finding (determining) the parameter that minimizes the loss, it can be expressed as follows in Equation (3).

[0065]

Number

[0066] For example, the data adjustment system 1 calculates the parameter that minimizes the loss (the left side of Equation (3)) using Equation (3). Here, assume that the empirical loss is twice differentiable and is a convex function with respect to the parameter θ. Below, aiming to understand the influence degree of the data which is the training point of the machine learning model, it will be shown how to calculate. Suppose there is no data at a certain training point, and consider what influence it will have on the machine learning model.

[0067] Note that, like the parameter (variable) with a "^" (hat) attached above "θ" shown on the left side of Equation (3), a parameter (variable) with a "^" attached above a certain character indicates, for example, a predicted value. Hereinafter, when referring to the parameter (variable) with a "^" attached above "θ" shown on the left side of Equation (3) in the text, it is denoted as "θ^" with "^" described following "θ". When removing a certain training point z (image z) from the machine learning model, it can be expressed as the following Equation (4).

[0068]

Number

[0069] For example, the data adjustment system 1 calculates the parameter (the left side of Equation (4)) when performing learning without using a certain training data (image z) by using Equation (4). For example, the influence degree is the difference between when the training point z (image z) is removed and when all data points including the training point z are present. This difference is shown as the following Equation (5).

[0070]

Number

[0071] Here, when recalculating the case where the image z is removed, the calculation cost is very high. Therefore, the data adjustment system 1 uses Influence functions to perform operations by effective approximation as shown below without recalculating (relearning) the case where the image z is removed.

[0072] This concept is a method of calculating the change in parameters assuming that the image z is weighted by a minute ε. Here, a new parameter (the left side of Equation (6)) is defined by using the following Equation (6).

[0073]

Number

[0074] By utilizing the results of previous studies by Cook and Weisberg in 1982, the influence degree of the weighted image z at the parameter θ^ (the left side of Equation (3)) can be expressed as in the following Equations (7) and (8).

[0075]

Number

[0076]

Number

[0077] Note that the previous studies by Cook and Weisberg are disclosed, for example, in the following literature. ·Residuals and Influence in Regression, Cook, R.D. and Weisberg, S <https: / / conservancy.umn.edu / handle / 11299 / 37076>

[0078] For example, Equation (7) represents the influence function corresponding to a certain image z. For example, Equation (7) represents the change amount of the parameter for a tiny ε. Also, for example, Equation (8) represents the Hessian (Hessian matrix). Here, assuming that it is a Hessian matrix with positive definite values and the inverse matrix also exists. Assuming that removing a certain data point z (image z) at a certain point is the same as being weighted by "ε = -1 / n", the parameter change when the image z is removed can be approximately expressed as in the following Equation (9).

[0079]

Number

[0080] That is, the data adjustment system 1 can measure (obtain) the influence degree when removing the data point z (image z) without re-learning.

[0081] Next, the data adjustment system 1 measures (obtains) the influence degree on the loss at a certain test point z test using the following formulas (10-1) to (10-3).

[0082]

Equation

[0083] In this way, the influence degree of the weighted image z at a certain test point z test can be formulated. Therefore, the data adjustment system 1 can measure (obtain) the influence degree of the data in the machine learning model by this operation. For example, the right side of formula (10-3) consists of the gradient with respect to the loss of a certain data, the inverse matrix of the Hessian, the gradient of the loss of a certain training data, etc. For example, the influence that a certain data has on the prediction (loss) of the model can be obtained by formula (10-3). Note that the above is an example, and the data adjustment system 1 may appropriately execute various operations to measure the influence degree that each image has on learning.

[0084] [1-1-4.Bayesian Deep Learning] Next, Bayesian Deep Learning will be described. The data adjustment system 1 can estimate, for example, by Bayesian Deep Learning using the method MT2 (Bayesian DNN), what causes the model accuracy not to increase. In this way, the data adjustment system 1 can make a judgment regarding the model accuracy by the technology of Bayesian Deep Learning. Hereinafter, Bayesian Deep Learning will be explained while describing the premises.

[0085] First, generally, the inference of deep learning models is highly accurate, but there are limitations to inference. Knowing the limitations where inference cannot be made is very important in effectively using deep learning. However, it is impossible to completely eliminate the uncertainty of deep learning. The following describes what uncertainty in deep learning is.

[0086] There are two types of uncertainty in deep learning. The uncertainty in deep learning can be divided into aleatoric uncertainty and epistemic uncertainty. The former, aleatoric uncertainty, is caused by noise in observations and not by insufficient data. For example, cases such as hidden and unseeable images (occlusions) fall under this (aleatoric uncertainty). The mouth area of a person's face covered by a mask cannot be observed as data in the first place because it is hidden by the mask. On the other hand, the latter, epistemic uncertainty, represents the uncertainty regarding insufficient data. If there is sufficient data, the epistemic uncertainty can be improved. However, generally, it has been difficult to clarify the epistemic uncertainty in the field of image recognition.

[0087] With the proposal of Bayesian Deep Learning, it has become possible to clarify uncertainty.

[0088] Note that Bayesian Deep Learning is disclosed, for example, in the following literature. ·What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vison, NIPS 2017, Alex Kendall and Yarin Gal <https: / / papers.nips.cc / paper / 7141-what-uncertainties-do-we-need-in-bayesian-deep-learning-for-computer-vision.pdf>

[0089] Bayesian deep learning combines Bayesian inference and deep learning. By using Bayesian inference, we can evaluate uncertainty because we can understand how the estimation results vary.

[0090] Bayesian deep learning is a method that uses dropout in the learning of deep learning and estimates from the variance results obtained in inference. Dropout is a technique that is very commonly used to reduce overfitting by randomly reducing the number of neurons in each layer.

[0091] The mathematical theory regarding the role of dropout in Bayesian deep learning is disclosed, for example, in the following literature. ·Dropout as Bayesian Approximation: Representing Model Uncertainty in Deep Learning, ICML 2016, Yarin Gal and Zoubin Ghahramani <https: / / arxiv.org / pdf / 1506.02142.pdf>

[0092] In conclusion, using dropout in deep learning is equivalent to performing Bayesian learning. For example, the data adjustment system 1 can calculate the posterior distribution of weights by combining the values obtained through learning, which are not deterministic, with dropout. For example, the data adjustment system 1 can estimate the variance of the posterior distribution from the variations in the multiple outputs generated with multiple dropout coefficients.

[0093] Bayesian deep learning samples from the weight distribution by using dropout not only during learning but also during inference. For example, the data adjustment system 1 can sample from the weight distribution by using dropout not only during learning but also during inference by means of the Monte Carlo dropout method. For example, the data adjustment system 1 can determine the uncertainty of the inference results by repeating the inference multiple times for the same input. The network trained with dropout has a structure with some neurons missing. Therefore, when the data adjustment system 1 inputs an input image for inference, it can obtain an output characterized by its weights by passing through the neurons missing due to dropout. Furthermore, when the same image is input, the output is obtained by passing through different paths within the network, so the weighted outputs are different. That is, the network with dropout can obtain different output distributions during inference for the same input image. A large variance in the output means a large uncertainty in the model. The average of the distribution obtained by multiple inferences represents the final predicted value, and the variance represents the uncertainty of the predicted value. Bayesian deep learning represents the uncertainty from the variance of the output during this inference. The data adjustment system 1 can perform an estimation (judgment) regarding the uncertainty of the model by means of Bayesian deep learning as described above.

[0094] [1-1-5. Others (GAN, Grad-CAM, LIME, etc.)] The data adjustment system 1 may use various methods, not limited to the above-mentioned Influence function and Bayesian Deep Learning. This will be described below.

[0095] The data adjustment system 1 may appropriately use various methods to automatically generate data (training data) for learning. For example, the data adjustment system 1 may (automatically) generate training data by GAN.

[0096] Note that GAN is disclosed in, for example, the following literature. ·Generative Adversarial Networks, Ian J. Goodfellow et al. <https: / / arxiv.org / abs / 1406.2661>

[0097] The data adjustment system 1 may generate data with high influence by GAN from the data measured to have high influence by Influence functions. For example, the data adjustment system 1 may generate data with high influence by the architecture of GAN including a discriminator that discriminates images with high influence and a generator that generates images with high influence. Note that the above is just an example, and the data adjustment system 1 may appropriately use GAN technology to generate data with high influence.

[0098] The data adjustment system 1 may visualize the basis for the output (judgment) of the model by appropriately using various methods. For example, the data adjustment system 1 generates basis information for visualizing the basis for the output (judgment) of the model after image input by Grad-CAM. The data adjustment system 1 generates basis information indicating the basis on which the model M1 for detecting smiles determines the presence or absence of a smile by Grad-CAM. For example, the data adjustment system 1 generates basis information by means of processing related to Grad-CAM as disclosed in the following literature. The data adjustment system 1 uses the technology of Grad-CAM, a visualization method applicable to the entire network including CNN, to generate basis information indicating the basis for the output of the model M1. For example, the data adjustment system 1 can visualize the parts that affect each class by calculating the weights of each channel from the final layer of the CNN and multiplying the weights together. In this way, the data adjustment system 1 can visualize which parts of the image the neural network including CNN focuses on for making a judgment.

[0099] ·Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization <https: / / arxiv.org / abs / 1610.02391>

[0100] Note that the description of the Grad-CAM technique will be omitted as appropriate. The data adjustment system 1 generates basis information by the method of Grad-CAM (see the above literature). For example, the data adjustment system 1 designates a target type (class) and generates information (image) corresponding to the designated class. For example, the data adjustment system 1 uses the Grad-CAM technique to generate information (image) for the designated class through various processes such as backpropagation. For example, the data adjustment system 1 designates the class of "smiling face" and generates an image regarding the basis information corresponding to the class of "smiling face". For example, the data adjustment system 1 generates an image showing the area (region) being focused on for the recognition (classification) of the class of "smiling face" in the form of a so-called heatmap (color map).

[0101] Also, the data adjustment system 1 associates the input data (image) with the basis information indicating the basis of its judgment result and stores it in the storage unit 120 (see FIG. 4) as a log (history). This makes it possible to verify by what judgment the data adjustment system 1 performed its subsequent operations for what input. Also, for example, the data adjustment system 1 may use the log of the input data (image) stored in the storage unit 120 and the basis information indicating the basis of its judgment result for various processes. For example, the data adjustment system 1 may generate data using the log of the input data (image) and the basis information indicating the basis of its judgment result. For example, the data adjustment system 1 may generate an image obtained by changing the input image so that the image includes the image of the area indicated by the heatmap, which is the basis information. Note that the above is an example, and the data adjustment system 1 may generate data from the log using various methods as appropriate.

[0102] Note that the basis information generated by the data adjustment system 1 is not limited to images such as heatmaps, and may be information in various forms such as character information and voice information. Further, the data adjustment system 1 is not limited to Grad-CAM, and may appropriately use various methods to visualize the basis for the output (judgment) of the model. For example, the data adjustment system 1 may generate basis information by methods such as LIME and TCAV (Testing with Concept Activation Vectors).

[0103] For example, the data adjustment system 1 may generate basis information using the technology of LIME. For example, the data adjustment system 1 may generate basis information by the processing related to LIME as disclosed in the following documents.

[0104] ·"Why Should I Trust You?": Explaining the Predictions of Any Classifier <https: / / arxiv.org / abs / 1602.04938>

[0105] Note that the explanation of the technology of LIME will be omitted as appropriate, but the data adjustment system 1 generates basis information by the method of LIME (see the above document). For example, the data adjustment system 1 generates another model (basis model) for local approximation to show the reason (basis) why the model made such a judgment. The data adjustment system 1 generates a basis model for local approximation for the combination of the input information and the output result corresponding to the input information. Then, the data adjustment system 1 generates basis information using the basis model. Further, the data adjustment system 1 may use a calculation method (generation method) of basis information such as "Testing with Concept Activation Vectors" called TCAV as disclosed in the following documents.

[0106] ·Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) <https: / / arxiv.org / pdf / 1711.11279.pdf>

[0107] For example, the data adjustment system 1 generates a plurality of input information by duplicating or modifying the underlying input information (target input information) such as an image. Then, the data adjustment system 1 inputs each of the plurality of input information into a model (model to be explained) that is the generation target of the basis information, and causes the model to be explained to output a plurality of output information corresponding to each input information. Then, the data adjustment system 1 uses the combination (pair) of each of the plurality of input information and each of the corresponding plurality of output information as learning data to train a basis model. In this way, the data adjustment system 1 generates a basis model that locally approximates with another interpretable model (such as a linear model) for the target input information.

[0108] In this way, when the data adjustment system 1 obtains the output of the model for a certain input, it generates a basis model for showing the basis (local explanation) of the output. For example, the data adjustment system 1 generates an interpretable model such as a linear model as the basis model. The data adjustment system 1 generates basis information based on information such as each parameter of the basis model such as a linear model. For example, the data adjustment system 1 generates basis information indicating that the influence of a feature amount with a large weight among the feature amounts of the basis model such as a linear model is large.

[0109] As described above, the data adjustment system 1 generates basis information based on the basis model learned using the input information and output result of the model. In this way, the data adjustment system 1 may generate basis information based on the state information including the output result of the model after inputting the input information to the model.

[0110] [1-2. Configuration of the data adjustment system according to the embodiment] The data adjustment system 1 shown in FIG. 3 will be described. The data adjustment system 1 is an information processing system that realizes an adjustment process for adjusting learning data. As shown in FIG. 3, the data adjustment system 1 includes a data adjustment device 100 and a plurality of terminal devices 10a, 10b, 10c, and 10d. When not distinguishing between the terminal devices 10a, 10b, 10c, 10d, etc., they may be described as the terminal device 10. Also, in FIG. 3, four terminal devices 10a, 10b, 10c, and 10d are illustrated, but the data adjustment system 1 may include more than four terminal devices 10 (for example, 20 or more than 100). The terminal device 10 and the data adjustment device 100 are communicably connected by wire or wirelessly via a predetermined communication network (network N). FIG. 3 is a diagram showing a configuration example of the data adjustment system according to the embodiment. Note that the data adjustment system 1 shown in FIG. 3 may include a plurality of data adjustment devices 100.

[0111] The data adjustment device 100 is an information processing device (computer) that measures the degree of influence that the data included in the data set used for learning a model by machine learning has on the learning, and adjusts the data set based on the measurement result. Also, the data adjustment device 100 executes a learning process using the data set. Further, the data adjustment device 100 requests the terminal device 10 for data to be added to the data set.

[0112] The terminal device 10 is a computer that provides data to the data adjustment device 100 in response to a request from the data adjustment device 100. In the example of FIG. 3, the terminal device 10a is a data server that holds data. The terminal device 10a may be a data server that holds data such as videos, images, and character information. For example, the terminal device 10a may be a data server that holds content data such as TV programs, movies, and music.

[0113] Also, in the example of FIG. 3, the terminal device 10b is a camera having an imaging function. The terminal device 10b is a camera that captures videos and images and holds the captured data.

[0114] In the example of FIG. 3, the terminal device 10c is an image sensor (imager) having an imaging function. For example, the terminal device 10c has a function of communicating with the data adjustment device 100 and has a function of transmitting captured images and videos to the data adjustment device 100. For example, the terminal device 10c captures an image or a video in response to a request from the data adjustment device 100 and transmits the captured image or video to the data adjustment device 100.

[0115] In the example of FIG. 3, the terminal device 10d is a moving body such as a UAV like a drone or a vehicle such as an automobile. For example, the terminal device 10d has a function of communicating with the data adjustment device 100 and may move in response to a request from the data adjustment device 100. The terminal device 10d has an imaging function such as an image sensor (imager), moves to a position according to a request from the data adjustment device 100, captures an image or a video at that position, and transmits the captured image or video to the data adjustment device 100.

[0116] Note that the terminal device 10 may be any device as long as it can realize the processing in the embodiment. The terminal device 10 may be, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. The terminal device 10 may be a wearable device (Wearable Device) worn by a user. For example, the terminal device 10 may be a wristwatch-type terminal, a glasses-type terminal, or the like. Further, the terminal device 10 may be a so-called household appliance such as a television or a refrigerator. For example, the terminal device 10 may be a robot that interacts with a human (user), such as a smart speaker, an entertainment robot, or a household robot. Further, the terminal device 10 may be a device arranged at a predetermined position such as a digital signage.

[0117] [1-3. Configuration of Data Adjustment Device According to Embodiment] Next, the configuration of a data adjustment device 100, which is an example of a data adjustment device that executes the data adjustment process according to the embodiment, will be described. FIG. 4 is a diagram showing a configuration example of the data adjustment device 100 according to the embodiment of the present disclosure.

[0118] As shown in FIG. 4, the data adjustment device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. Note that the data adjustment device 100 may include an input unit (e.g., a keyboard, a mouse, etc.) that receives various operations from an administrator or the like of the data adjustment device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0119] The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. The communication unit 110 is connected to the network N (see FIG. 3) by wire or wirelessly, and transmits and receives information to and from other information processing devices such as the terminal device 10. The communication unit 110 may also transmit and receive information to and from the terminal device 10.

[0120] The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 according to the embodiment includes, as shown in FIG. 4, a data information storage unit 121, a model information storage unit 122, a threshold information storage unit 123, and a knowledge information storage unit 125.

[0121] The data information storage unit 121 according to the embodiment stores various information related to the data used for learning. The data information storage unit 121 stores a data set used for learning. FIG. 5 is a diagram showing an example of the data information storage unit according to the embodiment of the present disclosure. For example, the data information storage unit 121 stores various information related to various data such as learning data used for learning and evaluation data used for accuracy evaluation (measurement). FIG. 5 shows an example of the data information storage unit 121 according to the embodiment. In the example of FIG. 5, the data information storage unit 121 includes items such as "data set ID", "data ID", and "data".

[0122] The "dataset ID" indicates identification information for identifying a dataset. The "data ID" indicates identification information for identifying an object. Also, "data" indicates data corresponding to the object identified by the data ID. That is, in the example of FIG. 5, vector data (data) corresponding to the object is associated and registered with the data ID for identifying the object.

[0123] In the example of FIG. 5, it shows that a dataset (dataset DS1) identified by the dataset ID "DS1" includes a plurality of data identified by data IDs "DID1", "DID2", "DID3", etc. For example, each data (learning data) identified by data IDs "DID1", "DID2", "DID3", etc. is image information or the like used for learning a smile detection model.

[0124] Note that the data information storage unit 121 is not limited to the above, and may store various information according to the purpose. The data information storage unit 121 stores correct answer information (correct label) corresponding to each data in association with each data. For example, the data information storage unit 121 stores correct answer information (correct label) indicating whether or not a smile is included in each data (image) in association with each data.

[0125] Also, the data information storage unit 121 may store in a manner that enables identification of whether each data is learning data, evaluation data, or the like. For example, the data information storage unit 121 stores the learning data and the evaluation data in a distinguishable manner. The data information storage unit 121 may store information for identifying whether each data is learning data or evaluation data. The data adjustment device 100 learns the model based on each data used as learning data and the correct answer information. The data adjustment device 100 measures the accuracy of the model based on each data used as evaluation data and the correct answer information. The data adjustment device 100 measures the accuracy of the model by collecting the result of comparing the output result output by the model when the evaluation data is input with the correct answer information.

[0126] The model information storage unit 122 according to the embodiment stores information related to the model. For example, the model information storage unit 122 stores information (model data) indicating the structure of the model (network). FIG. 6 is a diagram showing an example of the model information storage unit according to the embodiment of the present disclosure. FIG. 6 shows an example of the model information storage unit 122 according to the embodiment. In the example shown in FIG. 6, the model information storage unit 122 includes items such as "model ID", "usage", and "model data".

[0127] "Model ID" indicates identification information for identifying the model. "Usage" indicates the usage of the corresponding model. "Model data" indicates the data of the model. In FIG. 6, an example is shown in which conceptual information such as "MDT1" is stored in "model data", but actually, various information related to the network included in the model, such as information and functions, is included in the model.

[0128] In the example shown in FIG. 6, the model (model M1) identified by the model ID "M1" indicates that the usage is "image recognition (smile detection)". Model M1 is a model used for image recognition and is used for smile detection. Also, the model data of model M1 indicates that it is model data MDT1.

[0129] Note that the model information storage unit 122 is not limited to the above, and may store various information according to the purpose. For example, the model information storage unit 122 stores parameter information of the model learned (generated) by the learning process.

[0130] The threshold information storage unit 123 according to the embodiment stores various information related to the threshold. The threshold information storage unit 123 stores various information related to the threshold used for comparison with the score. FIG. 7 is a diagram showing an example of the threshold information storage unit according to the embodiment. The threshold information storage unit 123 shown in FIG. 7 includes items such as "threshold ID" and "threshold".

[0131] "Threshold ID" indicates identification information for identifying a threshold value. Also, "threshold value" indicates the specific value of the threshold value identified by the corresponding threshold ID. Further, for each threshold value, information indicating its use is stored in an associated manner.

[0132] In the example of FIG. 7, for the threshold value (first threshold value TH1) identified by the threshold ID "TH1", information indicating that it is used for discriminating data with a low degree of influence is stored in an associated manner. In this case, the first threshold value TH1 is used to discriminate data with a low degree of influence, that is, data to be excluded. Also, the value of the first threshold value TH1 is indicated as "VL1". Note that in the example of FIG. 7, it is shown by an abstract symbol such as "VL1", but the value of the first threshold value TH1 is a specific numerical value (for example, 0.3, etc.).

[0133] Also, for the threshold value (second threshold value TH2) identified by the threshold ID "TH2", information indicating that it is used for discriminating data with a high degree of influence is stored in an associated manner. In this case, the second threshold value TH2 is used to discriminate data with a high degree of influence, that is, data to be the target of adding new data. Also, the value of the second threshold value TH2 is indicated as "VL2". Note that in the example of FIG. 7, it is shown by an abstract symbol such as "VL2", but the value of the second threshold value TH2 is a specific numerical value (for example, 0.75, etc.).

[0134] Note that the threshold value information storage unit 123 is not limited to the above, and may store various information according to the purpose.

[0135] Returning to FIG. 4 and continuing the description. The control unit 130 is realized, for example, by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc., when a program stored inside the data adjustment device 100 (for example, an information processing program such as the data adjustment processing program according to the present disclosure) is executed using a RAM (Random Access Memory) or the like as a work area. Further, the control unit 130 is a controller and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0136] As shown in FIG. 4, the control unit 130 includes an acquisition unit 131, a learning unit 132, a measurement unit 133, an adjustment unit 134, and a transmission unit 135, and realizes or executes the functions and operations of information processing described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and any other configuration may be used as long as it can perform the information processing described later. Also, the connection relationship between the respective processing units included in the control unit 130 is not limited to the connection relationship shown in FIG. 4, and other connection relationships may be used.

[0137] The acquisition unit 131 acquires various types of information. The acquisition unit 131 acquires various types of information from an external information processing device. The acquisition unit 131 acquires various types of information from the terminal device 10.

[0138] The acquisition unit 131 acquires various types of information from the storage unit 120. The acquisition unit 131 acquires various types of information from the data information storage unit 121, the model information storage unit 122, and the threshold information storage unit 123.

[0139] The acquisition unit 131 acquires various types of information learned by the learning unit 132. The acquisition unit 131 acquires various types of information measured by the measurement unit 133. The acquisition unit 131 acquires various types of information adjusted by the adjustment unit 134.

[0140] The learning unit 132 learns various types of information. The learning unit 132 learns various types of information based on information from an external information processing device and information stored in the storage unit 120. The learning unit 132 learns various types of information based on the information stored in the data information storage unit 121. The learning unit 132 stores the model generated by learning in the model information storage unit 122.

[0141] The learning unit 132 performs learning processing. The learning unit 132 performs various types of learning. The learning unit 132 learns various types of information based on the information acquired by the acquisition unit 131. The learning unit 132 learns (generates) a model. The learning unit 132 learns various types of information such as models. The learning unit 132 generates a model by learning. The learning unit 132 uses techniques related to various machine learning to learn the model. For example, the learning unit 132 learns the parameters of the model (network). The learning unit 132 uses techniques related to various machine learning to learn the model.

[0142] The learning unit 132 learns the parameters of the network. For example, the learning unit 132 learns the parameters of the network of model M1. The learning unit 132 learns the parameters of the network of model M1.

[0143] The learning unit 132 performs learning processing based on the learning data (teacher data) stored in the data information storage unit 121. The learning unit 132 generates model M1 by performing learning processing using the learning data stored in the data information storage unit 121. For example, the learning unit 132 generates a model used for image recognition (smile detection). The learning unit 132 generates model M1 by learning the parameters of the network of model M1.

[0144] The learning method by the learning unit 132 is not particularly limited. For example, learning data that associates label information (such as the presence or absence of a smile) with an image group may be prepared, and the learning data may be input into a calculation model based on a multi-layer neural network for learning. Also, for example, a method based on a DNN (Deep Neural Network) such as CNN (Convolutional Neural Network) or 3D-CNN may be used. When the learning unit 132 targets time-series data such as moving images (videos) of videos and the like, a method based on a recurrent neural network (Recurrent Neural Network: RNN) or LSTM (Long Short-Term Memory units) that extends RNN may be used.

[0145] The learning unit 132 executes a learning process using a data set. The learning unit 132 executes a learning process using the data set adjusted by the adjustment unit 134. The learning unit 132 updates the model by executing a learning process using the data set adjusted by the adjustment unit 134. The learning unit 132 updates the parameters of the model by executing a learning process using the data set adjusted by the adjustment unit 134. The learning unit 132 updates the model M1 by executing a learning process using the data set adjusted by the adjustment unit 134.

[0146] The measurement unit 133 measures various processes. The measurement unit 133 functions as a measurement means. The measurement unit 133 functions as a measurement means for measuring the degree of influence exerted on learning by the learning data used for learning the neural network. The measurement unit 133 measures various processes based on information from an external information processing device. The measurement unit 133 measures various processes based on the information stored in the storage unit 120. The measurement unit 133 measures various processes based on the information stored in the data information storage unit 121, the model information storage unit 122, or the threshold information storage unit 123. The measurement unit 133 generates various information by measuring the processes.

[0147] Based on the various information acquired by the acquisition unit 131, the measurement unit 133 measures various processes. Based on the various information learned by the learning unit 132, the measurement unit 133 measures various processes. Based on the various information acquired by the acquisition unit 131, the measurement unit 133 extracts various information. Based on the various information learned by the learning unit 132, the measurement unit 133 extracts various information. Based on the information adjusted by the adjustment unit 134, the measurement unit 133 extracts various information.

[0148] The measurement unit 133 determines various information. The measurement unit 133 judges various information. The measurement unit 133 discriminates various information. Based on the influence degree of each data, the measurement unit 133 discriminates the necessity of each data.

[0149] The measurement unit 133 measures the influence degree that the learning data used for learning the model by machine learning has on the learning. The measurement unit 133 measures the influence degree based on the loss function. The measurement unit 133 measures the influence degree by a method available for influence degree measurement. The measurement unit 133 measures the influence degree by the Influence function. Based on the difference between the case of the data set and the case where one data is excluded from the data set, the measurement unit 133 measures the influence degree of one data. The measurement unit 133 measures the influence degree of the learning data used for learning the neural network.

[0150] The adjustment unit 134 adjusts various information. The adjustment unit 134 functions as an adjustment means for adjusting the data set. The adjustment unit 134 functions as an adjustment means for excluding the data measured to have a low influence degree from the data set, acquiring new data which is new data corresponding to the data measured to have a high influence degree, and adding the acquired new data to the data set. Based on the information from an external information processing device or the information stored in the storage unit 120, the adjustment unit 134 adjusts various information. Based on the information from another information processing device such as the terminal device 10, the adjustment unit 134 adjusts various information. Based on the information stored in the data information storage unit 121, the model information storage unit 122, or the threshold information storage unit 123, the adjustment unit 134 adjusts various information.

[0151] The adjustment unit 134 adjusts various types of information based on the various types of information acquired by the acquisition unit 131. The adjustment unit 134 adjusts various types of information based on the various types of information learned by the learning unit 132. The adjustment unit 134 adjusts various types of information based on the various types of information adjusted by the process measurement of the measurement unit 133.

[0152] The adjustment unit 134 adjusts the data set by excluding data from the data set or adding new data to the data set based on the measurement results by the measurement unit 133. The adjustment unit 134 excludes first data with a low influence degree from the data set. The adjustment unit 134 excludes first data with an influence degree lower than the first threshold value from the data set.

[0153] The adjustment unit 134 adds new data, which is new data corresponding to second data with a high influence degree, to the data set. The adjustment unit 134 adds new data corresponding to second data with an influence degree higher than the second threshold value to the data set. The adjustment unit 134 adds new data acquired from an external device to the data set. The adjustment unit 134 adds new data acquired from a storage unit that stores data to the data set.

[0154] The adjustment unit 134 generates new data and adds the generated new data to the data set. The adjustment unit 134 generates new data using the second data and adds the generated new data to the data set. The adjustment unit 134 generates new data by data augmentation and adds the generated new data to the data set. The adjustment unit 134 generates new data similar to the second data and adds the generated new data to the data set. For example, the adjustment unit 134 uses the second data as the original data and generates an image similar to the original data by data augmentation. For example, the adjustment unit 134 uses the second data as the original data and generates an image similar to the original data by reducing the original data, enlarging a part of the original data, rotating it left and right, or moving it in the up, down, left, and right directions. Note that the above is an example, and the adjustment unit 134 may generate new data to be added to the data set by various methods. For example, the adjustment unit 134 may generate new data to be added to the data set by a method such as GAN described above.

[0155] The transmission unit 135 transmits various information. The transmission unit 135 transmits various information to an external information processing device. The transmission unit 135 provides various information to an external information processing device. For example, the transmission unit 135 transmits various information to another information processing device such as the terminal device 10. The transmission unit 135 provides the information stored in the storage unit 120. The transmission unit 135 transmits the information stored in the storage unit 120.

[0156] The transmission unit 135 provides various information based on information from another information processing device such as the terminal device 10. The transmission unit 135 provides various information based on the information stored in the storage unit 120. The transmission unit 135 provides various information based on the information stored in the data information storage unit 121, the model information storage unit 122, or the threshold information storage unit 123.

[0157] The transmission unit 135 transmits request information for requesting new data to an external device. The transmission unit 135 transmits request information for requesting new data to the terminal device 10. The transmission unit 135 transmits request information for requesting data similar to the learning data whose influence degree on learning in the machine learning model is equal to or higher than a predetermined standard to the terminal device 10. The transmission unit 135 transmits request information for requesting data similar to the learning data whose influence degree on learning in the machine learning model is equal to or higher than a predetermined threshold to the terminal device 10.

[0158] [1-3-1. Model (Network) Example] As described above, the data adjustment device 100 may use a model (network) in the form of a neural network (NN) such as a deep neural network (DNN). Note that the data adjustment device 100 is not limited to a neural network, and may use various forms of models (functions) such as a regression model such as SVM (Support Vector Machine). Thus, the data adjustment device 100 may use a model (function) in any form. The data adjustment device 100 may use various regression models such as a non-linear regression model or a linear regression model.

[0159] Regarding this point, an example of the network structure of the model will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of the network corresponding to the model. The network NW1 shown in FIG. 8 shows a neural network including a plurality (multi-layer) of intermediate layers between the input layer INL and the output layer OUTL. The network NW1 shown in FIG. 8 corresponds to the neural network NN in FIG. 1. For example, the data adjustment device 100 may learn the parameters of the network NW1 shown in FIG. 8.

[0160] The network NW1 shown in FIG. 8 corresponds to the network of model M1 and is a conceptual diagram showing a neural network (model) used for image recognition. For example, when an image is input from the input layer INL side to the network NW1, the recognition result is output from the output layer OUTL. For example, the data adjustment device 100 inputs information to the input layer INL in the network NW1 to cause the output layer OUTL to output a recognition result corresponding to the input.

[0161] Note that in FIG. 8, the network NW1 is shown as an example of a model (network), but the network NW1 may be in various forms depending on the application and the like. For example, the data adjustment device 100 learns the model M1 by learning the parameters (weights) of the model M1 having the structure of the network NW1 shown in FIG. 8.

[0162] [1-4. Configuration of the terminal device according to the embodiment] Next, the configuration of the terminal device 10, which is an example of a terminal device that executes the information processing according to the embodiment, will be described. FIG. 9 is a diagram showing a configuration example of the terminal device according to the embodiment of the present disclosure.

[0163] As shown in FIG. 9, the terminal device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, a control unit 15, and a sensor unit 16. Note that the terminal device 10 may have any device configuration as long as it can collect data and provide it to the data adjustment device 100. For example, if the terminal device 10 includes a communication unit 11 that communicates with the data adjustment device 100 and a control unit 15 that performs a process of collecting data, the other configurations may be arbitrary. Depending on the type of the terminal device 10, for example, the terminal device 10 may not have any of the input unit 12, the output unit 13, the storage unit 14, and the sensor unit 16.

[0164] For example, when the terminal device 10 is an image sensor (imager), the terminal device 10 may have a configuration including only the communication unit 11, the control unit 15, and the sensor unit 16. For example, the image sensor device used in the image sensor (imager) is a CMOS (Complementary Metal Oxide Semiconductor). Note that the image sensor device used in the image sensor (imager) is not limited to CMOS, and may be various image sensor devices such as a CCD (Charge Coupled Device). Also, for example, when the terminal device 10 is a data server, the terminal device 10 may have a configuration including only the communication unit 11, the storage unit 14, and the control unit 15. Also, for example, when the terminal device 10 is a moving body, the terminal device 10 may have a configuration including a mechanism for realizing movement such as a driving unit (motor).

[0165] The communication unit 11 is realized by, for example, a NIC, a communication circuit, or the like. The communication unit 11 is connected to the network N (such as the Internet) by wire or wirelessly, and transmits and receives information to and from other devices such as the data adjustment device 100 via the network N.

[0166] The input unit 12 receives various inputs. The input unit 12 receives the user's operation. The input unit 12 may receive an operation on the terminal device 10 used by the user (user operation) as an operation input by the user. The input unit 12 may receive information regarding the user's operation using a remote controller (remote controller) via the communication unit 11. Also, the input unit 12 may have buttons provided on the terminal device 10, or a keyboard or a mouse connected to the terminal device 10.

[0167] For example, the input unit 12 may have a touch panel that can realize functions equivalent to those of a remote control, a keyboard, or a mouse. In this case, various types of information are input via the display (output unit 13) to the input unit 12. The input unit 12 receives various operations from the user via the display screen by means of the function of the touch panel realized by various sensors. That is, the input unit 12 receives various operations from the user via the display (output unit 13) of the terminal device 10. For example, the input unit 12 receives the user's operation via the display (output unit 13) of the terminal device 10.

[0168] The output unit 13 outputs various types of information. The output unit 13 has a function of displaying information. The output unit 13 is provided in the terminal device 10 and displays various types of information. The output unit 13 is realized, for example, by a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. The output unit 13 may also have a function of outputting sound. For example, the output unit 13 has a speaker for outputting sound.

[0169] The storage unit 14 is realized, for example, by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores various types of information used for displaying information.

[0170] Returning to FIG. 9 and continuing the explanation. The control unit 15 is realized, for example, by a CPU, an MPU, or the like, when a program (for example, an information processing program such as the data providing program according to the present disclosure) stored inside the terminal device 10 is executed using a RAM or the like as a working area. Further, the control unit 15 is a controller and may be realized by an integrated circuit such as an ASIC or an FPGA.

[0171] As shown in FIG. 9, the control unit 15 has a receiving unit 151, a collecting unit 152, and a transmitting unit 153, and realizes or executes the functions and operations of information processing described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in FIG. 9, and any other configuration may be used as long as it can perform the information processing described later.

[0172] The receiving unit 151 receives various types of information. The receiving unit 151 receives various types of information from an external information processing device. The receiving unit 151 receives various types of information from another information processing device such as the data adjustment device 100.

[0173] The receiving unit 151 receives, from an external device, request information indicating data that the external device having learning data used for learning a model by machine learning requests to acquire. The receiving unit 151 receives, from the data adjustment device 100, request information indicating data that the data adjustment device 100 requests to acquire. The receiving unit 151 receives, from an external device (such as the data adjustment device 100) having a machine learning model, request information for requesting learning data used for the machine learning. The receiving unit 151 receives request information for requesting data similar to learning data whose influence degree on learning in the machine learning model is equal to or higher than a predetermined standard.

[0174] The collection unit 152 collects various types of information. The collection unit 152 determines the collection of various types of information. The collection unit 152 collects various types of information based on information from an external information processing device. The collection unit 152 collects various types of information based on information from the data adjustment device 100. The collection unit 152 collects various types of information in response to an instruction from the data adjustment device 100. The collection unit 152 collects various types of information based on the information stored in the storage unit 14.

[0175] The collection unit 152 collects data corresponding to the request information received by the receiving unit 151. The collection unit 152 collects, as data to be provided to the data adjustment device 100 (data for providing), data corresponding to the request information received by the receiving unit 151. The collection unit 152 collects the data for providing by extracting, from the storage unit 14, data corresponding to the request information received by the receiving unit 151. The collection unit 152 collects the data for providing by detecting, by the sensor unit 16, data corresponding to the request information received by the receiving unit 151.

[0176] The transmission unit 153 transmits various types of information to an external information processing device. For example, the transmission unit 153 transmits various types of information to other information processing devices such as the data adjustment device 100. The transmission unit 153 transmits the information stored in the storage unit 14.

[0177] The transmission unit 153 transmits various types of information based on information from other information processing devices such as the data adjustment device 100. The transmission unit 153 transmits various types of information based on the information stored in the storage unit 14.

[0178] The transmission unit 153 transmits the provided data collected as data corresponding to the request information to an external device. The transmission unit 153 transmits the provided data collected as data corresponding to the request information to the data adjustment device 100. The transmission unit 153 transmits the provided data collected by the collection unit 152 to the data adjustment device 100.

[0179] For example, when the terminal device 10 has the sensor unit 16, the transmission unit 153 transmits the sensor information detected by the sensor unit 16 to the data adjustment device 100. The transmission unit 153 transmits the image information detected by the image sensor of the sensor unit 16 to the data adjustment device 100.

[0180] The sensor unit 16 detects various types of sensor information. The sensor unit 16 has a function as an imaging unit that captures images. The sensor unit 16 has the function of an image sensor and detects image information. The sensor unit 16 functions as an image input unit that receives an image as an input.

[0181] Note that the sensor unit 16 is not limited to the above and may have various sensors. The sensor unit 16 may have various sensors such as a sound sensor, a position sensor, an acceleration sensor, a gyro sensor, a temperature sensor, a humidity sensor, an illuminance sensor, a pressure sensor, a proximity sensor, and sensors for receiving biological information such as odor, sweat, heartbeat, pulse, and brain waves. Also, the sensors for detecting the above various types of information in the sensor unit 16 may be a common sensor or may be realized by different sensors respectively.

[0182] [1-5. Procedure of Information Processing According to Embodiment] Next, with reference to FIGS. 10 and 11, the procedures of various information processes according to the embodiment will be described.

[0183] [1-5-1. Procedure of Process Related to Data Adjustment Device] First, with reference to FIG. 10, the flow of the process related to the data adjustment device according to the embodiment of the present disclosure will be described. FIG. 10 is a flowchart showing the process of the data adjustment device according to the embodiment of the present disclosure. Specifically, FIG. 10 is a flowchart showing the procedure of information processing by the data adjustment device 100.

[0184] As shown in FIG. 10, the data adjustment device 100 measures the contribution degree given by each data included in the data set used for learning the model by machine learning (step S101). Then, based on the measurement result, the data adjustment device 100 adjusts the data set by excluding data from the data set or adding new data to the data set (step S102).

[0185] [1-5-2. Procedure of Process Related to Data Adjustment System] Next, with reference to FIG. 11, an example of a specific process related to the data adjustment system will be described. FIG. 11 is a sequence diagram showing the processing procedure of the data adjustment system according to the embodiment of the present disclosure.

[0186] As shown in FIG. 11, the data adjustment device 100 measures the contribution degree of each data in learning (step S201). For example, the data adjustment device 100 measures the contribution degree given by the learning data used for learning the model by machine learning.

[0187] The data adjustment device 100 excludes data with a low contribution degree (step S202). The data adjustment device 100 excludes data from the data set whose contribution degree in learning is below the threshold for determining a low contribution degree.

[0188] The data adjustment device 100 adds data corresponding to data with a high contribution degree (step S203). The data adjustment device 100 adds data corresponding to data whose contribution degree in learning is equal to or higher than the discrimination threshold for a high contribution degree.

[0189] In the example of FIG. 8, the data adjustment device 100 requests the terminal device 10 for data corresponding to data with a high contribution degree (step S204). For example, the data adjustment device 100 requests the terminal device 10 for data similar to the data with a high contribution degree.

[0190] The terminal device 10 for which data is requested collects the data corresponding to the request (step S205). Then, the terminal device 10 transmits the collected data to the data adjustment device 100 (step S206).

[0191] The data adjustment device 100 that has acquired data from the terminal device 10 adds the acquired data to the data set (step S207).

[0192] [1-6. Example of data adjustment based on influence degree] Here, an example of data adjustment based on the influence degree will be described after explaining the premise. Knowing the influence degree of data on a deep neural network in machine learning also leads to the improvement of the network. Specifically, increasing the data that gives a good influence degree is useful for improving the characteristics in machine learning. As a method of increasing those data, similar images can be increased by data augmentation (for example, rotating an image to increase similar images) as a method of data padding. Also, data similar to the data that gives a good influence can be found from the data on the network, and the data can be enhanced. By adding those data and re-learning the deep neural network, a more accurate deep neural network can be constructed. This point will be described with reference to FIG. 12. FIG. 12 is a flowchart showing an example of the process of data adjustment and learning based on the influence degree.

[0193] As shown in FIG. 12, the data adjustment device 100 calculates the influence degree of the neural network (step S301). For example, the data adjustment device 100 measures the influence degree that the learning data used for the learning of the neural network has on the learning.

[0194] Then, the data adjustment device 100 extracts data with a high degree of good influence (step S302). For example, the direction of reducing the loss is a good influence, the direction of increasing the loss is a bad influence, and the greater the degree in the direction of reducing the loss, the greater the degree of good influence. For example, the data adjustment device 100 extracts data whose degree of good influence is equal to or higher than a predetermined standard (threshold value, etc.).

[0195] Then, the data adjustment device 100 adds data (step S303). The data adjustment device 100 adds data similar to the data with a high degree of good influence to the learning data. For example, the data adjustment device 100 may generate data similar to the data with a high degree of good influence by data augmentation and add the generated data to the learning data. Also, for example, the data adjustment device 100 may add data similar to the data with a high degree of good influence among the data on the network to the learning data.

[0196] Then, the data adjustment device 100 adds data and performs re-learning (step S304). For example, the data adjustment device 100 performs re-learning of the model using the learning data to which data was added in step S303.

[0197] Then, the data adjustment device 100 updates the re-learned model (step S305). For example, the data adjustment device 100 updates the model before re-learning to the model after re-learning. For example, the data adjustment device 100 updates the parameters of the model to the parameters after re-learning.

[0198] [1-6-1. Specific Example of Adjustment] As a specific example of the processing in FIG. 12 described above, a case where the data adjustment device 100 is a camera equipped with an image classification function using a deep neural network will be described as an example. In this case, first, the data adjustment device 100 calculates data that has a positive influence. Then, the data adjustment device 100 generates and collects data similar to that data by data augmentation, or collects it from the network. The data adjustment device 100 adds the collected data and retrains the original deep neural network. As a result, the data adjustment device 100 can improve the accuracy of the image classification function of the camera.

[0199] Note that in data augmentation, a system (data adjustment system 1) that autonomously searches for data that has a positive influence may be configured. Thereby, the data adjustment system 1 can search for data without human intervention and automatically perform retraining. In this case, the data adjustment system 1 becomes a learning system that autonomously evolves the deep neural network. With this data adjustment system 1, the deep neural network can evolve its performance by itself.

[0200] [2. Other Embodiments] The processing according to each of the above-described embodiments may be implemented in various different forms (modification examples) other than the above-described embodiments and modification examples.

[0201] [2-1. Other Configuration Examples] Note that, in the above example, the case where the data adjustment device 100 and the terminal device 10 are separate entities is shown, but these devices may be integrated. For example, the data adjustment device 100 may be a device having a function of adjusting learning data and a function of collecting data. For example, the data adjustment device 100 may be an information processing device that acquires new learning data based on the degree of influence. In this case, the data adjustment device 100 includes a learned model using machine learning, a measurement unit that measures the degree of influence of the learning data used in the machine learning on the machine learning, and a control unit (acquisition unit, etc.) that acquires new learning data based on the degree of influence. The data adjustment device 100 may be a camera, a smartphone, a television, an automobile, a drone, a robot, or the like. Thus, the data adjustment device 100 may be a terminal device that autonomously collects learning data with a high degree of influence.

[0202] [2-2. Others] Also, among the respective processes described in the above embodiments, all or part of the processes described as being automatically performed can be manually performed, or all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, regarding the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings, they can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0203] Also, each component of each device shown in the drawings is conceptually functional and does not necessarily have to be physically configured as shown. That is, the specific form of the distribution and integration of each device is not limited to that shown, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads and usage situations.

[0204] Also, the above-described embodiments and modifications can be appropriately combined within a range that does not conflict with the processing content.

[0205] Also, the effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0206] [3. Effects according to the present disclosure] As described above, the data adjustment system according to the present disclosure (the data adjustment system 1 in the embodiment) includes an information processing apparatus (the data adjustment apparatus 100 in the embodiment) including a measurement unit and an adjustment unit, and a terminal apparatus (the terminal apparatus 10 in the embodiment). The measurement unit measures the degree of influence that the learning data used for the learning of the neural network has on the learning. The adjustment unit excludes the data measured to have a low degree of influence, and acquires new data, which is new data corresponding to the data measured to have a high degree of influence, from the terminal apparatus or the database, and adjusts the learning data by adding the acquired new data.

[0207] Thus, the data adjustment system according to the present disclosure uses the degree of influence that the learning data has on the learning to exclude data or add data. Thereby, the data adjustment system can adjust the data used for learning by adjusting the learning data by increasing or decreasing the data according to the degree of influence of each data.

[0208] As described above, the data adjustment apparatus according to the present disclosure (the data adjustment apparatus 100 in the embodiment) includes a measurement unit (the measurement unit 133 in the embodiment) and an adjustment unit (the adjustment unit 134 in the embodiment). The measurement unit measures the degree of influence that each data included in the learning data set used for the learning of the model by machine learning has on the learning. The adjustment unit adjusts the learning data by excluding predetermined data from the learning data or adding new data to the learning data based on the measurement result by the measurement unit.

[0209] In this way, the data adjustment device according to the present disclosure excludes or adds data by using the degree of influence that the learning data has on learning. Thereby, the data adjustment system can adjust the data used for learning by increasing or decreasing the data according to the degree of influence of each data, making it possible to adjust the data used for learning.

[0210] Further, the measurement unit measures the degree of influence based on the loss function. In this way, the data adjustment device can accurately measure the degree of influence of each data by measuring the degree of influence based on the loss function. Therefore, the data adjustment device can make it possible to adjust the data used for learning.

[0211] Further, the measurement unit measures the degree of influence by a method available for measuring the degree of influence. In this way, the data adjustment device can accurately measure the degree of influence of each data by measuring the degree of influence by a method available for measuring the degree of influence. Therefore, the data adjustment device can make it possible to adjust the data used for learning.

[0212] Further, the measurement unit measures the degree of influence by the Influence function. In this way, the data adjustment device can accurately measure the degree of influence of each data by measuring the degree of influence by the Influence function. Therefore, the data adjustment device can make it possible to adjust the data used for learning.

[0213] Further, the measurement unit measures the degree of influence of predetermined data based on the difference between the case of the learning data and the case where predetermined data is excluded from the learning data. In this way, the data adjustment device can accurately measure the degree of influence of that data by measuring the degree of influence based on the difference between the case where a certain data is excluded from the learning data and the case where it is not excluded. Therefore, the data adjustment device can make it possible to adjust the data used for learning.

[0214] In addition, the adjustment unit excludes the first data with a low influence degree from the learning data. In this way, the data adjustment device can appropriately remove the data that does not contribute to learning from the learning data by excluding the first data with a low influence degree from the learning data. Therefore, the data adjustment device can make the data used for learning adjustable.

[0215] In addition, the adjustment unit excludes the first data with an influence degree lower than the first threshold value from the learning data. In this way, the data adjustment device can appropriately remove the data that does not contribute to learning from the learning data by excluding the first data with an influence degree lower than the first threshold value from the learning data. Therefore, the data adjustment device can make the data used for learning adjustable.

[0216] In addition, the adjustment unit adds new data, which is new data corresponding to the second data with a high influence degree, to the learning data. In this way, the data adjustment device can appropriately add the data that contributes to learning to the learning data by adding the new data corresponding to the second data with a high influence degree to the learning data. Therefore, the data adjustment device can make the data used for learning adjustable.

[0217] In addition, the adjustment unit adds new data, which is new data corresponding to the second data with an influence degree higher than the second threshold value, to the learning data. In this way, the data adjustment device can appropriately add the data that contributes to learning to the learning data by adding the new data corresponding to the second data with an influence degree higher than the second threshold value to the learning data. Therefore, the data adjustment device can make the data used for learning adjustable.

[0218] In addition, the data adjustment device according to the present disclosure includes a transmission unit (in the embodiment, the transmission unit 135). The transmission unit transmits request information for requesting new data to an external device (in the embodiment, a terminal device 10 such as a data server, a camera, an image sensor, a moving body, etc.). The adjustment unit adds the new data acquired from the external device to the learning data. In this way, the data adjustment device can appropriately add data contributing to learning to the learning data by requesting new data from the external device and adding the new data acquired from the external device to the learning data. Therefore, the data adjustment device can adjust the data used for learning.

[0219] In addition, the adjustment unit adds the new data acquired from the storage unit that stores the data to the learning data. In this way, the data adjustment device can appropriately add data contributing to learning to the learning data by acquiring new data from the storage unit that stores the data and adding the acquired new data to the learning data. Therefore, the data adjustment device can adjust the data used for learning.

[0220] In addition, the adjustment unit generates new data and adds the generated new data to the learning data. In this way, the data adjustment device can appropriately add data contributing to learning to the learning data by generating new data and adding the generated new data to the learning data. Therefore, the data adjustment device can adjust the data used for learning.

[0221] In addition, the adjustment unit generates new data by data augmentation and adds the generated new data to the learning data. In this way, the data adjustment device can generate new data with a high contribution degree like the second data with a high contribution degree by data augmentation and add it to the learning data. Therefore, the data adjustment device can adjust the data used for learning.

[0222] In addition, the adjustment unit generates new data using the second data and adds the generated new data to the learning data. In this way, the data adjustment device can generate new data with a high contribution rate, such as the second data with a high contribution rate, by generating new data using the second data, and add it to the learning data. Therefore, the data adjustment device can adjust the data used for learning.

[0223] In addition, the adjustment unit generates new data similar to the second data and adds the generated new data to the learning data. In this way, the data adjustment device can generate new data similar to the second data with a high contribution rate by generating new data similar to the second data, and add it to the learning data. Therefore, the data adjustment device can adjust the data used for learning.

[0224] In addition, the measurement unit measures the influence degree of the learning data used for the learning of the neural network. In this way, the data adjustment device excludes data from the learning data used for the learning of the neural network or adds data to the learning data. Thereby, the data adjustment system can adjust the data used for the learning of the neural network by increasing or decreasing the data of the learning data according to the influence degree of each data, and adjusting the learning data.

[0225] In addition, the data adjustment device according to the present disclosure includes a learning unit (learning unit 132 in the embodiment). The learning unit executes a learning process using the learning data adjusted by the adjustment unit. In this way, the data adjustment device can perform learning using the learning data capable of learning an accurate model by executing a learning process using the adjusted learning data. The data adjustment device can repeat the adjustment process of the learning data and the learning process using the adjusted learning data, and thus can learn the model using the learning data capable of learning a more accurate model.

[0226] As described above, the terminal device according to the present disclosure (in the embodiment, the terminal device 10 such as a data server, a camera, an image sensor, a moving body, etc.) includes a receiving unit (in the embodiment, the receiving unit 151) and a transmitting unit (in the embodiment, the transmitting unit 153). The receiving unit receives request information for requesting learning data used for the machine learning from an external device (in the embodiment, the data adjustment device 100) having a machine learning model. The transmitting unit transmits the data collected as the data corresponding to the request information to the external device.

[0227] As described above, the terminal device according to the present disclosure provides the external device with the data corresponding to the request in response to the request from the external device having the learning data used for the learning of the model by machine learning. Thereby, the external device having the learning data can adjust the learning data by adding the data acquired from the terminal device to the learning data. Therefore, the terminal device can make the data used for learning adjustable.

[0228] In addition, the learning data requested by the request information according to the present disclosure is data similar to the learning data whose influence degree on the learning in the machine learning model is equal to or higher than a predetermined standard. By thus requesting data similar to the learning data whose influence degree is equal to or higher than a predetermined standard, useful data for learning is collected, and the learning process is executed using the data, so that learning can be performed using the learning data with which an accurate model can be learned.

[0229] As described above, the information processing device according to the present disclosure (in the embodiment, the data adjustment device 100) includes a learned model using machine learning, a measurement unit that measures the influence degree of the learning data used for the machine learning on the machine learning, and a control unit that acquires new learning data based on the influence degree.

[0230] As described above, the information processing device according to the present disclosure can collect useful data for learning and efficiently adjust the learning data by acquiring new learning data based on the influence degree of learning. Therefore, the information processing device can make the data used for learning adjustable.

[0231] [4. Hardware Configuration] Information devices such as the data adjusting device 100 and terminal device 10 according to the above-described embodiments and modifications are realized by a computer 1000 having a configuration as shown in FIG. 13, for example. FIG. 13 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of an information processing device such as the data adjusting device 100 and terminal device 10. The following description will be given taking the data adjusting device 100 according to the embodiment as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0232] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0233] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0234] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0235] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (such as the Internet). For example, the CPU 1100 receives data from other devices or transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0236] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. Also, the CPU 1100 transmits data to output devices such as a display, a speaker, and a printer via the input / output interface 1600. Further, the input / output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium (media). The media is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory or the like.

[0237] For example, when the computer 1000 functions as the data adjustment device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes functions such as the control unit 130 by executing an information processing program loaded on the RAM 1200. Also, the information processing program according to the present disclosure and the data in the storage unit 120 are stored in the HDD 1400. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be acquired from other devices via the external network 1550.

[0238] Note that the present technology can also have the following configuration. (1) A measurement unit that measures the degree of influence exerted by the learning data used for neural network learning on the learning; An adjustment unit that excludes the data measured to have a low degree of influence, acquires new data that is new data corresponding to the data measured to have a high degree of influence, and adjusts the learning data by adding the acquired new data; A data adjustment system having the above. (2) A measurement unit that measures the degree of influence exerted by the learning data used for learning of a model by machine learning on the learning; An adjustment unit that adjusts the learning data by excluding data from the learning data or adding new data to the learning data based on the measurement result by the measurement unit; A data adjustment device including the above. (3) The measurement unit measures the degree of influence based on a loss function The data adjustment device according to (2). (4) The measurement unit measures the degree of influence by a method available for degree of influence measurement The data adjustment device according to (2) or (3). (5) The measurement unit measures the degree of influence by Influence function The data adjustment device according to (4). (6) The measurement unit measures the degree of influence of the predetermined data based on the difference between the case of the learning data and the case where predetermined data is excluded from the learning data The data adjustment device according to any one of (2) to (5). (7) The adjustment unit excludes the first data having a low degree of influence from the learning data The data adjustment device according to any one of (2) to (6). (8) The adjustment unit Exclude the first data whose influence degree is lower than the first threshold value from the learning data. The data adjustment device according to (7). (9) The adjustment unit Add new data, which is new data corresponding to the second data with a high influence degree, to the learning data. The data adjustment device according to any one of (2) to (8). (10) The adjustment unit Add the new data corresponding to the second data whose influence degree is higher than the second threshold value to the learning data. The data adjustment device according to (9). (11) A transmission unit that transmits request information for requesting the new data to an external device, further comprising The adjustment unit Add the new data acquired from the external device to the learning data. The data adjustment device according to (9) or (10). (12) The adjustment unit Add the new data acquired from the storage unit that stores data to the learning data. The data adjustment device according to any one of (9) to (11). (13) The adjustment unit Generate the new data and add the generated new data to the learning data. The data adjustment device according to any one of (9) to (12). (14) The adjustment unit Generate the new data using the second data and add the generated new data to the learning data. The data adjustment device according to (13). (15) The adjustment unit Generate the new data by data augmentation and add the generated new data to the learning data. The data adjustment device according to (13) or (14). (16) The adjustment unit generates the new data similar to the second data and adds the generated new data to the learning data The data adjustment device according to any one of (13) to (15). (17) The measurement unit measures the influence degree of the learning data used for the learning of the neural network The data adjustment device according to any one of (2) to (16). (18) A learning unit that executes a learning process using the learning data after adjustment by the adjustment unit The data adjustment device according to any one of (2) to (17), further comprising (19) measures the influence degree that the learning data used for the learning of the model by machine learning has on the learning, and based on the measurement result, adjusts the learning data by excluding data from the learning data or adding new data to the learning data A data adjustment method for executing a process (20) A receiving unit that receives request information indicating data requested to be acquired by an external device having learning data used for learning of a model by machine learning from the external device, A transmitting unit that transmits the provided data collected as data corresponding to the request information to the external device A terminal device comprising (21) The learning data requested by the request information is data similar to the learning data having an influence degree on the learning in the machine learning model equal to or higher than a predetermined standard. The terminal device according to (20). (22) In an information processing apparatus A learned model using machine learning a measurement unit that measures the influence of learning data used in the machine learning on the machine learning; a control unit that acquires new learning data based on the influence degree; An information processing device comprising: [Explanation of symbols]

[0239] 1 Data Coordination System 100 Data adjustment device (information processing device) 110 Communications Department 120 Storage section 121 Data information storage unit 122 Model information storage unit 123 Threshold information storage unit 130 control section 131 Acquisition Department 132 Learning Department 133 Measuring section 134 Adjustment section 135 Transmitter 10 Terminal equipment (data servers, cameras, image sensors, mobile devices) 11 Communications Department 12 Input section 13 Output section 14 Storage section 15 Control Unit 151 Receiving unit 152 Collection Department 153 Transmitter 16 Sensor section

Claims

1. An information processing apparatus and a terminal device having a sensor unit, wherein the information processing apparatus includes: a measurement unit configured to measure the degree of influence that learning data used for learning of a neural network has on the learning; an adjustment unit configured to adjust the learning data by excluding data measured to have a low degree of influence or by acquiring, from the terminal device, new data that is new data corresponding to data measured to have a high degree of influence and adding the acquired new data; a transmission unit configured to transmit, to the terminal device, request information for requesting data that is the new data to be used as the learning data; an acquisition unit configured to acquire, from the terminal device, data detected by the sensor unit of the terminal device in response to the request information received by the terminal device; A data adjustment system comprising the above.

2. The measurement unit measures the degree of influence based on a loss function. The data adjustment system according to Claim 1.

3. The measurement unit measures the degree of influence by a method available for influence degree measurement. The data adjustment system according to Claim 1.

4. The measurement unit measures the degree of influence by an Influence function. The data adjustment system according to Claim 3.

5. The measurement unit measures the degree of influence of the predetermined data based on the difference between the case of the learning data and the case where predetermined data is excluded from the learning data. The data adjustment system according to Claim 1.

6. The adjustment unit excludes first data having a low degree of influence from the learning data. The data adjustment system according to Claim 1.

7. The adjustment unit excludes the first data having a degree of influence lower than a first threshold value from the learning data. The data adjustment system according to Claim 6.

8. The adjustment unit adds, to the learning data, new data that is new data corresponding to second data having a high degree of influence. The data adjustment system according to Claim 1.

9. The adjustment unit adds, to the learning data, the new data corresponding to the second data having a degree of influence higher than a second threshold value. The data adjustment system according to Claim 8.

10. Further comprising a transmission unit configured to transmit request information for requesting the new data to an external device, wherein the adjustment unit adds the new data acquired from the external device to the learning data. The data adjustment system according to Claim 8. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

11. The adjustment unit adds the new data obtained from the storage unit that stores data to the learning data. The data adjustment system according to claim 8.

12. The adjustment unit generates the new data and adds the generated new data to the learning data. The data adjustment system according to claim 8.

13. The adjustment unit generates the new data using the second data and adds the generated new data to the learning data. The data adjustment system according to claim 12.

14. The adjustment unit generates the new data by data augmentation and adds the generated new data to the learning data. The data adjustment system according to claim 12.

15. The adjustment unit generates the new data similar to the second data and adds the generated new data to the learning data. The data adjustment system according to claim 12.

16. The measurement unit measures the influence degree of the data included in the learning data used for learning the neural network. The data adjustment system according to claim 1.

17. A learning unit that executes a learning process using the learning data after adjustment by the adjustment unit, The data adjustment system according to claim 1, further comprising.

18. A data adjustment method executed by a data adjustment system including an information processing device and a terminal device having a sensor unit, wherein the information processing device measures the influence degree that the data included in the learning data used for learning the model by machine learning has on the learning, adjusts the learning data by excluding data from the learning data or adding new data to the learning data based on the measurement result, transmits request information for requesting data to be used as the learning data to the terminal device, the terminal device that has received the request information detects data corresponding to the request information by the sensor unit, The information processing device acquires the data detected by the terminal device from the terminal device. A data adjustment method for executing the process.

Images