Machine learning system, edge device, and information processing apparatus

Abstract

To efficiently select input data effective for learning.SOLUTION: A machine learning system comprises a first information processing apparatus and a second information processing apparatus. The first information processing apparatus has a first evaluation unit, a first selection unit, and a candidate data transmission unit. The first evaluation unit calculates a first evaluation value for each of a plurality of pieces of candidate data on the basis of a first evaluation criterion. The first selection unit selects whether or not to include each of a plurality of pieces of input data in the plurality of pieces of candidate data, in accordance with the first evaluation value. The candidate data transmission unit transmits the candidate data. The second information processing apparatus has a candidate data receiving unit, a second evaluation unit, and a second selection unit. The candidate data receiving unit receives the candidate data. The second evaluation unit calculates a second evaluation value for each of the plurality of pieces of candidate data on the basis of a second evaluation criterion different from the first evaluation criterion. The second selection unit selects whether or not to include each of the plurality of pieces of candidate data in a plurality of pieces of learning data, in accordance with the second evaluation value.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] Embodiments of the present invention relate to a machine learning system, an edge device, and an information processing apparatus.

Background Art

[0002] There is known a machine learning system that deploys a machine learning model learned in the cloud to an edge device equipped with, for example, a surveillance camera, and executes inference processing based on the machine learning model in the edge device. In such a system, since input data such as an image captured by a surveillance camera does not have to be transmitted to the cloud, the communication load is reduced.

[0003] Also, in such a machine learning system, a machine learning operation is known in which input data collected in an edge device is transmitted to the cloud and the machine learning model is re-learned in the cloud. By applying such a machine learning operation, the machine learning system can use a machine learning model adapted to the on-site environment.

[0004] By the way, in order to efficiently re-learn the machine learning model, the machine learning system has to select input data effective for re-learning from a large amount of input data.

[0005] However, since the resources for information processing of the edge device are limited, it has been difficult to execute a selection process with a large amount of computation. Also, since the cloud has a high information processing ability, it is possible to execute a selection process with a large amount of computation. However, when the machine learning system transmits a large amount of input data collected in the edge device to the cloud, the communication load becomes extremely large.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

SUMMARY OF THE INVENTION

PROBLEM TO BE SOLVED BY THE INVENTION

[0007] The problem to be solved by the present invention is to provide a machine learning system, an edge device, and an information processing apparatus that can efficiently select input data effective for learning from a plurality of input data.

MEANS FOR SOLVING THE PROBLEM

[0008] The machine learning system according to the embodiment selects a plurality of learning data for training a first machine learning model from a plurality of input data. The machine learning system includes a first information processing apparatus and a second information processing apparatus connected to the first information processing apparatus via a network. The first information processing apparatus includes a first evaluation unit, a first selection unit, and a candidate data transmission unit. The first evaluation unit calculates a first evaluation value representing the effectiveness when used for training the first machine learning model in each of the plurality of input data based on a predetermined first evaluation criterion. The first selection unit selects whether to include each of the plurality of input data in a plurality of candidate data by comparing each of the first evaluation values of the plurality of input data with a predetermined value. The candidate data transmission unit transmits each of the plurality of candidate data to the second information processing apparatus via the network. The second information processing apparatus includes a candidate data reception unit, a second evaluation unit, and a second selection unit. The candidate data reception unit receives each of the plurality of candidate data from the first information processing apparatus via the network. The second evaluation unit calculates a second evaluation value indicating the effectiveness when used for training the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion. The second selection unit selects whether to include each of the plurality of candidate data in the plurality of learning data by comparing each of the second evaluation values of the plurality of candidate data with a predetermined value.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] FIG. 1 is a diagram showing the configuration of a machine learning system 10 according to the embodiment.

[0012] The machine learning system 10 acquires a plurality of input data in time series, and executes an inference process for each of the acquired plurality of input data using a preset first machine learning model. Further, the machine learning system 10 selects a plurality of learning data for training the first machine learning model from among the acquired plurality of input data. Then, the machine learning system 10 retrains the first machine learning model using the plurality of learning data obtained by the selection.

[0013] The machine learning system 10 includes an edge device 21 and a cloud device 22.

[0014] The edge device 21 is an example of a first information processing device. The edge device 21 collects a plurality of input data in time series from the surrounding environment, and executes information processing for each of the collected plurality of input data. The edge device 21 transmits the result of the information processing for each of the plurality of input data to the cloud device 22 via a network.

[0015] The cloud device 22 is an example of a second information processing device. The cloud device 22 is connected to the edge device 21 via a network. The cloud device 22 acquires the result of the information processing for each of the plurality of input data from the edge device 21 via the network. The cloud device 22 outputs the result of the information processing for each of the plurality of input data to a user or the like, or executes further information processing on the result of the information processing.

[0016] The edge device 21 includes a first arithmetic processing device 25 of hardware. The first arithmetic processing device 25 is a processing circuit constituted by, for example, one or a plurality of CPUs (Central Processing Unit) or the like. The first arithmetic processing device 25 executes the information processing performed in the edge device 21.

[0017] The cloud device 22 is an information processing device such as a server device. The cloud device 22 may be, for example, a server device in which a plurality of information processing devices operate in cooperation. The cloud device 22 includes a second arithmetic processing device 26 of hardware. The second arithmetic processing device 26 is a processing circuit composed of, for example, one or more CPUs or the like. The second arithmetic processing device 26 executes information processing performed in the cloud device 22.

[0018] The second arithmetic processing device 26 executes information processing with higher arithmetic precision than the first arithmetic processing device 25. For example, when the first arithmetic processing device 25 executes a floating-point arithmetic operation with 32-bit precision, the second arithmetic processing device 26 executes a floating-point arithmetic operation with 64-bit precision, and so on. Thereby, when the second arithmetic processing device 26 executes the same arithmetic processing as the first arithmetic processing device 25, an arithmetic result different from that of the first arithmetic processing device 25 can be obtained.

[0019] FIG. 2 is a block diagram showing the functional configurations of the edge device 21 and the cloud device 22.

[0020] The edge device 21 includes an input data generation unit 31, an inference unit 32, and a result transmission unit 33. The cloud device 22 includes a result reception unit 34 and an output unit 35.

[0021] The input data generation unit 31 collects the observation results of observing the surroundings of the edge device 21. The input data generation unit 31 generates a plurality of input data arranged in time series based on the collected observation results. The input data generation unit 31 may be, for example, an imaging device. In this case, the input data generation unit 31 captures the surroundings every predetermined time and generates image data representing the captured surroundings as input data. Also, the input data generation unit 31 may be a microphone device. In this case, the input data generation unit 31 picks up the surrounding sound and generates voice data in a predetermined time length unit as input data. Also, the input data generation unit 31 may be one or a plurality of sensor devices. In this case, the input data generation unit 31 senses the temperature or humidity of the object every predetermined time and generates sensor data as input data.

[0022] The inference unit 32 sequentially acquires each of the plurality of input data in time series. The inference unit 32 performs an inference process on each of the plurality of input data in time series based on the first machine learning model, and outputs the inference result obtained by the inference process. For example, the inference unit 32 acquires each of the plurality of input data in time series, and performs an inference process every time each of the plurality of input data is acquired, and outputs the inference result obtained by the inference process.

[0023] The first machine learning model is, for example, a neural network whose parameters have been learned in advance. When the first machine learning model is a neural network, the inference unit 32 gives the acquired input data to the neural network and acquires the inference result output from the neural network.

[0024] The result transmission unit 33 transmits the inference result output from the inference unit 32 to the cloud device 22 via the network. The result transmission unit 33 may transmit the inference result to the cloud device 22 only when a predetermined inference result is output from the inference unit 32. For example, the result transmission unit 33 may transmit the inference result to the cloud device 22 only when the person included in the image data is identified as a predetermined person.

[0025] The result receiving unit 34 receives the inference result from the edge device 21 via the network.

[0026] When the inference result is received by the result receiving unit 34, the output unit 35 causes the received inference result to be output from, for example, a terminal device held by the user, or causes the received inference result to be displayed on a display device. Further, the output unit 35 may further transmit the received inference result to another information processing device.

[0027] The machine learning system 10 as described above can generate a plurality of input data in time series, and execute an inference process using a preset first machine learning model for each of the plurality of generated input data.

[0028] Furthermore, the edge device 21 includes a first evaluation unit 41, a first selection unit 42, and a candidate data transmission unit 43. Further, the cloud device 22 includes a candidate data reception unit 44, a second evaluation unit 45, a second selection unit 46, a storage unit 47, and a learning unit 48.

[0029] The first evaluation unit 41 acquires each of the plurality of input data generated by the input data generation unit 31. The first evaluation unit 41 calculates a first evaluation value representing the effectiveness when used for learning the first machine learning model in each of the plurality of input data based on a preset first evaluation criterion. For example, the first evaluation unit 41 acquires each of the plurality of input data in time series in the generation order, and calculates the first evaluation value every time each of the plurality of input data is acquired.

[0030] The first evaluation value may be a value representing the effectiveness when used for learning the first machine learning model, for example, by a real number. Further, the first evaluation value may be a binary value indicating whether it is effective or not.

[0031] In this embodiment, the first evaluation unit 41 calculates a first evaluation value for any first input data among a plurality of input data based on the relationship between the first input data and data different from the first input data. An example of the method for calculating the first evaluation value by the first evaluation unit 41 will be described later in detail with reference to FIG. 4.

[0032] The first selection unit 42 acquires a plurality of input data generated by the input data generation unit 31. Further, the first selection unit 42 acquires the first evaluation value for the acquired input data from the first evaluation unit 41.

[0033] The first selection unit 42 selects whether to include each of the plurality of input data in a plurality of candidate data by comparing the first evaluation value of each of the plurality of input data with a predetermined value. Each of the plurality of candidate data is data that is a candidate for a plurality of learning data for training the first machine learning model.

[0034] For example, when the first evaluation value is represented by a real number, the first selection unit 42 selects, as candidate data, the input data among the plurality of input data whose first evaluation value is greater than or equal to a predetermined value or less than or equal to a predetermined value. Also, for example, when the first evaluation value is represented by a binary value, the first selection unit 42 selects, as candidate data, the input data among the plurality of input data whose first evaluation value indicates validity.

[0035] Also, for example, the first selection unit 42 may acquire each of the plurality of input data in time series. In this case, every time the first selection unit 42 acquires each of the plurality of input data, it determines whether to select the acquired input data as candidate data.

[0036] The candidate data transmission unit 43 transmits each of the plurality of candidate data selected by the first selection unit 42 to the cloud device 22 via the network. For example, the candidate data transmission unit 43 acquires each of the plurality of candidate data from the first selection unit 42 in time series. Then, every time the candidate data transmission unit 43 acquires each of the plurality of candidate data, the acquired candidate data is transmitted to the cloud device 22 via the network.

[0037] The candidate data reception unit 44 receives each of the plurality of candidate data from the edge device 21 via the network. For example, the candidate data reception unit 44 receives each of the plurality of candidate data from the edge device 21 in time series.

[0038] The second evaluation unit 45 acquires each of the plurality of candidate data received by the candidate data reception unit 44. The second evaluation unit 45 calculates a second evaluation value representing the effectiveness when used for training the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion. For example, the second evaluation unit 45 acquires each of the plurality of candidate data in time series in the order of reception, and calculates the second evaluation value every time each of the plurality of candidate data is acquired.

[0039] The second evaluation value may be a value representing the effectiveness when used for training the first machine learning model, for example, represented by a real number. Also, the second evaluation value may be a binary value indicating whether it is effective or not.

[0040] In this embodiment, the second evaluation unit 45 calculates a second evaluation value for any first candidate data among a plurality of candidate data by analyzing an inference result or an intermediate result obtained by inputting the first candidate data into any machine learning model. More specifically, the second evaluation unit 45 calculates a second evaluation value that increases or decreases as the uncertainty of the inference result obtained by inputting the first candidate data into the first machine learning model increases, from the inference result or intermediate result obtained by inputting the first candidate data into any machine learning model. An example of the method for calculating the second evaluation value by the second evaluation unit 45 will be described later with reference to FIGS. 5 to 10 for details.

[0041] Note that when the candidate data transmission unit 43 of the edge device 21 calculates the second evaluation value using at least one of the inference result and the intermediate result obtained by inputting the first candidate data into the first machine learning model, which is calculated by the inference unit 32 of the edge device 21 in the second evaluation unit 45, the candidate data transmission unit 43 transmits at least one of the inference result and the intermediate result together with the first candidate data via the network. Also, in this case, the candidate data reception unit 44 receives at least one of the inference result and the intermediate result together with the first candidate data from the edge device 21. Then, the second evaluation unit 45 calculates the second evaluation value using the inference result and the intermediate result received by the candidate data reception unit 44.

[0042] The second selection unit 46 acquires a plurality of candidate data received by the candidate data reception unit 44. Further, the second selection unit 46 acquires the second evaluation value for the received candidate data from the second evaluation unit 45.

[0043] The second selection unit 46 selects whether to include each of the plurality of candidate data in the plurality of learning data by comparing the second evaluation value of each of the plurality of candidate data received by the candidate data reception unit 44 with a predetermined value. Each of the plurality of learning data is data that serves as a teacher for re-training the first machine learning model.

[0044] For example, when the second evaluation value is represented by a real number, the second selection unit 46 selects, as learning data, candidate data among the plurality of candidate data whose second evaluation value is equal to or greater than a predetermined value or equal to or less than a predetermined value. Further, for example, when the second evaluation value is represented by a binary value, the second selection unit 46 selects, as learning data, candidate data among the plurality of candidate data whose second evaluation value indicates a valid value.

[0045] Also, for example, the second selection unit 46 may acquire each of the plurality of candidate data in time series in the order of reception. In this case, every time the second selection unit 46 acquires each of the plurality of candidate data, it determines whether to select the acquired candidate data as learning data.

[0046] The storage unit 47 stores the plurality of learning data selected by the second selection unit 46.

[0047] The learning unit 48 trains the first machine learning model using the plurality of learning data stored in the storage unit 47, for example, when it receives a predetermined timing or a predetermined instruction. After training the first machine learning model, the learning unit 48 transmits the parameters set in the first machine learning model to the edge device 21 to update the parameters of the first machine learning model used by the edge device 21 for inference processing.

[0048] Such a machine learning system 10 can select a plurality of learning data for training the first machine learning model from among the plurality of acquired input data. Then, the machine learning system 10 can re-train the first machine learning model using the plurality of learning data obtained by the selection.

[0049] FIG. 3 is a flowchart showing the processing flow of the edge device 21 and the cloud device 22. The machine learning system 10 executes processing in the flow shown in FIG. 3.

[0050] The edge device 21 executes the processing from S12 to S17 every predetermined time (loop processing between S11 and S18).

[0051] Within the loop, first, at S11, the edge device 21 collects the observation results of observing the surroundings of the edge device 21 and generates input data based on the collected observation results. For example, the edge device 21 captures an image of the surroundings and generates image data representing the captured surroundings as the input data.

[0052] Subsequently, at S13, the edge device 21 performs an inference process on the input data based on the first machine learning model. For example, the edge device 21 uses the first machine learning model to classify which of a plurality of predetermined classes the acquired input data belongs to.

[0053] Subsequently, at S14, the edge device 21 transmits the inference result to the cloud device 22 via the network. Note that the edge device 21 may transmit the inference result to the cloud device 22 only when a predetermined inference result is output. For example, the edge device 21 may transmit the inference result to the cloud device 22 only when the person included in the image data is identified as a predetermined person.

[0054] Subsequently, at S15, the edge device 21 calculates a first evaluation value for the acquired input data based on the first evaluation criterion.

[0055] Subsequently, at S16, the edge device 21 determines whether to select the generated input data as candidate data that is a candidate for adoption as learning data based on the first evaluation value.

[0056] When selecting the generated input data as candidate data (Yes in S16), the edge device 21 proceeds with the process to S17.

[0057] At S17, the edge device 21 transmits the generated input data as candidate data to the cloud device 22.

[0058] When the edge device 21 does not select the generated input data as candidate data (No in S16) or when the transmission process in S17 is completed, the edge device 21 finishes the processing within the loop between S11 and S18 and repeats the processing from S12 after a predetermined time.

[0059] Note that the edge device 21 may execute the processing from S13 to S14 and the processing from S15 to S17 in parallel.

[0060] On the other hand, the cloud device 22 executes the processing from S21.

[0061] In S21, the cloud device 22 determines whether it has received candidate data from the edge device 21. If the cloud device 22 has not received candidate data from the edge device 21 (No in S21), the cloud device 22 waits in S21. If the cloud device 22 has received candidate data from the edge device 21 (Yes in S21), the cloud device 22 proceeds with the processing to S22.

[0062] In S22, the cloud device 22 calculates a second evaluation value for the received candidate data based on the second evaluation criterion.

[0063] Subsequently, in S23, the cloud device 22 determines whether to select the received candidate data as learning data based on the second evaluation value.

[0064] If the cloud device 22 selects the received candidate data as learning data (Yes in S23), the cloud device 22 proceeds with the processing to S24.

[0065] In S24, the cloud device 22 stores the received candidate data in the storage unit 47 as learning data.

[0066] If the cloud device 22 does not select the received candidate data as learning data (No in S23) or when the storage process in S24 is completed, the cloud device 22 returns the processing to S21 and repeats the processing from S21.

[0067] Then, when the cloud device 22 receives, for example, a predetermined timing or a predetermined instruction, it trains the first machine learning model using a plurality of learning data stored in the storage unit 47. After training the first machine learning model, the cloud device 22 transmits the parameters set in the first machine learning model to the edge device 21 to update the parameters of the first machine learning model used by the edge device 21 for inference processing.

[0068] FIG. 4 is a diagram for explaining an example of a method for calculating the first evaluation value.

[0069] In the present embodiment, the first evaluation unit 41 calculates the first evaluation value for each of the plurality of input data based on the correlation with other input data. More specifically, the first evaluation unit 41 calculates the first evaluation value for the first input data among the plurality of input data based on the correlation between the first input data and one or more second input data different from the first input data among the plurality of input data. Then, based on the first evaluation value calculated in this way, the first selection unit 42 determines whether to select the first input data as one of a plurality of candidate data that are candidates for adoption as learning data.

[0070] A plurality of time-series input data have the property that the correlation between two or more data that are close in time is high and they are likely to be similar. For this reason, the edge device 21 calculates the first evaluation value based on the correlation between the first input data and one or more second input data different from the first input data, so that a plurality of candidate data with large differences from each other can be selected from the plurality of input data. It is expected that the machine learning model can make accurate inferences for a wide range of input data by learning using a plurality of data with large differences from each other, that is, data with variations. Therefore, the edge device 21 can accurately relearn the first machine learning model by selecting such input data as one of the plurality of candidate data and transmitting it to the cloud device 22 as a candidate for learning data.

[0071] Furthermore, the process of calculating the correlation between data has a lighter load compared to, for example, the process of analyzing the inference results or intermediate results of a machine learning model. Therefore, the edge device 21 can determine whether to select the first input data as candidate data through a relatively simple process by calculating a first evaluation value based on the correlation between the first input data and one or more second input data, and it is easy to implement even if the computing resources of the edge device 21 are limited.

[0072] For example, the first evaluation unit 41 calculates, as the first evaluation value, a value corresponding to the time difference between the acquisition time of the first input data and the acquisition time of the second input data that was selected as one of a plurality of candidate data immediately before the first input data among the one or more second input data. For example, and the first selection unit 42 compares the first evaluation value with a predetermined reference value and selects the first input data as one of the plurality of candidate data. For example, the first evaluation unit 41 generates a larger first evaluation value as the time difference becomes larger. In this case, the first selection unit 42 selects the first input data whose first evaluation value is larger than the reference value as candidate data.

[0073] The greater the difference in acquisition time between the first input data and the second input data included in a plurality of time-series input data, the higher the likelihood of a greater difference. Therefore, the first evaluation unit 41 can cause the input data that allows the machine learning model to be learned with high accuracy to be selected as candidate data by increasing the first evaluation value of the input data with a long acquisition time difference from the previous candidate data.

[0074] Further, for example, the first evaluation unit 41 may calculate, as a first evaluation value, a value corresponding to the degree of difference between the first input data and the k candidate data immediately preceding the first input data (where k is an integer of 1 or more) among the plurality of candidate data. Then, the first selection unit 42 compares the first evaluation value with a predetermined reference value and selects the first input data as one of the plurality of candidate data. For example, the first evaluation unit 41 generates a first evaluation value that becomes a larger value as the difference is larger. In this case, the first selection unit 42 selects the first input data whose first evaluation value is larger than the reference value as the candidate data. The degree of difference between data can be calculated using evaluation indexes used in general image processing, such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), and normalized cross-correlation value, for image data. The degree of difference between other data such as voice and text can also be calculated using similar evaluation indexes.

[0075] For example, the first evaluation unit 41 may calculate the degree of difference based on the average value or the maximum value of the differences between the first input data and each of the k candidate data. Further, the first evaluation unit 41 may calculate the degree of difference based on the average value or the maximum value of the differences between the feature amount obtained by inputting the first input data into a predetermined function or a predetermined model and the feature amounts obtained by inputting each of the k candidate data into the predetermined function or the predetermined model. Also, for example, the first evaluation unit 41 may input the first input data and each of the k candidate data into a function or a model that calculates a correlation value representing the strength of the correlation relationship between two data, calculate the correlation value, and calculate the degree of difference based on the correlation value. By calculating the first evaluation value in this way, the first evaluation unit 41 can also cause the candidate data for accurately training the machine learning model to be selected.

[0076] In addition, the comparison target of the dissimilarity of the first evaluation unit 41 may be selected from the learning data used when training the first machine learning model. For example, k representative values can be obtained from the learning data of the first machine learning model, and the dissimilarity from the first input data can be calculated. Alternatively, the dissimilarity can be calculated from the difference in feature amounts obtained by inputting the k representative values of the learning data and the first input data into a predetermined function or a predetermined model. Or, a new machine learning model that outputs the dissimilarity from the learning data of the first machine learning model is trained separately from the first machine learning model, and candidate data can be selected based on the dissimilarity obtained by inputting the first input data into this machine learning model.

[0077] In addition, the first evaluation value may be a value representing two values, valid or invalid. In this case, the first evaluation unit 41 calculates the first evaluation value based on a random number so that valid or invalid occurs with a preset probability. Further, the first selection unit 42 selects the first input data indicating valid as one of a plurality of candidate data. For example, when the probability of being valid is set to 0.1% in the first evaluation unit 41, the first evaluation value of the first input data is set to a value indicating valid with a probability of 0.1% and a value indicating invalid with a probability of 99.9%.

[0078] In addition, the first evaluation unit 41 may calculate the first evaluation value based on a combination of any two or more of the above-described time difference, the above-described dissimilarity, and the above-described probability.

[0079] For example, when the first evaluation value is a value representing two values, valid or invalid, the first evaluation unit 41 may calculate the first evaluation value by increasing the probability of being valid as the time difference between the acquisition time of the first input data and the acquisition time of the second input data selected as one of a plurality of candidate data immediately before the first input data among one or two or more second input data is larger.

[0080] Further, for example, when the difference between the first input data and the previous k candidate data for the first input data is greater than or equal to a preset threshold value, and a value indicating validity is determined based on a random number with a predetermined probability or a probability according to a time difference, the first evaluation value may be set as the value indicating validity.

[0081] Note that, for example, the first evaluation unit 41 may adopt any calculation method among the above-described time difference, the above-described difference degree, or the above-described probability, or a combination of two or more of these, according to the data processing load in the edge device 21. For example, when the data processing load is smaller than a predetermined value, the first evaluation unit 41 may adopt a calculation method using a combination of two or more. When the data processing load is larger than a predetermined value, for example, the first evaluation unit 41 may adopt a calculation method using any one of the above-described time difference, the above-described difference degree, or the above-described probability. Thereby, the first evaluation unit 41 can select candidate data for accurately training the machine learning model within the range of the computing power of the edge device 21.

[0082] By calculating the first evaluation value as described above and transmitting candidate data that is a candidate for learning data from among a plurality of input data, the edge device 21 can reduce the possibility of transmitting a plurality of candidate data that are similar to each other to the cloud device 22. Thereby, the edge device 21 can select a plurality of candidate data effective for relearning from among a plurality of input data by relatively simple processing, and can reduce the amount of data transmitted to the cloud device 22.

[0083] FIG. 5 is a diagram for explaining a first example of a method for calculating a second evaluation value.

[0084] In this embodiment, the second evaluation unit 45 calculates a second evaluation value for the first candidate data among the plurality of candidate data transmitted from the edge device 21 based on an inference result or an intermediate result obtained by inputting the first candidate data into a machine learning model. For example, the second evaluation unit 45 calculates, as the second evaluation value, a value corresponding to the possibility that the inference result becomes uncertain when the first candidate data is input into the first machine learning model. Then, the second selection unit 46 determines whether to select the first candidate data as one of the plurality of learning data based on the second evaluation value calculated in this way.

[0085] When the value in the input data changes slightly, the inference result of the machine learning model may change. Also, when the parameters such as the weights in the neural network or the connection relationship in the machine learning model change slightly, the inference result may change. Also, in a machine learning model that classifies which of a plurality of classes an input data belongs to, there may be a case where input data with a high classification probability of belonging to two or more classes simultaneously is input. Also, when the arithmetic precision of the hardware on which the machine learning model is executed is different, there may be a case where input data with a changing inference result is input.

[0086] When such each input data is input, the inference result of the machine learning model varies, changes, and becomes highly uncertain. Therefore, it is expected that the machine learning model can stably and accurately perform inference processing even when a wide range of data is input by re-learning using input data with a high uncertainty of such an inference result. Therefore, the cloud device 22 selects such input data as one of the plurality of learning data and uses it for re-learning the first machine learning model. Thereby, the cloud device 22 can re-learn the first machine learning model with high accuracy.

[0087] Furthermore, the process of calculating the second evaluation value by analyzing the inference result or intermediate result obtained by inputting into the machine learning model has a heavier load compared to the process of calculating the correlation relationship between data. However, the cloud device 22 has more computing resources compared to the edge device 21. Therefore, the cloud device 22 can relatively easily determine whether to select the first candidate data as one of the plurality of learning data. In addition, the cloud device 22 calculates the second evaluation value for each of some candidate data among the plurality of input data. Therefore, the cloud device 22 can reduce the processing amount of calculating the second evaluation value.

[0088] In the first example, the first machine learning model classifies whether the input data belongs to any of a plurality of classes. In this case, the second evaluation unit 45 acquires the classification probabilities belonging to each of the plurality of classes obtained by inputting the first candidate data into the first machine learning model. Further, the second evaluation unit 45 calculates, as the second evaluation value, a value corresponding to the degree of difference representing the difference between the classification probability of the class to which the first candidate data is classified among the plurality of classes and the classification probabilities of each of the one or more classes classified as not belonging. Then, the second selection unit 46 compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data.

[0089] For example, the second evaluation unit 45 calculates a second evaluation value that becomes a large value when the degree of difference is small, that is, when the classification probabilities belonging to two or more classes are high. In this case, the second selection unit 46 selects the first candidate data as one of the plurality of learning data when the second evaluation value is greater than the reference value.

[0090] For example, in the example of FIG. 5, for the first machine learning model, when the first candidate data is input, the classification probability of the first class is 35%, the classification probability of the second class is 40%, and the classification probability of the third class is 25%. In such a case, the second evaluation unit 45 calculates the difference (5%) between the classification probability of the second class (40%) and the classification probability of the first class (35%) as the degree of difference, and calculates a second evaluation value that becomes a larger value as the calculated degree of difference is smaller.

[0091] Input data with a small difference between the classification probability of the class obtained as the inference result and the classification probabilities of other classes has a high probability of belonging to two or more classes, and the inference result in the first machine learning model is likely to be uncertain. Therefore, the second evaluation unit 45 according to the first example can cause such candidate data to be selected as one of a plurality of learning data, thereby enabling the first machine learning model to be relearned with high accuracy.

[0092] Note that such classification probabilities for each class are output from the final layer or the layer immediately before the final layer when the machine learning model is a neural network. Therefore, when the second evaluation unit 45 calculates such a second evaluation value, the candidate data transmission unit 43 of the edge device 21 may transmit the candidate data to the cloud device 22 and transmit the intermediate result obtained from the layer that outputs the inference result and the classification probability by the inference unit 32 to the cloud device 22. Then, the second evaluation unit 45 may calculate the second evaluation value based on the inference result and the intermediate result received from the edge device 21. Thereby, the second evaluation unit 45 can reduce the amount of computation for calculating the second evaluation value.

[0093] FIG. 6 is a diagram for explaining a second example of a method for calculating the second evaluation value.

[0094] The second evaluation unit 45 according to the second example calculates, as the second evaluation value, a value corresponding to the degree of difference representing the difference between the first evaluation data calculated by the first arithmetic processing device 25 of the hardware with the first arithmetic accuracy and the second evaluation data calculated by the second arithmetic processing device 26 of the hardware with a second arithmetic accuracy higher than the first arithmetic accuracy.

[0095] More specifically, the first evaluation data according to the second example is calculated by the first arithmetic processing unit 25 included in the edge device 21, and includes at least one of the output data of the first machine learning model and the intermediate data output from a predetermined arithmetic position in the first machine learning model, which is obtained by inputting the first candidate data into the first machine learning model. Further, the second evaluation data according to the second example is calculated by the second arithmetic processing unit 26 included in the cloud device 22, and includes the data corresponding to the first evaluation data among the output data of the first machine learning model and the intermediate data output from a predetermined arithmetic position in the first machine learning model, which is obtained by inputting the first candidate data into the first machine learning model.

[0096] Then, the second selection unit 46 according to the second example compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data. For example, the second evaluation unit 45 calculates a second evaluation value that becomes a larger value as the degree of difference is larger. In this case, the second selection unit 46 selects the first candidate data as one of the plurality of learning data when the second evaluation value is larger than the reference value.

[0097] When the estimation results calculated by the two arithmetic processing units with different arithmetic precisions are different in this way, the input data is regarded as having a high possibility that the inference result in the first machine learning model is uncertain. Therefore, the second evaluation unit 45 according to the second example can make the first machine learning model relearn with high accuracy by selecting such input data as one of the plurality of learning data.

[0098] Note that the candidate data transmission unit 43 of the edge device 21 according to the second example acquires the first evaluation data from the inference unit 32 and transmits it to the cloud device 22 via the network. The second evaluation unit 45 acquires the first evaluation data received from the edge device 21. The second evaluation unit 45 calculates the second evaluation data by the second arithmetic processing unit 26 provided in the cloud device 22. Then, the second evaluation unit 45 calculates a second evaluation value based on the first evaluation data received from the edge device 21 and the second evaluation data calculated by the second arithmetic processing unit 26 provided in the cloud device 22. Thereby, since the second evaluation unit 45 does not have to execute the arithmetic processing for calculating the first evaluation data, the amount of arithmetic for calculating the second evaluation value can be reduced.

[0099] FIG. 7 is a diagram for explaining a third example of a method for calculating the second evaluation value by the second evaluation unit 45.

[0100] The second evaluation unit 45 according to the third example calculates, as the second evaluation value, a value corresponding to the degree of difference between the first evaluation data obtained by inputting the first candidate data into the first machine learning model and the second evaluation data obtained by inputting data obtained by changing a part of the first candidate data into the first machine learning model.

[0101] More specifically, the first evaluation data according to the third example includes at least one of the output data of the first machine learning model and the intermediate data output from a predetermined arithmetic position in the first machine learning model, which are obtained by inputting the first candidate data into the first machine learning model. Further, the second evaluation data according to the third example includes data corresponding to the first evaluation data among the output data of the first machine learning model and the intermediate data output from a predetermined arithmetic position in the first machine learning model, which are obtained by inputting data obtained by changing some values of the first candidate data into the first machine learning model.

[0102] Then, similar to the second example, the second selection unit 46 according to the third example compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data.

[0103] When the estimation results are different for input data with such minute changes, such input data is considered likely to result in uncertain inference results in the first machine learning model. Therefore, the second evaluation unit 45 according to the third example can accurately relearn the first machine learning model by selecting such input data as one of the plurality of learning data.

[0104] In the third example, the candidate data transmission unit 43 of the edge device 21 may transmit the first evaluation data to the cloud device 22, similar to the second example. And in the third example, the second evaluation unit 45 may calculate a second evaluation value based on the first evaluation data received from the edge device 21 and the second evaluation data calculated by the cloud device 22. Thereby, the second evaluation unit 45 can reduce the amount of computation for calculating the second evaluation value.

[0105] FIG. 8 is a diagram for explaining a fourth example of a method for calculating a second evaluation value by the second evaluation unit 45.

[0106] The second evaluation unit 45 according to the fourth example calculates, as the second evaluation value, a value corresponding to the degree of difference between the first evaluation data obtained by inputting the first candidate data into the first machine learning model and the second evaluation data obtained by inputting the first candidate data into a second machine learning model in which a part of the first machine learning model is changed.

[0107] More specifically, the first evaluation data according to the fourth example includes at least one of the output data of the first machine learning model and the intermediate data output from a predetermined calculation position in the first machine learning model, which is obtained by inputting the first candidate data into the first machine learning model. The second evaluation data according to the fourth example includes the data corresponding to the first evaluation data among the output data of the second machine learning model and the intermediate data output from a predetermined calculation position in the second machine learning model, which is obtained by inputting the first candidate data into the second machine learning model in which a part of the first machine learning model is changed.

[0108] The second machine learning model is, for example, a model in which some of the internal parameters of the first machine learning model are changed. For example, when the second machine learning model is a neural network, the weight parameter in the transition that transmits the firing signal from any first node to the next second node is set to 0, so that the path for transmitting the firing signal from the first node to the second node is cut off. Also, the second machine learning model may be a model in which some of the weight parameters of the first machine learning model are slightly changed.

[0109] And, similar to the second example, the second selection unit 46 according to the fourth example compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data.

[0110] When the estimation results calculated by two arithmetic processing units with slightly different configurations are different in this way, the input data is considered to have a high possibility of uncertainty in the inference result in the first machine learning model. Therefore, the second evaluation unit 45 according to the fourth example can cause the first machine learning model to be relearned with high accuracy by selecting such input data as one of the plurality of learning data.

[0111] In the fourth example, the candidate data transmission unit 43 of the edge device 21 may transmit the first evaluation data to the cloud device 22 in the same manner as in the second example. And, in the fourth example, the second evaluation unit 45 may calculate the second evaluation value based on the first evaluation data received from the edge device 21 and the second evaluation data calculated by the cloud device 22. Thereby, the second evaluation unit 45 can reduce the amount of calculation for calculating the second evaluation value.

[0112] FIG. 9 is a diagram for explaining a fifth example of a method for calculating the second evaluation value by the second evaluation unit 45.

[0113] The second evaluation unit 45 according to the fifth example calculates, as a second evaluation value, a value representing the variation of a plurality of output data. Here, the plurality of output data are a plurality of inference results obtained by inputting the first candidate data into a plurality of machine learning models learned with learning parameters different from those of the first machine learning model. For example, the plurality of machine learning models include a second machine learning model, a third machine learning model, a fourth machine learning model, …, an nth machine learning model. In this case, each of the second to nth machine learning models is learned with learning parameters different from those of the first machine learning model and is learned with mutually different learning parameters. Each of the second to nth machine learning models may be a neural network or another model.

[0114] Then, similar to the second example, the second selection unit 46 according to the fifth example compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data.

[0115] When the estimation results calculated using such a plurality of mutually different machine learning models are different, the input data is considered to have a high possibility of uncertainty in the inference result in the first machine learning model. Therefore, the second evaluation unit 45 according to the fifth example can cause the first machine learning model to be relearned with high accuracy by selecting such input data as one of the plurality of learning data.

[0116] FIG. 10 is a diagram for explaining a sixth example of a method for calculating a second evaluation value by the second evaluation unit 45.

[0117] The second evaluation unit 45 according to the sixth example calculates, as a second evaluation value, a value based on the degree of difference representing the difference between the first output data and each of one or more second output data. Here, the first output data is an inference result obtained by inputting the first candidate data into the first machine learning model. Also, each of the one or more second output data is one or more inference results obtained by inputting the first candidate data into one or more machine learning models trained with learning parameters different from those of the first machine learning model. For example, the plurality of machine learning models includes a second machine learning model, a third machine learning model, a fourth machine learning model, …, an nth machine learning model. In this case, each of the second to nth machine learning models is trained with learning parameters different from those of the first machine learning model and is trained with different learning parameters from each other. Each of the one or more machine learning models may be a neural network or another model.

[0118] The second evaluation unit 45 may calculate the degree of difference based on, for example, the average value or the maximum value of the differences between the first output data and each of the one or more second output data. Also, the second evaluation unit 45 may calculate the degree of difference based on, for example, the average value or the maximum value of the differences between the value obtained by inputting the first output data into a predetermined function or a predetermined model and the values obtained by inputting each of the one or more second output data into the predetermined function or the predetermined model. Further, for example, the second evaluation unit 45 may input the first output data and each of the one or more second output data into a function or a model that calculates a correlation value representing the strength of the correlation between two data, calculate the correlation value, and calculate the degree of difference based on the correlation value.

[0119] Then, similar to the second example, the second selection unit 46 according to the sixth example compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data.

[0120] When the estimation result calculated using the first machine learning model is different from the estimation result calculated using the second machine learning model, the input data is considered to have a high probability of resulting in an uncertain inference result in the first machine learning model. Therefore, the second evaluation unit 45 according to the sixth example can accurately re-train the first machine learning model by selecting such input data as one of a plurality of learning data.

[0121] Note that the candidate data transmission unit 43 of the edge device 21 according to the sixth example acquires the first output data from the inference unit 32 and transmits it to the cloud device 22 via the network. The second evaluation unit 45 acquires the first output data received from the edge device 21. The second evaluation unit 45 calculates each of one or more second output data. Then, the second evaluation unit 45 calculates a second evaluation value based on the first output data received from the edge device 21 and the one or more calculated second output data. As a result, the second evaluation unit 45 does not need to execute the arithmetic process for calculating the first output data, so that the amount of arithmetic operations for calculating the second evaluation value can be reduced.

[0122] FIG. 11 is a block diagram showing the functional configurations of the edge device 21 and the cloud device 22 according to the modified example.

[0123] The edge device 21 and the cloud device 22 may have a configuration as shown in FIG. 11. That is, the cloud device 22 further includes a feedback unit 61. The edge device 21 further includes a probability change unit 62.

[0124] The edge device 21 generates a plurality of input data arranged in time series, and for each generated input data, determines whether to select the generated input data as candidate data. Then, when the generated input data is selected as candidate data, the edge device 21 transmits the candidate data to the cloud device 22 via the network.

[0125] Also, every time the cloud device 22 receives candidate data from the edge device 21, it determines whether to select the received candidate data as learning data. If the received candidate data is selected as learning data, the selected candidate data is stored in the storage unit 47 as learning data.

[0126] Here, in the modification example, every time learning data is selected by the second selection unit 46, the feedback unit 61 transmits, via the network, adoption information indicating that the input data corresponding to the selected learning data has been selected as learning data to the edge device 21.

[0127] The probability change unit 62 receives the adoption information from the cloud device 22 via the network. When the probability change unit 62 receives the adoption information, it increases the probability of selecting, as candidate data, the input data acquired within a predetermined time range after the input data indicated in the adoption information, compared to the probability of selecting other time ranges.

[0128] A plurality of time-series input data has the property that the correlation between two or more data that are temporally close is high and the possibility of similarity is high. Therefore, for other input data in the vicinity in time of the input data for which it is determined that the inference result in the first machine learning model is likely to be uncertain, it is similarly predicted that the inference result is likely to be uncertain. Accordingly, by increasing the probability of selecting, as candidate data, the input data in the vicinity of the input data for which it is determined that the inference result is likely to be uncertain, the probability change unit 62 can cause the cloud device 22 to transmit more input data that is likely to be adopted as learning data. Thereby, the probability change unit 62 can accurately relearn the first machine learning model.

[0129] As described above, the machine learning system 10 according to this embodiment can reduce the possibility of transmitting a plurality of candidate data that are similar to each other and not effective for learning to the cloud device 22 by calculating the first evaluation value at the edge device 21 and transmitting the candidate data that are candidates for learning data from among the plurality of input data. As a result, the edge device 21 can select a plurality of candidate data effective for learning from among the plurality of input data by relatively simple processing, and can reduce the amount of data transmitted to the cloud device 22.

[0130] Furthermore, since the machine learning system 10 according to this embodiment calculates the second evaluation value at the cloud device 22 with high computing power and selects learning data from among the plurality of candidate data, it is possible to accurately select input data for which the inference result is likely to be uncertain. Therefore, according to the machine learning system 10 according to this embodiment, it is possible to accurately and efficiently select input data that is effective for learning from among the plurality of input data.

[0131] FIG. 12 is a diagram showing an example of the hardware configuration of the information processing apparatus constituting the edge device 21 and the cloud device 22. The information processing apparatus constituting the edge device 21 and the cloud device 22 is realized by a hardware configuration similar to that of a computer as shown in FIG. 12, for example. Note that the information processing apparatus constituting the cloud device 22 may be configured not to include the operation input device 304 and the display device 305.

[0132] The information processing apparatus includes a CPU 301, a RAM (Random Access Memory) 302, a ROM (Read Only Memory) 303, an operation input device 304, a display device 305, a storage device 306, and a communication device 307. And each of these parts is connected by a bus.

[0133] The CPU 301 is a processor that executes arithmetic processing, control processing, etc. according to a program. The CPU 301 uses a predetermined area of the RAM 302 as a work area and executes various processes in cooperation with programs stored in the ROM 303, the storage device 306, etc.

[0134] The RAM 302 is a memory such as an SDRAM (Synchronous Dynamic Random Access Memory). The RAM 302 functions as a work area for the CPU 301. The ROM 303 is a memory that stores programs and various information in a non-rewritable manner.

[0135] The operation input device 304 is an input device such as a mouse and a keyboard. The operation input device 304 receives information input by the user as an instruction signal and outputs the instruction signal to the CPU 301.

[0136] The display device 305 is a display device such as an LCD (Liquid Crystal Display). The display device 305 displays various information based on a display signal from the CPU 301.

[0137] The storage device 306 is a device that writes and reads data to and from a semiconductor storage medium such as a flash memory, or a magnetic or optically recordable storage medium, etc. The storage device 306 writes and reads data to and from the storage medium according to control from the CPU 301. The communication device 307 communicates with external devices via a network according to control from the CPU 301.

[0138] The program executed by the information processing apparatus that realizes the edge device 21 has a module configuration including an input data generation module, an inference module, a result transmission module, a first evaluation module, a first selection module, and a candidate data transmission module. By being expanded and executed on the RAM 302 by the CPU 301 (processor), this program causes the information processing apparatus to function as an input data generation unit 31, an inference unit 32, a result transmission unit 33, a first evaluation unit 41, a first selection unit 42, and a candidate data transmission unit 43. Note that part or all of the input data generation unit 31, inference unit 32, result transmission unit 33, first evaluation unit 41, first selection unit 42, and candidate data transmission unit 43 of the information processing apparatus may be realized by a hardware circuit.

[0139] The program executed by the information processing apparatus that realizes the cloud device 22 has a module configuration including a result reception module, an output module, a candidate data reception module, a second evaluation module, a second selection module, and a learning module. By being expanded and executed on the RAM 302 by the CPU 301 (processor), this program causes the information processing apparatus to function as a result reception unit 34, an output unit 35, a candidate data reception unit 44, a second evaluation unit 45, a second selection unit 46, and a learning unit 48. Note that part or all of the result reception unit 34, output unit 35, candidate data reception unit 44, second evaluation unit 45, second selection unit 46, and learning unit 48 of the information processing apparatus may be realized by a hardware circuit.

[0140] Also, the program executed by a computer is a file in a form installable or executable on the computer, and is provided by being recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk, a CD-R, or a DVD (Digital Versatile Disk).

[0141] Alternatively, the program may be stored on a computer connected to a network such as the Internet and provided by allowing it to be downloaded via the network. Further, the program may be configured to be provided or distributed via a network such as the Internet. Also, the program executed by the information processing apparatus may be configured to be provided by being pre - incorporated into a ROM 303 or the like.

[0142] Although some embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention and are included in the invention described in the claims and its equivalent scope.

[0143] (Supplementary Note) Note that the above - described embodiments can be summarized into the following technical proposals.

[0144] Technical Proposal 1 A machine learning system that selects a plurality of training data for training a first machine learning model from a plurality of input data, a first information processing apparatus, a second information processing apparatus connected to the first information processing apparatus via a network, comprising: the first information processing apparatus includes a first evaluation unit that calculates a first evaluation value representing the effectiveness when used for training the first machine learning model in each of the plurality of input data based on a predetermined first evaluation criterion; a first selection unit that selects whether to include each of the plurality of input data in a plurality of candidate data by comparing each of the first evaluation values of the plurality of input data with a predetermined value; a candidate data transmission unit that transmits each of the plurality of candidate data to the second information processing apparatus via the network has, the second information processing apparatus a candidate data receiving unit that receives each of the plurality of candidate data from the first information processing apparatus via the network; a second evaluation unit that calculates a second evaluation value indicating the effectiveness when used for learning of the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion; a second selection unit that selects whether to include each of the plurality of candidate data in the plurality of learning data by comparing the second evaluation value of each of the plurality of candidate data with a predetermined value; A machine learning system having

[0145] Technical solution 2 The first evaluation unit calculates the first evaluation value for the first input data among the plurality of input data based on the relationship between the first input data and data different from the first input data. The machine learning system according to Technical solution 1.

[0146] Technical solution 3 The first evaluation unit calculates, as the first evaluation value, a value corresponding to the time difference between the acquisition time of the first input data and the acquisition time of second input data selected as one of the plurality of candidate data immediately before the first input data among one or two or more second input data different from the first input data, The first selection unit compares the first evaluation value with a predetermined reference value and selects the first input data as one of the plurality of candidate data. The machine learning system according to Technical solution 2.

[0147] Technical solution 4 The first evaluation unit calculates, as the first evaluation value, a value corresponding to the degree of difference representing the difference between the first input data and the k candidate data immediately before the first input data among the plurality of candidate data (k is an integer of 1 or more). The first selection unit compares the first evaluation value with a predetermined reference value, and selects the first input data as one of the plurality of candidate data. The machine learning system according to Technical Proposal 2.

[0148] Technical Proposal 5 The first evaluation unit calculates, as the first evaluation value, a value corresponding to a degree of difference representing a difference between the first input data and one or more data used for training the first machine learning model. The first selection unit compares the first evaluation value with a predetermined reference value, and selects the first input data as one of the plurality of candidate data. The machine learning system according to Technical Proposal 2.

[0149] Technical Proposal 6 The first evaluation value represents two values, valid or invalid. The first evaluation unit calculates the first evaluation value based on a random number so that the valid or the invalid occurs with a preset probability. The first selection unit selects the first input data as one of the plurality of candidate data when the first evaluation value indicates selection. The machine learning system according to any one of Technical Proposals 2 to 5.

[0150] Technical Proposal 7 The second evaluation unit calculates, for the first candidate data among the plurality of candidate data, the second evaluation value by analyzing an inference result or an intermediate result obtained by inputting the first candidate data into a machine learning model. The machine learning system according to any one of Technical Proposals 1 to 6.

[0151] Technical Proposal 8 The first machine learning model classifies whether input data belongs to any one of a plurality of classes. The second evaluation unit obtains classification probabilities belonging to each of the plurality of classes, which are obtained by inputting the first candidate data into the first machine learning model, and calculates, as the second evaluation value, a value corresponding to a difference degree representing a difference between the classification probability of the class to which the first candidate data is classified among the plurality of classes and the classification probabilities of each of one or more classes that are classified as not belonging, The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The machine learning system according to Technical Proposal 7.

[0152] Technical Proposal 9 The second evaluation unit calculates, as the second evaluation value, a value corresponding to a difference degree representing a difference between first evaluation data calculated by a first arithmetic processing device of hardware with a first arithmetic accuracy and second evaluation data calculated by a second arithmetic processing device of hardware with a second arithmetic accuracy higher than the first arithmetic accuracy, The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The first evaluation data includes at least one of output data of the first machine learning model and intermediate data output from a predetermined arithmetic position in the first machine learning model, which are obtained by inputting the first candidate data into the first machine learning model. The second evaluation data includes data corresponding to the first evaluation data among the output data of the first machine learning model and the intermediate data output from the predetermined arithmetic position in the first machine learning model, which are obtained by inputting the first candidate data into the first machine learning model. The machine learning system according to Technical Proposal 7.

[0153] Technical Proposal 10 The second evaluation unit calculates, as the second evaluation value, a value corresponding to a degree of difference representing a difference between first evaluation data obtained by inputting the first candidate data into the first machine learning model and second evaluation data obtained by inputting data obtained by changing a part of the first candidate data into the first machine learning model. The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The first evaluation data includes at least one of output data of the first machine learning model and intermediate data output from a predetermined calculation position in the first machine learning model. The second evaluation data includes data corresponding to the first evaluation data among output data of the first machine learning model and intermediate data output from the predetermined calculation position in the first machine learning model. The machine learning system according to Technical Proposal 7.

[0154] Technical Proposal 11 The second evaluation unit calculates, as the second evaluation value, a value corresponding to a degree of difference representing a difference between first evaluation data obtained by inputting the first candidate data into the first machine learning model and second evaluation data obtained by inputting the first candidate data into a second machine learning model obtained by changing a part of the first machine learning model. The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The first evaluation data includes at least one of output data of the first machine learning model and intermediate data output from a predetermined calculation position in the first machine learning model. The second evaluation data includes data corresponding to the first evaluation data among output data of the second machine learning model and intermediate data output from the predetermined calculation position in the second machine learning model. The machine learning system according to Technical Proposal 7.

[0155] Technical Proposal 12 The second evaluation unit calculates, as the second evaluation value, a value representing the variation of a plurality of output data. The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The plurality of output data are a plurality of inference results obtained by inputting the first candidate data into a plurality of machine learning models trained with learning parameters different from those of the first machine learning model. The machine learning system according to Technical Proposal 7.

[0156] Technical Proposal 13 The second evaluation unit calculates, as the second evaluation value, a value based on the degree of difference representing the difference between the first output data and each of one or more second output data. The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The first output data is an inference result obtained by inputting the first candidate data into the first machine learning model. Each of the one or more second output data is one or more inference results obtained by inputting the first candidate data into one or more machine learning models trained with learning parameters different from those of the first machine learning model. The machine learning system according to Technical Proposal 7.

[0157] Technical Proposal 14 The second information processing device further includes a feedback unit that transmits, to the first information processing device, adoption information indicating that the corresponding input data has been selected as the learning data every time the learning data is selected. The first information processing device receives the adoption information, and makes the probability of selecting, as candidate data, the input data acquired in a predetermined time range after the input data indicated in the adoption information higher than the probability of selecting other time ranges. The machine learning system according to any one of Technical Proposals 1 to 13.

[0158] Technical Solution 15 The first information processing device further includes: an input data generation unit that collects observation results of observing the surroundings and generates the plurality of input data in time series; an inference unit that performs inference processing in time series on each of the plurality of input data based on the first machine learning model and outputs the inference results obtained by the inference processing in time series; and further includes: the first selection unit determines whether to select each of the plurality of input data as a candidate in time series based on the corresponding first evaluation value; the second selection unit determines whether to include each of the plurality of candidate data in the plurality of learning data in time series based on the corresponding second evaluation value The machine learning system according to any one of Technical Solutions 1 to 14.

[0159] Technical Solution 16 The second information processing device further includes: a storage unit that stores the plurality of learning data; a learning unit that trains the first machine learning model using the plurality of learning data stored in the storage unit; The machine learning system according to any one of Technical Solutions 1 to 15, further including:

[0160] Technical Solution 17 a first information processing device including a first arithmetic processing device of hardware; a second information processing device including a second arithmetic processing device that is hardware different from the first arithmetic processing device and performs information processing with higher arithmetic accuracy than the first arithmetic processing device; and includes: The first information processing device uses the first arithmetic processing device to generate first evaluation data including at least one of output data of the first machine learning model and intermediate data output from a predetermined arithmetic position in the first machine learning model, which are obtained by inputting each of the plurality of input data into the first machine learning model; The second information processing device generates second evaluation data including at least one of the output data of the first machine learning model and the intermediate data output from the predetermined calculation position of the first machine learning model, which is obtained by inputting each of the plurality of input data into the first machine learning model using the second arithmetic processing device. The second information processing device selects, as learning data for training the first machine learning model, input data among the plurality of input data for which the difference between the first evaluation data and the second evaluation data is greater than a predetermined reference value. Machine learning system.

[0161] Technical proposal 18 A machine learning system comprising an edge device and an information processing device connected to the edge device via a network, and selecting a plurality of learning data for training a first machine learning model from among a plurality of input data, wherein the edge device is The edge device a first evaluation unit that calculates a first evaluation value representing the effectiveness when used for learning the first machine learning model in each of the plurality of input data based on a predetermined first evaluation criterion; a first selection unit that selects whether to include each of the plurality of input data in a plurality of candidate data by comparing each of the first evaluation values of the plurality of input data with a predetermined value; a candidate data transmission unit that transmits each of the plurality of candidate data to the information processing device via the network; and has The information processing device a candidate data reception unit that receives each of the plurality of candidate data from the edge device via the network; a second evaluation unit that calculates a second evaluation value indicating the effectiveness when used for learning the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion; A second selection unit that selects whether to include each of the plurality of candidate data in the plurality of learning data by comparing each of the second evaluation values of the plurality of candidate data with a predetermined value. An edge device having

[0162] Technical proposal 19 An information processing apparatus in a machine learning system including an edge device and an information processing apparatus connected to the edge device via a network, the information processing apparatus selecting a plurality of learning data for training a first machine learning model from a plurality of input data, The edge device A first evaluation unit that calculates a first evaluation value representing the effectiveness when used for training the first machine learning model in each of the plurality of input data based on a predetermined first evaluation criterion; A first selection unit that selects whether to include each of the plurality of input data in a plurality of candidate data by comparing each of the first evaluation values of the plurality of input data with a predetermined value; A candidate data transmission unit that transmits each of the plurality of candidate data to the information processing apparatus via the network; and has The information processing apparatus A candidate data reception unit that receives each of the plurality of candidate data from the edge device via the network; A second evaluation unit that calculates a second evaluation value indicating the effectiveness when used for training the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion; A second selection unit that selects whether to include each of the plurality of candidate data in the plurality of learning data by comparing each of the second evaluation values of the plurality of candidate data with a predetermined value; and has an information processing apparatus.

Explanation of reference numerals

[0163] 10 Machine learning system 21 Edge device 22 Cloud device 25 First arithmetic processing unit 26 Second arithmetic processing unit 31 Input data generation unit 32 Inference unit 33 Result transmission unit 34 Result reception unit 35 Output unit 41 First evaluation unit 42 First selection unit 43 Candidate data transmission unit 44 Candidate data reception unit 45 Second evaluation unit 46 Second selection unit 47 Memory unit 48 Learning unit 61 Feedback unit 62 Probability change unit

Claims

1. A machine learning system that selects a plurality of training data for training a first machine learning model from among a plurality of input data, comprising: a first information processing device; a second information processing device connected to the first information processing device via a network; and comprising: The first information processing device: a first evaluation unit that calculates a first evaluation value representing the effectiveness when used for training the first machine learning model in each of the plurality of input data based on a predetermined first evaluation criterion; a first selection unit that selects whether to include each of the plurality of input data in a plurality of candidate data by comparing each of the first evaluation values of the plurality of input data with a predetermined value; a candidate data transmission unit that transmits each of the plurality of candidate data to the second information processing device via the network; and having: The second information processing device: a candidate data reception unit that receives each of the plurality of candidate data from the first information processing device via the network; a second evaluation unit that calculates a second evaluation value indicating the effectiveness when used for training the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion; a second selection unit that selects whether to include each of the plurality of candidate data in the plurality of training data by comparing each of the second evaluation values of the plurality of candidate data with a predetermined value; A machine learning system having.

2. The first evaluation unit calculates the first evaluation value for the first input data among the plurality of input data based on the relationship between the first input data and data different from the first input data. The machine learning system according to claim 1.

3. The first evaluation unit calculates, as the first evaluation value, a value corresponding to the time difference between the acquisition time of the first input data and the acquisition time of the second input data selected as one of the plurality of candidate data immediately before the first input data among one or more second input data different from the first input data. The first selection unit selects the first input data as one of the plurality of candidate data by comparing the first evaluation value with a predetermined reference value. The machine learning system according to claim 2.

4. The first evaluation unit calculates, as the first evaluation value, a value corresponding to a degree of difference representing a difference between the first input data and the k candidate data immediately preceding the first input data among the plurality of candidate data (k is an integer of 1 or more). The first selection unit compares the first evaluation value with a predetermined reference value and selects the first input data as one of the plurality of candidate data. The machine learning system according to claim 2.

5. The first evaluation unit calculates, as the first evaluation value, a value corresponding to a degree of difference representing a difference between the first input data and one or more data used for training the first machine learning model. The first selection unit compares the first evaluation value with a predetermined reference value and selects the first input data as one of the plurality of candidate data. The machine learning system according to claim 2.

6. The first evaluation value represents two values, valid or invalid. The first evaluation unit calculates the first evaluation value based on a random number so that the valid or the invalid occurs with a preset probability. The first selection unit selects the first input data as one of the plurality of candidate data when the first evaluation value indicates selection. The machine learning system according to claim 2.

7. The second evaluation unit calculates the second evaluation value for the first candidate data among the plurality of candidate data by analyzing an inference result or an intermediate result obtained by inputting the first candidate data into a machine learning model. The machine learning system according to any one of claims 1 to 6.

8. The first machine learning model classifies to which class among a plurality of classes the input data belongs. The second evaluation unit obtains classification probabilities belonging to each of the plurality of classes obtained by inputting the first candidate data into the first machine learning model, and a difference between the classification probability of the class to which the first candidate data is classified as belonging among the plurality of classes and the classification probabilities of one or more classes classified as not belonging. A value corresponding to the degree of difference is calculated as the second evaluation value. The second selection unit compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data. The machine learning system according to claim 7.

9. The second evaluation unit calculates, as the second evaluation value, a value corresponding to a degree of difference representing a difference between first evaluation data calculated by a first arithmetic processing unit of hardware with a first arithmetic accuracy and second evaluation data calculated by a second arithmetic processing unit of hardware with a second arithmetic accuracy higher than the first arithmetic accuracy. The second selection unit compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data. The first evaluation data includes at least one of output data of the first machine learning model and intermediate data output from a predetermined arithmetic position in the first machine learning model, which is obtained by inputting the first candidate data into the first machine learning model. The second evaluation data includes data corresponding to the first evaluation data among the output data of the first machine learning model and the intermediate data output from the predetermined arithmetic position in the first machine learning model, which is obtained by inputting the first candidate data into the first machine learning model. The machine learning system according to claim 7.

10. The second evaluation unit calculates, as the second evaluation value, a value corresponding to a degree of difference representing a difference between first evaluation data obtained by inputting the first candidate data into the first machine learning model and second evaluation data obtained by inputting data obtained by changing a part of the first candidate data into the first machine learning model. The second selection unit compares the second evaluation value with a predetermined reference value and selects the first candidate data as one of the plurality of learning data. The first evaluation data includes at least one of output data of the first machine learning model and intermediate data output from a predetermined arithmetic position in the first machine learning model. The second evaluation data includes data corresponding to the first evaluation data among the output data of the first machine learning model and the intermediate data output from the predetermined arithmetic position in the first machine learning model. The machine learning system according to claim 7.

11. The second evaluation unit calculates, as the second evaluation value, a value corresponding to a degree of difference representing a difference between first evaluation data obtained by inputting the first candidate data into the first machine learning model and second evaluation data obtained by inputting the first candidate data into a second machine learning model obtained by changing a part of the first machine learning model. The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The first evaluation data includes at least one of output data of the first machine learning model and intermediate data output from a predetermined calculation position in the first machine learning model. The second evaluation data includes data corresponding to the first evaluation data among output data of the second machine learning model and intermediate data output from the predetermined calculation position in the second machine learning model. The machine learning system according to claim 7.

12. The second evaluation unit calculates, as the second evaluation value, a value representing the variation of a plurality of output data. The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The plurality of output data are a plurality of inference results obtained by inputting the first candidate data into a plurality of machine learning models trained with learning parameters different from those of the first machine learning model. The machine learning system according to claim 7.

13. The second evaluation unit calculates, as the second evaluation value, a value based on the degree of difference representing the difference between the first output data and each of one or more second output data. The second selection unit compares the second evaluation value with a predetermined reference value, and selects the first candidate data as one of the plurality of learning data. The first output data is an inference result obtained by inputting the first candidate data into the first machine learning model. Each of the one or more second output data is one or more inference results obtained by inputting the first candidate data into one or more machine learning models trained with learning parameters different from those of the first machine learning model. The machine learning system according to claim 7.

14. The second information processing device further includes a feedback unit that transmits, to the first information processing device, adoption information indicating that the corresponding input data has been selected as the learning data every time the learning data is selected. The first information processing device receives the adoption information, and makes the probability of selecting the input data acquired in a predetermined time range after the input data indicated in the adoption information as the candidate data higher than the probability of selecting other time ranges. The machine learning system according to claim 1.

15. The first information processing device includes an input data generation unit that collects observation results of observing the surroundings and generates the plurality of input data in time series, and an inference unit that performs inference processing in time series on each of the plurality of input data based on the first machine learning model and outputs the inference results obtained by the inference processing in time series. The first information processing device further includes: the first selection unit determines whether to select each of the plurality of input data as a candidate based on the corresponding first evaluation value in time series; the second selection unit determines whether to include each of the plurality of candidate data in the plurality of learning data based on the corresponding second evaluation value in time series. The machine learning system according to claim 1.

16. A first information processing device including a first arithmetic processing device of hardware, and a second information processing device including a second arithmetic processing device that is hardware different from the first arithmetic processing device and executes information processing with higher arithmetic accuracy than the first arithmetic processing device. The machine learning system includes: the first information processing device generates first evaluation data including at least one of output data of the first machine learning model and intermediate data output from a predetermined arithmetic position in the first machine learning model, obtained by inputting each of the plurality of input data into the first machine learning model using the first arithmetic processing device; the second information processing device generates second evaluation data including at least one of the output data of the first machine learning model and the intermediate data output from the predetermined arithmetic position of the first machine learning model, obtained by inputting each of the plurality of input data into the first machine learning model using the second arithmetic processing device; the second information processing device selects, as learning data for training the first machine learning model, input data among the plurality of input data for which the difference between the first evaluation data and the second evaluation data is greater than a predetermined reference value. Machine learning system.

17. In a machine learning system including an edge device and an information processing device connected to the edge device via a network, and selecting a plurality of learning data for training a first machine learning model from among a plurality of input data, the edge device wherein the edge device A first evaluation unit that calculates a first evaluation value representing the effectiveness when used for learning the first machine learning model in each of the plurality of input data based on a predetermined first evaluation criterion; A first selection unit that selects whether to include each of the plurality of input data in a plurality of candidate data by comparing each of the first evaluation values of the plurality of input data with a predetermined value; A candidate data transmission unit that transmits each of the plurality of candidate data to the information processing device via the network; having; The information processing device is A candidate data reception unit that receives each of the plurality of candidate data from the edge device via the network; A second evaluation unit that calculates a second evaluation value indicating the effectiveness when used for learning the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion; A second selection unit that selects whether to include each of the plurality of candidate data in the plurality of learning data by comparing each of the second evaluation values of the plurality of candidate data with a predetermined value; An edge device having.

18. An information processing device in a machine learning system that includes an edge device and an information processing device connected to the edge device via a network, and selects a plurality of learning data for learning a first machine learning model from among a plurality of input data, The edge device is A first evaluation unit that calculates a first evaluation value representing the effectiveness when used for learning the first machine learning model in each of the plurality of input data based on a predetermined first evaluation criterion; A first selection unit that selects whether to include each of the plurality of input data in a plurality of candidate data by comparing each of the first evaluation values of the plurality of input data with a predetermined value; A candidate data transmission unit that transmits each of the plurality of candidate data to the information processing device via the network; having; The information processing device is A candidate data reception unit that receives each of the plurality of candidate data from the edge device via the network; A second evaluation unit that calculates a second evaluation value indicating the effectiveness when used for learning the first machine learning model in each of the plurality of candidate data based on a predetermined second evaluation criterion different from the first evaluation criterion; A second selection unit that selects whether to include each of the plurality of candidate data in the plurality of learning data by comparing each of the second evaluation values of the plurality of candidate data with a predetermined value; An information processing apparatus having the above.

Images