Learning device, estimation system, learning method, and program

Abstract

In order to, by using a code obtained by reducing the dimensions of raw data, construct a model for performing highly accurate estimation, while correctly capturing a causal relationship, this training device comprises: an acquisition unit that acquires a data set of raw data and correct-answer data; a feature quantity calculation unit that uses the raw data to calculate a feature quantity; a model construction unit that constructs an encoding model which outputs a code corresponding to the feature quantity in response to the input of the raw data, and an estimation model which outputs an estimation result corresponding to the raw data in response to the input of the code; and a training processing unit that, on the basis of the relationship between the code and the feature quantity, trains the encoding model and the estimation model such that the estimation result matches the correct-answer data.

Description

[Technical field]

[0001] The present disclosure relates to a learning device and the like that constructs a model for making estimations using data through machine learning. [Background technology]

[0002] With the spread of IoT (Internet of Things) technology, various information about people and things can be collected from various IoT devices. Attempts have been made to apply machine learning to information collected by IoT devices and utilize the information in fields such as medicine / healthcare and security. The classification accuracy of a model built by machine learning can be analyzed using a general-purpose data set. For example, Non-Patent Document 1 discloses a data set related to clothing and shoes. In an IoT device such as a sensor with a limited power supply capacity, for example, restrictions are imposed on communication processing, which consumes more power than other processing. In such an IoT device, it is difficult to transmit a large amount of data frequently.

[0003] Patent Document 1 discloses a data analysis system that reduces the dimension of observation data observed by an instrument and transmits the low-dimensional observation data to an instrument that analyzes the observation data. The instrument inputs the observation data to an input layer of a trained neural network and processes the data up to a predetermined intermediate layer. The trained neural network is configured so that the number of nodes in the predetermined intermediate layer is smaller than the number of nodes in the output layer. The trained neural network is also trained in advance so that, under a predetermined constraint, the probability distribution of low-dimensional observation data for observation data with different analysis results is less likely to overlap than when there is no predetermined constraint. The instrument transmits the results of processing up to the predetermined intermediate layer to the instrument as low-dimensional observation data. The instrument analyzes the observation data observed by the instrument by inputting the received low-dimensional observation data to an intermediate layer next to the predetermined intermediate layer and processing the data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 203232 [Non-patent literature]

[0005] [Non-Patent Document 1] H. Xiao, K. Rasul, R. Vollgraf, “Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms”, arXiv preprint arXiv:1708.07747 (2017). Summary of the Invention [Problem to be solved by the invention]

[0006] In the method of Patent Document 1, low-dimensional observation data that has been processed up to a predetermined intermediate layer is transmitted from an instrument to an apparatus. Therefore, according to the method of Patent Document 1, the amount of data transmitted from an instrument to an apparatus can be reduced. However, analysis using reduced-dimensional data may result in a decrease in estimation accuracy.

[0007] In addition, a model trained by machine learning as in Patent Document 1 can capture the correlation between the measurement data and the correct answer data, but cannot capture the causal relationship between the measurement data and the correct answer data. For example, when a model that discriminates hallux valgus using sensor data related to foot movement is trained by machine learning, training may be performed differently from the assumption. Since hallux valgus is common in women, a model trained by machine learning may be trained to extract the characteristics of a woman's gait from the sensor data. In other words, a model built on the assumption that hallux valgus will be discriminated may be trained to discriminate whether or not a person is female based on the sensor data. However, there is no causal relationship between hallux valgus and being female. Therefore, when a model trained in this way is applied to a man, the estimation accuracy of hallux valgus discrimination may be reduced.

[0008] An object of the present disclosure is to provide a learning device etc. that can construct a model that performs highly accurate estimation while correctly grasping causal relationships by using codes that reduce the dimensionality of raw data. [Means for solving the problem]

[0009] A learning device of one embodiment of the present disclosure includes an acquisition unit that acquires a dataset of raw data and ground truth data, a feature calculation unit that calculates features using the raw data, a model construction unit that constructs an encoding model that outputs a code corresponding to the feature in response to input raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input the code, and a learning processing unit that trains the encoding model and the estimation model based on the relationship between the code and the feature so that the estimation result matches the ground truth data.

[0010] In a learning method according to one aspect of the present disclosure, a computer acquires a dataset of raw data and ground truth data, calculates features using the raw data, and constructs an encoding model that outputs a code corresponding to the feature in response to input raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input code, and trains the encoding model and the estimation model based on the relationship between the code and the feature so that the estimation result matches the ground truth data.

[0011] A program according to one aspect of the present disclosure causes a computer to execute the following processes: acquiring a dataset of raw data and ground truth data; calculating features using the raw data; constructing an encoding model that outputs a code corresponding to the feature in response to input raw data; and an estimation model that outputs an estimation result corresponding to the raw data in response to input the code; and training the encoding model and the estimation model based on the relationship between the code and the feature so that the estimation result matches the ground truth data. Effect of the Invention

[0012] According to the present disclosure, it is possible to provide a learning device or the like that can construct a model that performs highly accurate estimation while correctly grasping causal relationships by using codes that reduce the dimensionality of raw data. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of a configuration of a learning device according to a first embodiment. [Diagram 2] 2 is a conceptual diagram for explaining an example of model training by a learning processing unit of the learning device according to the first embodiment. FIG. [Diagram 3] 4 is a flowchart for explaining an example of an operation of the learning device according to the first embodiment. [Figure 4] 4 is a flowchart for explaining an example of a training process by a learning processing unit of the learning device according to the first embodiment. [Diagram 5] FIG. 11 is a conceptual diagram for explaining another example of model training by the learning processing unit of the learning device according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing an example of the configuration of a learning device according to a second embodiment. [Figure 7] FIG. 11 is a conceptual diagram for explaining an example of model training by a learning processing unit of a learning device according to a second embodiment. [Figure 8] 10 is a flowchart illustrating an example of an operation of a learning device according to a second embodiment. [Figure 9] 13 is a flowchart for explaining an example of a training process by a learning processing unit of a learning device according to a second embodiment. [Figure 10] FIG. 11 is a conceptual diagram for explaining another example of model training by the learning processing unit of the learning device according to the second embodiment. [Figure 11] FIG. 13 is a block diagram showing an example of the configuration of a learning device according to a third embodiment. [Figure 12] FIG. 13 is a conceptual diagram for explaining an example of model training by a learning processing unit of a learning device according to a third embodiment. [Figure 13] 13 is a flowchart illustrating an example of an operation of the learning device according to the third embodiment. [Figure 14] 13 is a flowchart for explaining an example of a training process by a learning processing unit of a learning device according to a third embodiment. [Figure 15] FIG. 13 is a block diagram showing an example of the configuration of a learning device according to a fourth embodiment. [Figure 16] FIG. 13 is a block diagram showing an example of a configuration of an estimation system according to a fifth embodiment. [Figure 17] FIG. 13 is a conceptual diagram showing an example of the arrangement of measurement devices in an estimation system according to a fifth embodiment. [Figure 18] FIG. 13 is a block diagram showing an example of the configuration of a measurement device of an estimation system according to a fifth embodiment. [Figure 19] FIG. 13 is a block diagram showing an example of a configuration of an estimation device of an estimation system according to a fifth embodiment. [Figure 20] FIG. 13 is a conceptual diagram for explaining estimation and model updating by an estimation system according to a fifth embodiment. [Figure 21] FIG. 13 is a conceptual diagram showing an example in which an estimation result by the estimation system according to the fifth embodiment is displayed on a screen of a mobile terminal. [Figure 22] 13 is a flowchart for explaining an example of an operation of a measurement device in an estimation system according to a fifth embodiment. [Figure 23] 13 is a flowchart illustrating an example of an operation of the estimation device of the estimation system according to the fifth embodiment. [Figure 24] FIG. 13 is a block diagram showing an example of a configuration of an estimation system according to a sixth embodiment. [Diagram 25] FIG. 13 is a block diagram showing an example of the configuration of a measurement device of an estimation system according to a sixth embodiment. [Figure 26] FIG. 13 is a block diagram showing an example of a configuration of an estimation device of an estimation system according to a sixth embodiment. [Figure 27] FIG. 13 is a conceptual diagram for explaining estimation and model updating by an estimation system according to a sixth embodiment. [Figure 28] FIG. 13 is a conceptual diagram showing an example in which an estimation result by an estimation system according to a sixth embodiment is displayed on a screen of a mobile terminal. [Figure 29] 23 is a flowchart for explaining an example of an operation of a measurement device in an estimation system according to a sixth embodiment. [Diagram 30] FIG. 13 is a block diagram showing an example of the configuration of an estimation system according to a seventh embodiment. [Diagram 31] FIG. 23 is a block diagram showing an example of the configuration of a measurement device of an estimation system according to a seventh embodiment. [Diagram 32] FIG. 13 is a block diagram showing an example of a configuration of an estimation device of an estimation system according to a seventh embodiment. [Diagram 33] FIG. 13 is a conceptual diagram for explaining estimation and model updating by an estimation system according to a seventh embodiment. [Diagram 34] FIG. 13 is a conceptual diagram showing an example in which an estimation result by the estimation system according to the seventh embodiment is displayed on a screen of a mobile terminal. [Diagram 35] 23 is a flowchart for explaining an example of the operation of a measurement device in the estimation system according to the seventh embodiment. [Diagram 36] FIG. 13 is a conceptual diagram for explaining the configuration and training of a model in a validation example. [Figure 37] 13 is a table summarizing details of the model configuration in the verification example. [Figure 38] FIG. 13 is a diagram for explaining a configuration of a model in Comparative Example 1. [Figure 39] FIG. 11 is a diagram for explaining a configuration of a model in Comparative Example 2. [Diagram 40] 13 is a table summarizing the classification accuracy etc. of the verification example and comparative example 1-2. [Diagram 41] 11 is a diagram for explaining the relationship between the communication load and the classification accuracy in the verification example and the comparative example 1-2. FIG. [Diagram 42] 13 is a table for explaining reconstructed feature amounts reconstructed by a reconstructed model of a verification example. [Diagram 43]FIG. 2 is a block diagram showing an example of a hardware configuration for implementing the learning device or the estimation device according to each embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments for carrying out the present invention will be described with reference to the drawings. However, the embodiments described below are limited in a manner that is technically preferable for carrying out the present invention, but the scope of the invention is not limited to the following. In addition, in all the drawings used to explain the following embodiments, the same reference numerals are used for similar parts unless otherwise specified. In addition, in the following embodiments, repeated explanations of similar configurations and operations may be omitted.

[0015] (First embodiment) First, a learning device according to a first embodiment will be described with reference to the drawings. The learning device of this embodiment learns data collected by IoT (Internet of Things) devices and constructs a model for analyzing the data. For example, the learning device of this embodiment learns sensor data (raw data) related to foot movements and constructs a model for estimating a physical state (estimated result). The method of this embodiment can be applied to time-series data of sensor data, image analysis, etc.

[0016] (composition) 1 is a block diagram showing an example of the configuration of a learning device 10 according to this embodiment. The learning device 10 includes an acquisition unit 11, a feature amount calculation unit 12, a model construction unit 13, and a learning processing unit 15.

[0017] The acquiring unit 11 acquires a plurality of data sets (also called training data sets) used for constructing a model by the model constructing unit 13. For example, the acquiring unit 11 acquires the training data sets from a database (not shown) in which the training data sets are accumulated. The training data sets include data sets that combine raw data measured by an IoT device and supervised data corresponding to the raw data. The acquiring unit 11 acquires the raw data and supervised data that correspond to each other from the plurality of data sets included in the training data set.

[0018] The raw data may be pre-processed in some way. For example, the raw data may be pre-processed by noise removal using a low-pass filter or a high-pass filter. For example, the raw data may be pre-processed by outlier removal or missing value interpolation. For example, the raw data may be pre-processed by frequency conversion, integration, differentiation, or the like. For example, the raw data may be pre-processed by statistical processing such as averaging or variance calculation. For example, when the raw data is time-series data, the pre-processing may involve cutting out a predetermined section. For example, when the raw data is image data, the pre-processing may involve cutting out a predetermined region. The pre-processing performed on the raw data is not limited to those listed here.

[0019] For example, the data set is information combining sensor data (raw data) related to foot movement and a physical condition (correct answer data). For example, the sensor data includes sensor data such as acceleration and angular velocity. For example, the sensor data may include speed, position (trajectory), angle, etc. obtained by integrating acceleration and angular velocity. For example, the physical condition includes physical conditions such as the degree of pronation / supination of the foot, the progress of hallux valgus, and the risk of falling. There are no particular limitations on the data set acquired by the acquisition unit 11 as long as it is information combining explanatory variables (raw data) and objective variables (correct answer data).

[0020] The feature amount calculation unit 12 acquires raw data from the acquisition unit 11. The feature amount calculation unit 12 calculates feature amounts using the acquired raw data. In other words, the feature amount calculation unit 12 extracts feature amounts from the acquired raw data. For example, the feature amount calculation unit 12 calculates feature amounts that can be used to estimate a physical condition. For example, the feature amount calculation unit 12 calculates feature amounts that can be used to estimate a physical condition such as the degree of pronation / supination or the progression of hallux valgus.

[0021] The model construction unit 13 acquires raw data from the acquisition unit 11. The model construction unit 13 includes an encoding unit 131, a reconstruction unit 133, and an estimation unit 135. The model construction unit 13 constructs a group of models that use the raw data to output an estimation result. For example, the model construction unit 13 constructs a machine learning model (also called a learning model or a model) such as a deep neural network (DNN), a convolutional neural network (CNN), or a recurrent neural network (RNN). Note that the model constructed by the model construction unit 13 is not limited to a DNN, a CNN, or an RNN. The group of models constructed by the model construction unit 13 are trained by machine learning by a learning processing unit 15.

[0022] The encoding unit 131 includes an encoding model 151. The encoding unit 131 constructs the encoding model 151 that outputs a code in response to input raw data. The code includes a feature that enables estimation of correct answer data corresponding to the raw data while reducing the amount of data compared to the raw data. The number of dimensions of the code is set to be lower than that of the raw data. For example, the encoding unit 131 constructs the encoding model 151 that outputs a code in response to input raw data by unsupervised learning.

[0023] For example, the encoding model 151 outputs 10 Hz time series data (code) in response to input of time series data (raw data) measured at a cycle of 100 Hertz (Hz). For example, the encoding model 151 outputs time series data (code) in which the amount of data has been reduced by averaging or noise removal in response to input of time series data equivalent to raw data. For example, the encoding model 151 outputs 7×7 pixel image data (code) in response to input of 28×28 pixel image data (raw data). There are no limitations on the data capacity, format, etc. of the code as long as the code has a smaller data amount than the raw data and includes a feature that enables estimation of correct answer data corresponding to the raw data.

[0024] The reconstruction unit 133 includes a reconstruction model 153. The reconstruction unit 133 constructs the reconstruction model 153 that outputs a reconstruction result (also called a reconstruction feature) in response to an input of a code. The reconstruction feature is a feature of the original raw data of the code that is reconstructed based on the code. For example, the reconstruction unit 133 constructs the reconstruction model 153 that outputs a reconstructed feature in response to an input of a code by supervised learning using a data set of the code and the feature corresponding to the code. The reconstructed feature includes a feature similar to the feature corresponding to the reconstructed feature, but does not completely match the feature.

[0025] The estimation unit 135 includes an estimation model 155. The estimation unit 135 constructs an estimation model 155 that outputs an estimation result corresponding to the original raw data of a code calculated by the encoding unit 131 in response to the input of the code. For example, the estimation unit 135 constructs an estimation model 155 that outputs an estimation result in response to the input of a code by supervised learning using a data set of a code and correct answer data corresponding to the original raw data of the code. For example, the estimation unit 135 may construct an estimation model 155 that outputs an estimation result in response to the input of a code or a feature by supervised learning using a data set of a code or a feature and correct answer data corresponding to the original raw data of the code or the feature. For example, the estimation unit 135 may construct an estimation model 155 that outputs an estimation result in response to the input of a code and a feature by supervised learning using a data set of a code, a feature corresponding to the code, and correct answer data corresponding to the original raw data of the code and the feature.

[0026] The learning processing unit 15 trains, by machine learning, the group of models constructed in the model construction unit 13. That is, the learning processing unit 15 trains, by machine learning, the encoding model 151, the reconstruction model 153, and the estimation model 155 constructed in the model construction unit 13.

[0027] 2 is a conceptual diagram for explaining the training of the encoding model 151, the reconstructed model 153, and the estimated model 155 by the learning processing unit 15. In FIG. 2, the acquisition unit 11 and the learning processing unit 15 are omitted.

[0028] The learning processing unit 15 trains the coding model 151 of the coding unit 131 and the estimation model 155 of the estimation unit 135 so that the estimation result of the estimation model 155 matches the supervised data. In other words, the learning processing unit 15 optimizes the model parameters of the coding model 151 and the estimation model 155 so that the error between the estimation result of the estimation model 155 and the supervised data is minimized.

[0029] Furthermore, the learning processing unit 15 trains the encoding model 151 of the encoding unit 131 and the reconstructed model 153 of the reconstruction unit 133 so that the reconstruction result (reconstructed feature amount) of the reconstructed model 153 matches the feature amount. That is, the learning processing unit 15 optimizes the model parameters of the encoding model 151 and the reconstructed model 153 so that the error between the reconstruction result (reconstructed feature amount) of the reconstructed model 153 and the feature amount is minimized.

[0030] In this embodiment, the encoding model 151 is regularized by training the encoding model 151 and the reconstruction model 153 so that the error between the reconstruction feature and the feature is minimized. As a result, it is possible to avoid a decrease in versatility due to erroneous overlearning. For example, when estimating the progress of hallux valgus, a symptom common in women, using sensor data related to foot movement, the feature included in the code may contain more female features than hallux valgus features. In such a case, although it was intended to construct an estimation model 155 for estimating the progress of hallux valgus, there is a possibility that an estimation model 155 for estimating whether or not the person is female is constructed. According to this embodiment, erroneous overlearning can be prevented by regularizing the encoding model 151, and therefore versatility and accuracy of the estimation model 155 are improved.

[0031] The model trained by the learning processing unit 15 is implemented in an estimation system (not shown) that performs estimation based on raw data. For example, the estimation system includes a measurement device (not shown) that measures measurement data (raw data) and an estimation device (not shown) that performs estimation based on the measurement data. For example, the encoding model is implemented in the measurement device, and the estimation model is implemented in the estimation device. The measurement device encodes the measurement data using the encoding model and transmits the code to the estimation model. The estimation device inputs the code received from the measurement device to the estimation model and outputs an estimation result by the estimation model. Details of the estimation system that uses the model trained by the learning processing unit 15 will be described later.

[0032] (operation) Next, the operation of the learning device 10 of this embodiment will be described with reference to the drawings. Fig. 3 is a flowchart for explaining an example of the operation of the learning device 10. In the explanation following the flowchart of Fig. 3, the components of the learning device 10 will be mainly described.

[0033] In FIG. 3, first, the acquisition unit 11 acquires raw data and correct answer data from a training data set (step S11).

[0034] Next, the feature amount calculation unit 12 calculates the feature amount using the raw data (step S12).

[0035] Next, the encoding unit 131 inputs the raw data to the encoding model 151 and calculates a code (step S13).

[0036] Next, the estimation unit 135 inputs the code to the estimation model 155 and calculates the estimation result (step S14).

[0037] Next, the reconstruction unit 133 inputs the code to the reconstruction model 153 and calculates the reconstruction result (reconstruction feature) (step S15). Steps S14 and S15 may be executed in reverse order or in parallel.

[0038] Next, the learning processing unit 15 executes a training process (step S16). The details of the training process will be described later. The model parameters of the model trained by the learning processing unit 15 are set in the encoding model and the estimation model implemented in the estimation system (not shown).

[0039] [Training process] Next, the training process (step S16 in FIG. 3) by the learning processing unit 15 will be described with reference to the drawings. FIG. 4 is a flowchart for explaining the training process by the learning processing unit 15. In the process according to the flowchart in FIG. 4, the learning processing unit 15 will be described as the subject of the operation.

[0040] In FIG. 4, first, the learning processing unit 15 trains the encoding model 151 of the encoding unit 131 and the estimation model 155 of the estimation unit 135 so that the estimation result using the estimation model 155 matches the correct data (step S111).

[0041] Next, the learning processing unit 15 trains the encoding model 151 of the encoding unit 131 and the reconstruction model 153 of the reconstruction unit 133 so that the reconstruction result (reconstructed feature) using the reconstruction model 153 matches the corresponding feature (step S112). The model parameters of the models trained by the learning processing unit 15 are set in the encoding model and the estimation model implemented in an estimation system (not shown).

[0042] [Variation 1] Next, a first modified example of the present embodiment will be described with reference to the drawings. In this modified example, the learning processing unit 15 trains the estimation model 155 of the estimation unit 135 using the feature calculated by the feature calculation unit 12. The learning processing unit 15 may be configured to train the estimation model 155 of the estimation unit 135 using the reconstructed feature output from the reconstruction model 153. The reconstructed feature and the feature have the same dimension. An estimation result can be obtained whether the reconstructed feature is input to the estimation model 155 trained using the feature, or whether the feature is input to the estimation model 155 trained using the reconstructed feature.

[0043] Fig. 5 is a conceptual diagram for explaining the training of the encoding model 151, the reconstructed model 153, and the estimated model 155 by the learning processing unit 15 of this modification. In Fig. 5, the acquisition unit 11 and the learning processing unit 15 are omitted.

[0044] In this modification, the feature calculated by the feature calculation unit 12 is input to the estimation unit 135. The estimation unit 135 constructs an estimation model 155 using a data set in which a code calculated from raw data and the feature calculated from the raw data are explanatory variables, and an estimation result associated with the raw data is a response variable. The estimation unit 135 may construct the estimation model 155 using a data set in which a code calculated from raw data and a feature reconstructed by inputting the code to the reconstruction model 153 are explanatory variables, and an estimation result associated with the raw data is a response variable.

[0045] The learning processing unit 15 of this modification trains the coding model 151 of the coding unit 131 and the estimation model 155 of the estimation unit 135 so that the estimation result of the estimation model 155 matches the supervised data. That is, the learning processing unit 15 of this modification optimizes the model parameters of the coding model 151 and the estimation model 155 so that the error between the estimation result of the estimation model 155 and the supervised data is minimized.

[0046] Furthermore, the learning processing unit 15 of this modification trains the encoding model 151 of the encoding unit 131 and the reconstructed model 153 of the reconstruction unit 133 so that the reconstruction result (reconstructed feature amount) of the reconstructed model 153 matches the feature amount. That is, the learning processing unit 15 optimizes the model parameters of the encoding model 151 and the reconstructed model 153 so that the error between the reconstruction result (reconstructed feature amount) of the reconstructed model 153 and the feature amount is minimized.

[0047] In this modification, in addition to the code, the estimation is performed using the feature of the original raw data of the code. Therefore, according to this modification, the amount of information used for estimation increases, and the accuracy of the estimation is improved. In addition to the code, this modification may be further modified to perform estimation using a reconstructed feature reconstructed by the reconstruction model 153 using the code. When the method of this modification is applied to an estimation system (not shown), a communication load is generated for transmitting the feature, but the estimation accuracy is improved compared to the method of FIG. 2. In this modification, an example is shown in which estimation is performed using the feature calculated by the feature calculation unit 12 and the code calculated using the encoding model 151. This modification may be configured to perform estimation using the reconstructed feature reconstructed using the reconstruction model 153 and the code calculated using the encoding model 151.

[0048] As described above, the learning device according to the present embodiment includes an acquisition unit, a feature calculation unit, a model construction unit, and a learning processing unit. The acquisition unit acquires a data set of raw data and correct answer data. The feature calculation unit calculates a feature using the raw data. The model construction unit constructs an encoding model that outputs a code corresponding to the feature in response to input of raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input of the code. In addition, the model construction unit constructs a reconstruction model that outputs a reconstructed feature corresponding to the original feature of the code in response to input of the code. The learning processing unit trains the encoding model and the reconstruction model so that the reconstructed feature matches the feature. In addition, the learning processing unit trains the encoding model and the estimation model so that the estimation result matches the correct answer data based on the relationship between the code and the feature.

[0049] The learning device of this embodiment trains an encoding model and an estimation model based on the relationship between the code and the feature amount so that the estimation result matches the supervised data. Therefore, the learning device of this embodiment can construct an encoding model that outputs a code that maximizes the learning effect of the estimation model. In addition, the learning device of this embodiment trains an encoding model and a reconstruction model so that the reconstruction result (reconstructed feature amount) by the reconstruction model matches the feature amount. Therefore, the learning device of this embodiment can construct an encoding model that outputs a code that correctly captures the causal relationship between raw data and supervised data. That is, according to this embodiment, a model that correctly captures the causal relationship and maximizes the learning effect can be constructed.

[0050] In one aspect of the present embodiment, the model construction unit constructs an estimation model using a code output from the encoding model in response to input of raw data and a feature value calculated using the raw data. According to this aspect, by using the feature value in addition to the code, it is possible to perform estimation with higher accuracy.

[0051] In one aspect of the present embodiment, the model construction unit constructs an estimation model using a reconstructed feature and a code output from the reconstruction model in response to an input of a code. According to this aspect, by using the reconstructed feature in addition to the code, it is possible to perform estimation with higher accuracy.

[0052] In general, feature calculation is a complex algorithm and is written in a high-level language. Therefore, every time a new algorithm is developed, it takes time to port the program to the IoT device. For the coding model according to this embodiment, a machine learning model such as DNN, CNN, RNN, etc. can be used. Therefore, in the method of this embodiment, if the IoT device has an environment for executing the machine learning model, it is sufficient to simply transfer the model. Specifically, in the method of this embodiment, it is sufficient to simply transfer model parameters indicating the functional configuration of the model, such as the number of layers of the NN, the number of units in each layer, the activation function, and the connection weight, to the estimation system. Therefore, according to this embodiment, it is possible to eliminate the need to port programs to IoT devices one by one every time a new algorithm is developed.

[0053] A learning model trained by general machine learning can capture the correlation between raw data and correct data, but cannot capture the causality between raw data and correct data. For example, assume that a learning model that discriminates hallux valgus is trained by machine learning using sensor data (gait data) related to foot movement. Hallux valgus tends to be more prevalent in women. Therefore, a learning model trained by machine learning may extract the characteristics of a woman's gait from the gait data. In other words, a learning model trained on hallux valgus may be trained to discriminate whether or not a person is female based on gait data. That is, if the relationship between the feature amount and the code is not based on the code, the code output from the encoding model will include information for discriminating whether or not a person is female. However, there is no causal relationship between hallux valgus and being female. Therefore, if such a model is applied to a man, the estimation accuracy in discriminating hallux valgus may decrease.

[0054] In general methods, the coding model and the estimation model are designed separately, so it is not possible to evaluate whether the estimation result matches the ground truth data at the feature design stage. Therefore, simply using features as codes does not necessarily maximize the learning effect.

[0055] In this embodiment, a feature based on knowledge that correctly captures causal relationships is introduced into a method that utilizes machine learning. The learning device of this embodiment trains an encoding model and an estimation model based on the relationship between the code and the feature so that the estimation result matches the correct data. As a result, the feature is reflected in the code input to the estimation model, and the code contains features based on the causal relationship between the raw data and the estimation result. In other words, the code output from the encoding model constructed by the learning device of this embodiment contains features based on knowledge. According to this embodiment, the power of data and the power of knowledge can be combined by using knowledge as a constraint condition for code generation.

[0056] For example, based on medical knowledge, pronation of the subtalar joint during walking makes it easier for hallux valgus to be induced. In other words, there is a causal relationship between the pronation of the ankle and hallux valgus. Therefore, if the gait feature of pronation / supination is treated as a code and an estimation model is constructed to estimate hallux valgus using the code, hallux valgus can be estimated regardless of gender.

[0057] Second Embodiment Next, a learning device according to a second embodiment will be described with reference to the drawings. The learning device according to this embodiment differs from the first embodiment in that it trains an encoding model and an estimation model based on the mutual information between a code calculated from raw data and a feature quantity extracted from the raw data, without using a reconstruction model.

[0058] (composition) 6 is a block diagram showing an example of the configuration of the learning device 20 of this embodiment. The learning device 20 includes an acquisition unit 21, a feature amount calculation unit 22, a model construction unit 23, and a learning processing unit 25.

[0059] The acquisition unit 21 acquires a plurality of data sets (also called training data sets) used for constructing a model by the model construction unit 23. The acquisition unit 21 has a similar configuration to the acquisition unit 11 of the first embodiment. The acquisition unit 21 acquires raw data and supervised data corresponding to each other from a plurality of data sets included in the training data sets.

[0060] The feature amount calculation unit 22 acquires raw data from the acquisition unit 21. The feature amount calculation unit 22 has a similar configuration to the feature amount calculation unit 12 of the first embodiment. The feature amount calculation unit 22 calculates feature amounts using the acquired raw data.

[0061] The model construction unit 23 acquires raw data from the acquisition unit 21. The model construction unit 23 has an encoding unit 231 and an estimation unit 235. The model construction unit 23 constructs a group of models that use the raw data to output an estimation result. For example, the model construction unit 23 constructs a machine learning model such as DNN, CNN, or RNN. Note that the models constructed by the model construction unit 23 are not limited to DNN, CNN, or RNN. The group of models constructed by the model construction unit 23 are trained by machine learning by the learning processing unit 25.

[0062] The encoding unit 231 includes an encoding model 251. The encoding unit 231 has a similar configuration to the encoding unit 131 of the first embodiment. The encoding model 251 has a similar configuration to the encoding model 151 of the first embodiment. The encoding unit 231 constructs the encoding model 251 that outputs a code in response to input of raw data.

[0063] The estimation unit 235 includes an estimation model 255. In response to input of a code calculated by the encoding unit 231 and a feature of the original raw data of the code, the estimation unit 235 constructs an estimation model 255 that outputs an estimation result corresponding to the raw data. For example, the estimation unit 235 constructs the estimation model 255 that outputs an estimation result in response to input of a code and a feature by supervised learning using a data set of a code, a feature of the original raw data of the code, and ground truth data corresponding to the raw data.

[0064] The learning processing unit 25 trains, by machine learning, the group of models constructed in the model construction unit 23. That is, the learning processing unit 25 trains, by machine learning, the encoding model 251 and the estimation model 255 constructed in the model construction unit 23.

[0065] Fig. 7 is a conceptual diagram for explaining the training of the encoding model 251 and the estimation model 255 by the learning processing unit 25. In Fig. 7, the acquisition unit 21 and the learning processing unit 25 are omitted.

[0066] The learning processing unit 25 trains the coding model 251 of the coding unit 231 and the estimation model 255 of the estimation unit 235 so that the estimation result of the estimation model 255 matches the supervised data. That is, the learning processing unit 25 optimizes the model parameters of the coding model 251 and the estimation model 255 so that the error between the estimation result of the estimation model 255 and the supervised data is minimized.

[0067] Further, the learning processing unit 25 calculates the mutual information between the code output from the coding model 251 and the feature calculated from the original raw data of the code. The learning processing unit 25 trains the coding model 251 of the coding unit 231 and the estimation model 255 of the estimation unit 235 so as to reduce the mutual information between the code and the feature calculated from the original raw data of the code. That is, the learning processing unit 25 optimizes the model parameters of the coding model 251 and the estimation model 255 so as to reduce the mutual information between the code and the feature calculated from the original raw data of the code. For example, when the random variable of the code is X and the random variable of the feature is Y, the learning processing unit 25 calculates the mutual information I using the following formula. TIFF0007670130000001.tif10150

[0068] In the above formula 1, p(x, y) represents the joint distribution of random variables X and Y. In addition, p(x) is a probability density function representing the marginal distribution of random variable X, and p(y) is a probability density function representing the marginal distribution of random variable Y.

[0069] In this embodiment, the encoding model 251 and the estimation model 255 are trained so that the amount of mutual information between a code and a feature calculated from the original raw data of the code is reduced. As a result, the encoding model 251 encodes information that complements the feature. As a result, according to this embodiment, it is possible to reduce the load on computation resources such as computation time, memory, and power compared to the first embodiment.

[0070] The model trained by the learning processing unit 25 is implemented in an estimation system (not shown) that performs estimation based on raw data. For example, the estimation system includes a measurement device (not shown) that measures measurement data (raw data) and an estimation device (not shown) that performs estimation based on the measurement data. For example, the encoding model is implemented in the measurement device, and the estimation model is implemented in the estimation device. The measurement device encodes the measurement data using the encoding model and calculates the feature amount of the measurement data. The measurement device transmits the code and the feature amount of the raw data that is the source of the code to the estimation model. The estimation device inputs the code and the feature amount received from the measurement device to the estimation model, and outputs the estimation result by the estimation model. The estimation system that uses the model trained by the learning processing unit 25 will be described in detail later.

[0071] (operation) Next, the operation of the learning device 20 of this embodiment will be described with reference to the drawings. Fig. 8 is a flowchart for explaining an example of the operation of the learning device 20. In the explanation following the flowchart of Fig. 8, the components of the learning device 20 will be mainly described.

[0072] In FIG. 8, first, the acquisition unit 21 acquires raw data and correct answer data from a training data set (step S21).

[0073] Next, the feature amount calculation unit 22 calculates the feature amount using the raw data (step S22).

[0074] Next, the encoding unit 231 inputs the raw data into the encoding model 251 and calculates the code (step S23).

[0075] Next, the estimation unit 235 inputs the code and the feature amount of the original raw data of the code to the estimation model 255, and calculates the estimation result (step S24).

[0076] Next, the learning processing unit 25 executes a training process (step S25). The details of the training process will be described later. The model parameters of the model trained by the learning processing unit 25 are set in the encoding model and the estimation model implemented in the estimation system (not shown).

[0077] [Training process] Next, the training process (step S25 in FIG. 8) by the learning processing unit 25 will be described with reference to the drawings. Fig. 9 is a flowchart for explaining the training process by the learning processing unit 25. In the process according to the flowchart in Fig. 9, the learning processing unit 25 will be described as the subject of the operation.

[0078] In FIG. 9, first, the learning processing unit 25 trains the encoding model 251 of the encoding unit 231 and the estimation model 255 of the estimation unit 235 so that the estimation result using the estimation model 255 matches the correct data (step S211).

[0079] Next, the learning processing unit 25 trains the encoding model 251 of the encoding unit 231 and the estimation model 255 of the estimation unit 235 so as to reduce the amount of mutual information between the code and the feature (step S212). The model parameters of the models trained by the learning processing unit 25 are set in the encoding model and the estimation model implemented in the estimation system (not shown).

[0080] [Modification 2] Next, a second modification of the present embodiment will be described with reference to the drawings. In this modification, the learning processing unit 25 does not calculate the mutual information between the code and the feature.

[0081] Fig. 10 is a conceptual diagram for explaining the training of the encoding model 251 and the estimation model 255 by the learning processing unit 25 of this modified example. In Fig. 10, the acquisition unit 21 and the learning processing unit 25 are omitted.

[0082] In this modification, the features calculated by the feature calculation unit 22 are input to the estimation unit 235. The estimation unit 235 constructs an estimation model 255 using a data set in which a code calculated from raw data and the feature calculated from the raw data are used as explanatory variables, and an estimation result associated with the raw data is used as a response variable.

[0083] The learning processing unit 25 of this modification trains the encoding model 251 of the encoding unit 231 and the estimation model 255 of the estimation unit 235 so that the estimation result of the estimation model 255 matches the supervised data. That is, the learning processing unit 25 of this modification optimizes the model parameters of the encoding model 251 and the estimation model 255 so that the error between the estimation result of the estimation model 255 and the supervised data is minimized.

[0084] In this modified example, there is a possibility that there will be a lot of overlap between the code and feature information, but since the mutual information between the code and feature information is not calculated, the computational resources can be simplified compared to the method in FIG.

[0085] As described above, the learning device according to this embodiment includes an acquisition unit, a feature calculation unit, a model construction unit, and a learning processing unit. The acquisition unit acquires a data set of raw data and supervised data. The feature calculation unit calculates features using the raw data. The model construction unit constructs an encoding model that outputs a code corresponding to the feature in response to input of raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input of the code. The learning processing unit trains the encoding model and the estimation model based on the relationship between the code and the feature so that the estimation result matches the supervised data.

[0086] The learning device of this embodiment trains the encoding model and the estimation model so that the estimation result of the estimation model matches the ground truth data. Therefore, according to this embodiment, the encoding unit can generate a code that maximizes the learning effect.

[0087] In one aspect of this embodiment, the learning processing unit trains the estimation model so that the mutual information between the code output from the encoding model in response to the input of raw data and the feature calculated using the raw data is reduced. According to this aspect, the encoding model is trained so that the mutual information between the code and the feature is reduced, so that the code generated by the encoding unit does not overlap with the information contained in the feature. That is, according to the method of this embodiment, by combining the code and the feature that are complementary to each other, it is possible to maximize the learning effect while correctly grasping the causal relationship. As a result, according to this embodiment, like the first embodiment, it is possible to construct a model that maximizes the learning effect while correctly grasping the causal relationship.

[0088] In one aspect of the present embodiment, the model construction unit constructs an estimation model using a code output from the encoding model in response to input of raw data and a feature value calculated using the raw data. According to this aspect, when making an estimation using the estimation model, the feature value is used in addition to the code, thereby making it possible to perform estimation with higher accuracy.

[0089] In the method of the first embodiment, it is necessary to increase the scale or complexity of the machine learning model according to the complexity of the algorithm for calculating the feature. This is because, if the degree of freedom of the machine learning model is not high enough to accommodate the complexity of the algorithm for calculating the feature, the feature cannot be reconstructed and performance will be degraded. If the machine learning model becomes large-scale / complex, the load on the computational resources such as time, memory, and power may increase. In contrast, in the method of the present embodiment, the encoding model encodes only the information that complements the feature, so a model that is small-scale and simpler than that of the first embodiment is sufficient. Therefore, according to the present embodiment, the load on the computational resources can be reduced compared to the first embodiment.

[0090] (Third embodiment) Next, a learning device according to a third embodiment will be described with reference to the drawings. The learning device according to the third embodiment is different from the first and second embodiments in that the learning device combines and fine-tunes an encoding model and an estimation model that are individually pre-trained.

[0091] (composition) 11 is a block diagram showing an example of the configuration of a learning device 30 of this embodiment. The learning device 30 includes an acquisition unit 31, a feature amount calculation unit 32, a model construction unit 33, and a learning processing unit .

[0092] The acquisition unit 31 acquires a plurality of data sets (also called training data sets) used for constructing a model by the model construction unit 33. The acquisition unit 31 has a similar configuration to the acquisition unit 11 of the first embodiment. The acquisition unit 31 acquires raw data and supervised data corresponding to each other from a plurality of data sets included in the training data sets.

[0093] The feature amount calculation unit 32 acquires raw data from the acquisition unit 31. The feature amount calculation unit 32 has a similar configuration to the feature amount calculation unit 12 of the first embodiment. The feature amount calculation unit 32 calculates feature amounts using the acquired raw data.

[0094] The model construction unit 33 acquires raw data from the acquisition unit 31. The model construction unit 33 includes an encoding unit 331, an estimation unit 335, a first pre-learning unit 337, and a second pre-learning unit 338. The model construction unit 33 constructs a group of models that use the raw data to output an estimation result. For example, the model construction unit 33 constructs a machine learning model such as DNN, CNN, or RNN. Note that the model constructed by the model construction unit 33 is not limited to DNN, CNN, or RNN. The group of models constructed by the model construction unit 33 are trained by machine learning by the learning processing unit 35.

[0095] The encoding unit 331 includes an encoding model 351. The encoding unit 331 has a similar configuration to the encoding unit 131 of the first embodiment. The encoding model 351 has a similar configuration to the encoding model 151 of the first embodiment. The encoding unit 331 differs from the encoding unit 131 of the first embodiment in that model parameters of a first pre-trained model 357 constructed in a first pre-training unit 337 are set in the encoding model 351. The encoding unit 331 constructs the encoding model 351 that outputs a code in response to input of raw data.

[0096] The estimation unit 335 includes an estimation model 355. The estimation unit 335 has a configuration similar to that of the estimation unit 135 of the first embodiment. The estimation unit 335 differs from the encoding unit 131 of the first embodiment in that model parameters of a second pre-trained model 358 constructed in a second pre-training unit 338 are set in the estimation model 355. In response to input of a code calculated by the encoding unit 331, the estimation unit 335 constructs an estimation model 355 that outputs an estimation result corresponding to the original raw data of the code.

[0097] The first pre-learning unit 337 includes a first pre-trained model 357. The first pre-learning unit 337 constructs the first pre-trained model 357 that outputs a first conversion value in response to input of raw data. The first conversion value is an estimated value corresponding to a feature amount calculated from the raw data. For example, the first pre-learning unit 337 constructs the first pre-trained model 357 that outputs a first conversion value in response to input of raw data by supervised learning using a data set of raw data and a feature amount corresponding to the raw data. The first conversion value includes a feature similar to the feature amount corresponding to the first conversion value.

[0098] The second pre-learning unit 338 includes a second pre-trained model 358. The second pre-learning unit 338 constructs a second pre-trained model 358 that outputs a second conversion value in response to an input of a feature. The second conversion value is an estimated value corresponding to correct answer data that is paired with the original raw data of the feature. For example, the second pre-learning unit 338 constructs a second pre-trained model 358 that outputs a second conversion value in response to an input of a feature by supervised learning using a dataset of the feature and correct answer data that is paired with the original raw data of the feature. For example, the second pre-learning unit 338 may construct a second pre-trained model 358 that outputs a second conversion value in response to an input of a reconstructed feature by supervised learning using a dataset of the reconstructed feature and correct answer data that is paired with the original raw data of the reconstructed feature. The second conversion value includes features similar to the correct answer data corresponding to the second conversion value.

[0099] The learning processing unit 35 trains, by machine learning, the group of models constructed in the model construction unit 33. That is, the learning processing unit 35 trains, by machine learning, the encoding model 351, the estimation model 355, the first pre-trained model 357, and the second pre-trained model 358 constructed in the model construction unit 33.

[0100] Fig. 12 is a conceptual diagram for explaining the training of the encoding model 351, the estimation model 355, the first pre-trained model 357, and the second pre-trained model 358 by the learning processing unit 35. In Fig. 12, the acquisition unit 31 and the learning processing unit 35 are omitted.

[0101] The learning processing unit 35 trains the first pre-trained model 357 of the first pre-training unit 337 so that the first conversion value, which is the output of the first pre-trained model 357, matches the feature amount calculated from the original raw data of the first conversion value. In other words, the learning processing unit 35 optimizes the model parameters of the first pre-trained model 357 so that the error between the first conversion value and the feature amount calculated from the original raw data of the first conversion value is minimized.

[0102] The learning processing unit 35 trains the second pre-trained model 358 of the second pre-training unit 338 so that the second converted value, which is the output of the second pre-trained model 358, matches the correct answer data corresponding to the original raw data of the second converted value. In other words, the learning processing unit 35 optimizes the model parameters of the second pre-trained model 358 so that the error between the second converted value and the correct answer data corresponding to the original raw data of the second converted value is minimized.

[0103] The learning processing unit 35 sets the model parameters of the first pre-trained model 357 optimized by training to the encoding model 351. The learning processing unit 35 also sets the model parameters of the second pre-trained model 358 optimized by training to the estimation model 355.

[0104] The learning processing unit 35 trains the encoding model 351 of the encoding unit 331 and the estimation model 355 of the estimation unit 335 so that the estimation result of the estimation model 355 matches the supervised data. In other words, the learning processing unit 35 optimizes the model parameters of the encoding model 351 and the estimation model 355 so that the error between the estimation result of the estimation model 355 and the supervised data is minimized.

[0105] In this embodiment, the model parameters of the first pre-learning model, which is trained so that the first conversion value calculated using raw data matches the feature, are set as the initial values ​​of the encoding model 351. Therefore, the nature of the feature remains in the encoding model 351, and it is possible to generate a code that correctly captures the causal relationship between the raw data and the correct answer data. The encoding model 351 and the estimation model 355 are trained so that the estimation result matches the correct answer data. Therefore, the encoding model 351 can output a code with a maximized learning effect.

[0106] The model trained by the learning processing unit 35 is implemented in an estimation system (not shown) that performs estimation based on raw data. For example, the estimation system includes a measurement device (not shown) that measures measurement data (raw data) and an estimation device (not shown) that performs estimation based on the measurement data. For example, the encoding model is implemented in the measurement device, and the estimation model is implemented in the estimation device. The measurement device encodes the measurement data using the encoding model and calculates the feature amount of the measurement data. The measurement device transmits the code and the feature amount of the raw data that is the source of the code to the estimation model. The estimation device inputs the code and the feature amount received from the measurement device to the estimation model and outputs the estimation result by the estimation model. The estimation system that uses the model trained by the learning processing unit 35 will be described in detail later.

[0107] (operation) Next, the operation of the learning device 30 of this embodiment will be described with reference to the drawings. Fig. 13 is a flowchart for explaining an example of the operation of the learning device 30. In the explanation following the flowchart of Fig. 13, the components of the learning device 30 will be mainly described.

[0108] In FIG. 13, first, the acquisition unit 31 acquires raw data and correct answer data from a training data set (step S31).

[0109] Next, the feature amount calculation unit 32 calculates the feature amount using the raw data (step S32).

[0110] Next, the encoding unit 331 inputs the raw data into the encoding model 351 and calculates the code (step S33).

[0111] Next, the estimation unit 335 inputs the code to the estimation model 355 and calculates the estimation result (step S34).

[0112] Next, the learning processing unit 35 executes a training process (step S35). The details of the training process will be described later. The model parameters of the model trained by the learning processing unit 35 are set in the encoding model and the estimation model implemented in the estimation system (not shown).

[0113] [Training process] Next, the training process (step S35 in FIG. 13) by the learning processing unit 35 will be described with reference to the drawings. Fig. 14 is a flowchart for explaining the training process by the learning processing unit 35. In the process according to the flowchart in Fig. 14, the learning processing unit 35 will be described as the subject of operations.

[0114] In FIG. 14, first, the learning processing unit 35 inputs raw data into the first pre-trained model 357 to calculate a first conversion value (step S311).

[0115] Next, the learning processing unit 35 trains the first pre-trained model 357 so that the first conversion amount matches the feature amount (step S312).

[0116] Next, the learning processing unit 35 sets the model parameters of the first pre-trained model 357 in the encoding model 351 (step S313).

[0117] Next, the learning processing unit 35 inputs the feature amount to the second pre-trained model 358 and calculates the second conversion value (step S314).

[0118] Next, the learning processing unit 35 trains the second pre-trained model 358 so that the second conversion amount matches the correct data (step S315).

[0119] Next, the learning processing unit 35 sets the model parameters of the second pre-trained model 358 in the estimation model 355 (step S316). Steps S314 to S316 may be performed before steps S311 to S313, or may be performed in parallel with steps S311 to S313.

[0120] Next, the learning processing unit 35 trains the encoding model 351 of the encoding unit 331 and the estimation model 355 of the estimation unit 335 so that the estimation result of the estimation model 355 matches the correct data (step S317).

[0121] As described above, the learning device according to the present embodiment includes an acquisition unit, a feature calculation unit, a model construction unit, and a learning processing unit. The acquisition unit acquires a data set of raw data and correct answer data. The feature calculation unit calculates a feature using the raw data. The model construction unit constructs an encoding model that outputs a code corresponding to the feature in response to input of raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input of the code. In addition, the model construction unit constructs a first pre-learning model that outputs a first conversion value corresponding to the feature in response to input of raw data. The learning processing unit trains the first pre-learning model so that the first conversion value matches the feature. The learning processing unit sets the model parameters of the first pre-learning model after training as initial values ​​of the encoding model. The learning processing unit trains the encoding model and the estimation model so that the estimation result matches the correct answer data.

[0122] In this embodiment, the model parameters of the first pre-learning model, which is trained so that the first conversion value calculated using raw data matches the feature, are used as the initial values ​​of the model parameters of the encoding model. Therefore, according to this embodiment, it is possible to construct an encoding model that outputs a code that includes the nature of the feature and correctly captures the causal relationship. That is, according to this aspect, as in the first and second embodiments, it is possible to construct a model that performs high-precision estimation while correctly capturing the causal relationship by using a code that reduces the dimension of raw data.

[0123] In addition, in this embodiment, the model parameters of the second pre-learning model, which is trained so that the second conversion value calculated using the feature amount matches the correct data, are used as the initial values ​​of the model parameters of the estimation model. Therefore, according to this embodiment, the correspondence between the feature amount and the correct data is learned in advance, so that the fine tuning of the encoding model and the estimation model can be converged early.

[0124] (Fourth embodiment) Next, a learning device according to a fourth embodiment will be described with reference to the drawings. The learning device of this embodiment has a simplified configuration of the learning devices of the first to third embodiments.

[0125] (composition) 15 is a block diagram showing an example of the configuration of a learning device 40 of this embodiment. The learning device 40 includes an acquisition unit 41, a feature amount calculation unit 42, a model construction unit 43, and a learning processing unit 45. The model construction unit 43 includes an encoding model 451 and an estimation model 455.

[0126] The acquisition unit 41 acquires a data set of raw data and supervised data. The feature amount calculation unit 42 calculates features using the raw data. The model construction unit 43 constructs an encoding model that outputs a code corresponding to the feature amount in response to the input of raw data. The model construction unit 43 also constructs an estimation model that outputs an estimation result corresponding to the raw data in response to the input of a code. The learning processing unit 45 trains the encoding model 451 and the estimation model 455 based on the relationship between the code and the feature amount so that the estimation result matches the supervised data.

[0127] According to the learning device of this embodiment, by training an encoding model and an estimation model based on the relationship between codes and features, it is possible to construct a model that performs highly accurate estimation while correctly capturing causal relationships using codes that have been reduced in dimension from raw data.

[0128] Fifth embodiment Next, an estimation system according to a fifth embodiment will be described with reference to the drawings. The estimation system according to this embodiment includes an encoding model and an estimation model constructed by the learning device according to the first or third embodiment. The estimation system according to this embodiment measures physical quantities (sensor data) related to foot movement by a measurement device installed in footwear worn by a user. For example, the physical quantities related to foot movement include acceleration in three axial directions (also called spatial acceleration) measured by an acceleration sensor and angular velocity around three axes (also called spatial angular velocity) measured by an angular velocity sensor. The estimation system according to this embodiment estimates the gait and physical state of the user based on the measured sensor data.

[0129] (composition) 16 is a block diagram showing an example of the configuration of an estimation system 50 of this embodiment. The estimation system 50 includes a measurement device 51 and an estimation device 53. The measurement device 51 and the estimation device 53 may be connected by wire or wirelessly. Furthermore, the measurement device 51 and the estimation device 53 may be configured as a single device.

[0130] The measuring device 51 is installed on the foot. For example, the measuring device 51 is installed on footwear such as shoes. In this embodiment, an example in which the measuring device 51 is placed at the back side of the arch of the foot will be described. The measuring device 51 has an inertial measurement device including, for example, an acceleration sensor and an angular velocity sensor. An example of an inertial measurement device is an IMU (Inertial Measurement Unit). The IMU includes an acceleration sensor that measures acceleration in three axial directions and an angular velocity sensor that measures angular velocity around three axes. The measuring device 51 may be realized by an inertial measurement device such as a VG (Vertical Gyro) or an AHRS (Attitude Heading). The measuring device 51 may be realized by a GPS / INS (Global Positioning System / Inertial Navigation System). The measuring device 51 has a microcomputer or microcontroller that performs control processing and arithmetic processing of the measuring device 51, and a real-time clock (hereinafter also referred to as a clock) that keeps time. The microcomputer or microcontroller is also called a control unit (controller).

[0131] FIG. 17 is a conceptual diagram showing an example of arranging the measuring device 51 in the shoe 500. In the example of FIG. 17, the measuring device 51 is arranged at a position that hits the back side of the arch of the foot. For example, the measuring device 51 is arranged in an insole inserted into the shoe 500. For example, the measuring device 51 is arranged on the bottom surface of the shoe 500. For example, the measuring device 51 is embedded in the body of the shoe 500. The measuring device 51 may be detachable from the shoe 500, or may not be detachable from the shoe 500. Note that the measuring device 51 may be arranged at a position other than the back side of the arch of the foot, as long as it can acquire sensor data related to the movement of the foot. The measuring device 51 may also be arranged in socks worn by the user or in an accessory such as an anklet worn by the user. The measuring device 51 may also be directly attached to the foot or embedded in the foot. FIG. 17 shows an example in which the measuring device 51 is arranged in the shoes 500 of both the left and right feet, but the measuring device 51 may also be arranged in the shoe 500 of one foot.

[0132] The measuring device 51 includes an acceleration sensor and an angular velocity sensor. The measuring device 51 measures physical quantities related to the movement of the foot of a user wearing the footwear, such as acceleration in three axial directions (also called spatial acceleration) measured by an acceleration sensor and angular velocity around three axes (also called spatial angular velocity) measured by an angular velocity sensor. The physical quantities related to the movement of the foot measured by the measuring device 51 include velocity and position (trajectory) in three axial directions and angles around three axes, which are calculated by integrating the acceleration and angular velocity. The measuring device 51 converts the measured analog data (physical quantities) into digital data (also called sensor data).

[0133] An encoding model constructed by the learning device of the first or third embodiment is implemented in the measurement device 51. For example, model parameters set by the learning device of the first or third embodiment are set in the encoding model. The measurement device 51 inputs the converted sensor data to the encoding model and encodes it.

[0134] The measurement device 51 transmits the encoded sensor data (code) to the estimation device 53. For example, the measurement device 51 is connected to the estimation device 53 via a mobile terminal (not shown) carried by a user. When communication between the measurement device 51 and the mobile terminal is successful and a code is transmitted from the measurement device 51 to the mobile terminal, the measurement in that measurement time period is completed. For example, when communication between the measurement device 51 and the mobile terminal is successful, the time of the clock of the measurement device 51 may be synchronized with the time of the mobile terminal. When communication between the measurement device 51 and the mobile terminal is unsuccessful and a code is not transmitted from the measurement device 51 to the mobile terminal, the code in that measurement time period may be retransmitted in the next or subsequent measurement time period. For example, when communication between the measurement device 51 and the mobile terminal is unsuccessful, the transmission of the code in that measurement time period may be repeated until the communication is successful. For example, when communication between the measurement device 51 and the mobile terminal is unsuccessful, the transmission of the code in that measurement time period may be repeated within a predetermined time. The code for the measurement time period in which the transmission failed may be stored in a storage device (not shown) such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) until the next transmission timing.

[0135] When the measuring devices 51 are mounted on both the left and right feet, the times of the measuring devices 51 mounted on both feet can be synchronized by synchronizing the clock times of the measuring devices 51 with the time of the mobile terminal. The measuring devices 51 mounted on both feet may perform measurements at the same timing, or may perform measurements at different timings. For example, if the timing of measurements by the measuring devices 51 mounted on both feet is significantly different based on the measurement time of both feet or the number of measurement failures, a correction may be made to reduce the difference in the measurement timing. The correction of the measurement timing may be made in the estimation device 53 capable of processing the code transmitted from the measuring devices 51 installed on both feet, or in a higher-level system.

[0136] A mobile terminal (not shown) connected to the measurement device 51 is realized by a communication device that can be carried by a user. For example, the mobile terminal is a mobile communication device having a communication function such as a smartphone, a smart watch, or a mobile phone. The mobile terminal receives sensor data related to the movement of the user's feet from the measurement device 51. The mobile terminal transmits the received code to a cloud or a server in which the estimation device 53 is implemented. Note that the function of the estimation device 53 may be realized by application software or the like (also called an app) installed on the mobile terminal. In this case, the mobile terminal processes the received code using the app installed on the mobile terminal.

[0137] For example, when starting to use the estimation system 50 of this embodiment, an app that executes the functions of the estimation system 50 is downloaded to the user's mobile terminal and user information is registered. For example, when registering user information in the measurement device 51, if the time on the mobile terminal and the time on the clock of the measurement device 51 are synchronized, the unique time of the measurement device 51 can be set to match the universal time.

[0138] The estimation device 53 acquires a code from the measurement device 51. The estimation device 53 is implemented with an estimation model constructed by the learning device of the first or third embodiment. The estimation model is set with model parameters set by the learning device of the first or third embodiment. The measurement device 51 inputs the acquired code to the estimation model and outputs an estimation result using the original sensor data (raw data) of the code. For example, the estimation device 53 estimates a score related to the user's gait. For example, the score is a value that indexes an evaluation related to the user's gait. For example, the estimation device 53 estimates the user's physical condition. For example, the physical condition includes the degree of pronation / supination of the foot, the progress of hallux valgus, the progress of knee osteoarthritis, muscle strength, balance ability, body flexibility, and the like. The estimation process by the estimation device 53 is not particularly limited as long as it is an estimation process related to gait.

[0139] For example, the estimation device 53 is implemented in a cloud, a server, or the like (not shown). For example, the estimation device 53 may be realized by an application server. For example, the estimation device 53 may be realized by an app installed on a mobile terminal (not shown). For example, the result of the estimation process by the estimation device 53 is displayed on the screen of a mobile terminal (not shown) or a terminal device (not shown) carried by a user. For example, the result of the estimation process by the estimation device 53 is output to a system that uses the result. There is no particular limitation on the use of the result of the estimation process by the estimation device 53.

[0140] [Measuring equipment] Next, a detailed configuration of the measurement device 51 will be described with reference to the drawings. Fig. 18 is a block diagram showing an example of a detailed configuration of the measurement device 51. The measurement device 51 has an acceleration sensor 511, an angular velocity sensor 512, a control unit 513, an encoding unit 515, and a transmission unit 517. The measurement device 51 also includes a real-time clock and a power supply, which are not shown.

[0141] Acceleration sensor 511 is a sensor that measures acceleration in three axial directions (also called spatial acceleration). Acceleration sensor 511 outputs the measured acceleration to control unit 513. For example, a piezoelectric type, a piezoresistance type, a capacitance type, or other type of sensor can be used for acceleration sensor 511. Note that the measurement type of the sensor used for acceleration sensor 511 is not limited as long as it can measure acceleration.

[0142] Angular velocity sensor 512 is a sensor that measures angular velocity (also called spatial angular velocity) in three axial directions. Angular velocity sensor 512 outputs the measured angular velocity to control unit 513. For example, a vibration type, a capacitance type, or other type of sensor can be used for angular velocity sensor 512. Note that the measurement method of the sensor used for angular velocity sensor 512 is not limited as long as it can measure angular velocity.

[0143] The control unit 513 acquires the acceleration in three axial directions and the angular velocity around the three axes from each of the acceleration sensor 511 and the angular velocity sensor 512. The control unit 513 converts the acquired acceleration and angular velocity into digital data, and outputs the converted digital data (also called sensor data) to the encoding unit 515. The sensor data includes at least the acceleration data converted into digital data and the angular velocity data converted into digital data. The acceleration data includes an acceleration vector in the three axial directions. The angular velocity data includes an angular velocity vector around the three axes. The acceleration data and the angular velocity data are associated with the acquisition time of the data. The control unit 513 may be configured to output sensor data obtained by applying corrections such as mounting error, temperature correction, and linearity correction to the acquired acceleration data and angular velocity data. The control unit 513 may also generate angle data around the three axes using the acquired acceleration data and angular velocity data.

[0144] For example, the control unit 513 is a microcomputer or a microcontroller that performs overall control of the measuring device 51 and data processing. For example, the control unit 513 has a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, etc. The control unit 513 controls the acceleration sensor 511 and the angular velocity sensor 512 to measure the angular velocity and acceleration. For example, the control unit 513 performs AD conversion (Analog-to-Digital Conversion) of physical quantities (analog data) such as the measured angular velocity and acceleration, and stores the converted digital data in a flash memory. Note that the physical quantities (analog data) measured by the acceleration sensor 511 and the angular velocity sensor 512 may be converted into digital data in each of the acceleration sensor 511 and the angular velocity sensor 512. The digital data stored in the flash memory is output to the encoding unit 515 at a predetermined timing.

[0145] The encoding unit 515 acquires sensor data from the control unit 513. The encoding unit 515 includes an encoding model 551. The encoding model 551 is an encoding model constructed by the learning device of the first or third embodiment. For example, model parameters set by the learning device of the first or third embodiment are set in the encoding model 551. The encoding unit 515 inputs the acquired sensor data to the encoding model 551 and encodes the data. The encoding unit 515 outputs the code obtained by encoding the sensor data to the transmission unit 517.

[0146] The transmitting unit 517 acquires the code from the encoding unit 515. The transmitting unit 517 transmits the acquired code to the estimation device 53. The transmitting unit 517 may transmit the code to the estimation device 53 via a wired connection such as a cable, or may transmit the code to the estimation device 53 via wireless communication. For example, the transmitting unit 517 is configured to transmit the code to the estimation device 53 via a wireless communication function (not shown) conforming to a standard such as Bluetooth (registered trademark) or WiFi (registered trademark). Note that the communication function of the transmitting unit 517 may conform to a standard other than Bluetooth (registered trademark) or WiFi (registered trademark). The transmitting unit 517 also has a function of receiving data transmitted from the estimation device 53. For example, the transmitting unit 517 receives update data of model parameters, universal time data, and the like from the estimation device 53. The transmitting unit 517 outputs the received data to the control unit 513.

[0147] [Estimation device] Next, a detailed configuration of the estimation device 53 included in the estimation system 50 will be described with reference to the drawings. Fig. 19 is a block diagram showing an example of the configuration of the estimation device 53. The estimation device 53 has a receiving unit 531, an estimation unit 535, and an output unit 537.

[0148] The receiving unit 531 receives the code from the measurement device 51. The receiving unit 531 outputs the received code to the estimation unit 535. For example, the receiving unit 531 receives the code from the measurement device 51 via wireless communication. For example, the receiving unit 531 is configured to receive the code from the measurement device 51 via a wireless communication function (not shown) conforming to a standard such as Bluetooth (registered trademark) or WiFi (registered trademark). Note that the communication function of the receiving unit 531 may conform to a standard other than Bluetooth (registered trademark) or WiFi (registered trademark). For example, the receiving unit 531 may receive the code from the measurement device 51 via a wired connection such as a cable. The receiving unit 531 also has a function of transmitting data to the measurement device 51. For example, the receiving unit 531 transmits update data of the model parameters, universal time data, and the like to the measurement device 51.

[0149] The estimation unit 535 acquires the code from the receiving unit 531. The estimation unit 535 includes an estimation model 555. The estimation model 555 is an estimation model constructed by the learning device of the first or third embodiment. For example, the model parameters set by the learning device of the first or third embodiment are set in the estimation model 555. The estimation unit 535 inputs the acquired code to the estimation model 555 and outputs an estimation result based on the original sensor data (raw data) of the code. For example, the estimation unit 535 estimates a score related to the user's gait. For example, the score is a value that indexes an evaluation related to the user's gait. For example, the estimation unit 535 estimates the user's physical condition. For example, the physical condition includes the degree of pronation / supination of the foot, the progress of hallux valgus, the progress of knee osteoarthritis, muscle strength, balance ability, body flexibility, and the like. There is no particular limitation on the estimation process by the estimation unit 535 as long as it is an estimation process related to gait. The estimation unit 535 outputs the estimation result to the output unit 537.

[0150] The output unit 537 acquires the estimation result by the estimation unit 535. The output unit 537 outputs the estimation result by the estimation unit 535. For example, the output unit 537 outputs the estimation result by the estimation unit 535 to a display device (not shown). For example, the estimation result by the estimation unit 535 is displayed on the screen of the display device. For example, the estimation result by the estimation unit 535 is output to a system that uses the estimation result. There are no particular limitations on the use of the estimation result by the estimation unit 535.

[0151] FIG. 20 is a conceptual diagram for explaining the processing by the estimation system 50. In the example of FIG. 20, the estimation device 53 and the learning device 55 are implemented in a cloud or a server. FIG. 20 shows a state in which a user is walking while carrying a mobile terminal 560. A measurement device 51 is installed in a shoe 500 worn by the user. The measurement device 51 is wirelessly connected to the mobile terminal 560. The mobile terminal 560 is connected to the estimation device 53 implemented in the cloud or a server via a network 570. In addition, a learning device 55 similar to the learning device of the first or third embodiment is implemented in the cloud or a server. For example, at the timing of setting the model parameters, such as during initial setting or update, the learning device 55 transmits update data of the model parameters to the measurement device 51 and the estimation device 53.

[0152] The measurement device 51 measures sensor data related to foot movement such as acceleration and angular velocity as the user walks. The encoding unit 515 of the measurement device 51 inputs the measured sensor data to the encoding model 551 and encodes it. The measurement device 51 transmits the code obtained by encoding the sensor data to the mobile terminal 560. The code transmitted from the measurement device 51 is transmitted to the estimation device 53 via the mobile terminal 560 carried by the user and the network 570. When the measurement device 51 obtains update data for the model parameters of the encoding model 551 from the learning device 55, the measurement device 51 updates the model parameters of the encoding model 551.

[0153] The estimation device 53 receives the code from the measuring device 51 via the network 570. The estimation unit 535 of the estimation device 53 inputs the received code into the estimation model 555 to perform estimation. The estimation device 53 outputs an estimation result based on the estimation model 555. The estimation result output from the estimation device 53 is transmitted to a mobile terminal 560 carried by a user via the network 570. When the estimation device 53 obtains update data for the model parameters of the estimation model 555 from the learning device 55, the estimation device 53 updates the model parameters of the estimation model 555.

[0154] FIG. 21 is an example in which information on the estimation result by the estimation device 53 is displayed on the screen of the mobile terminal 560 carried by the user. In the example of FIG. 21, the estimation result "your walking score is XX" is displayed on the screen of the mobile terminal 560. The user who views the screen of the mobile terminal 560 can recognize the score related to his / her own gait. The estimation result by the estimation device 53 is not limited to the screen of the mobile terminal 560, and may be the screen of a stationary personal computer or a dedicated terminal as long as it is a screen that can be viewed by the user. In addition, the estimation result by the estimation device 53 may be an image representing the estimation result instead of text information. In addition, the estimation result by the estimation device 53 may be notified by a preset pattern such as sound or vibration.

[0155] (operation) Next, the operation of the estimation system 50 of this embodiment will be described with reference to the drawings. In the following, the operations of the measurement device 51 and the estimation device 53 will be described individually.

[0156] [Measuring equipment] Fig. 22 is a flowchart for explaining an example of the operation of the measuring device 51. In the explanation following the flowchart of Fig. 22, the measuring device 51 will be described as the subject of the operation.

[0157] 22, first, the measurement device 51 measures a physical quantity related to the movement of the foot (step S511). For example, the physical quantity related to the movement of the foot is an acceleration in three axial directions or an angular velocity around three axes.

[0158] Next, the measuring device 51 converts the measured physical quantity into digital data (sensor data) (step S512).

[0159] Next, the measuring device 51 inputs the sensor data (raw data) to the encoding model 551 to calculate a code (step S513).

[0160] Next, the measuring device 51 transmits the calculated code to the estimation device 53 (step S514).

[0161] When the measuring device 51 receives the update data (Yes in step S515), it updates the model parameters of the encoding model 551 (step S516). When the measuring device 51 does not receive the update data (No in step S515), it proceeds to step S517 without updating the model parameters of the encoding model 551.

[0162] If the measurement is to be stopped (Yes in step S517), the process according to the flowchart in Fig. 22 ends. The measurement may be stopped at a preset timing, or may be stopped in response to an operation by the user. If the measurement is not to be stopped (No in step S517), the process returns to step S511.

[0163] [Measuring equipment] Fig. 23 is a flowchart for explaining an example of the operation of the estimating device 53. In the explanation following the flowchart of Fig. 23, the estimating device 53 will be described as the subject of the operation.

[0164] In FIG. 23, first, the estimation device 53 receives a code from the measurement device 51 (step S521).

[0165] Next, the estimation device 53 inputs the code to the estimation model 555 and calculates the estimation result (step S522).

[0166] Next, the estimation device 53 outputs the calculated estimation result (step S523).

[0167] When the estimation device 53 receives the update data (Yes in step S524), it updates the model parameters of the estimation model 555 (step S525). When the estimation device 53 has not received the update data (No in step S524), it proceeds to step S526 without updating the model parameters of the estimation model 555.

[0168] If the estimation is to be stopped (Yes in step S526), ​​the process according to the flowchart in Fig. 23 ends. The estimation may be stopped at a preset timing, or may be stopped in response to an operation by the user. If the estimation is not to be stopped (No in step S526), ​​the process returns to step S521.

[0169] As described above, the estimation system of the present embodiment includes a measurement device and an estimation device. The measurement device includes a coding model constructed by the learning device of the first or third embodiment, and at least one measuring instrument. The measurement device inputs raw data measured by the measuring instrument to the coding model. The measurement device transmits a code output from the coding model in response to the input of the raw data to the estimation device. The estimation device includes an estimation model constructed by the learning device of the first or third embodiment. The estimation device receives the code transmitted from the measurement device. The estimation device inputs the received code to the estimation model. The estimation device outputs an estimation result output from the estimation model in response to the input of the code.

[0170] In this embodiment, the coding model constructed by the learning device of the first or third embodiment is implemented in a measurement device, and the estimation model constructed by the learning device of the first or third embodiment is implemented in an estimation device. According to this embodiment, since a code coded by the coding model is transmitted, the amount of data in communication can be reduced. Also, according to this embodiment, by using the coding model and the estimation model trained based on the relationship between the code and the feature amount, it is possible to perform highly accurate estimation while correctly grasping the causal relationship. That is, according to this embodiment, by using the coding model and the estimation model trained based on the relationship between the code and the feature amount, it is possible to construct a model that performs highly accurate estimation while correctly grasping the causal relationship by using a code that reduces the dimension of raw data.

[0171] In one aspect of this embodiment, the measurement device is worn by a user and measures sensor data (raw data) related to the user's movements. The measurement device transmits a code output from an encoding model in response to input of the measured sensor data to the estimation device. The estimation device receives the code transmitted from the measurement device. The estimation device transmits information related to the estimation result to a mobile terminal having a display unit viewable by the user. For example, the information related to the estimation result transmitted to the mobile terminal is displayed on the display unit of the mobile terminal. A user who visually recognizes the information related to the estimation result displayed on the display unit of the mobile terminal can recognize the estimation result.

[0172] Sixth embodiment Next, an estimation system according to a sixth embodiment will be described with reference to the drawings. The estimation system of this embodiment includes an encoding model and an estimation model constructed by the learning device of the second embodiment. This embodiment differs from the fifth embodiment in that a feature amount is calculated using sensor data, and estimation is performed using the calculated feature amount and a code corresponding to the feature amount.

[0173] (composition) 24 is a block diagram showing an example of the configuration of an estimation system 60 of this embodiment. The estimation system 60 includes a measurement device 61 and an estimation device 63. The measurement device 61 and the estimation device 63 may be connected by wire or wirelessly. The measurement device 61 and the estimation device 63 may be configured as a single device. The arrangement of the measurement device 61 and the estimation device 63 is the same as in the fifth embodiment. In the following, the same points as in the fifth embodiment may be omitted.

[0174] [Measuring equipment] 25 is a block diagram showing an example of a detailed configuration of the measurement device 61. The measurement device 61 has an acceleration sensor 611, an angular velocity sensor 612, a control unit 613, an encoding unit 615, a feature extraction unit 616, and a transmission unit 617. The measurement device 61 also includes a real-time clock and a power supply, which are not shown.

[0175] The acceleration sensor 611 has the same configuration as the acceleration sensor 511 of the fifth embodiment. The acceleration sensor 611 measures acceleration in three axial directions (also called spatial acceleration). The acceleration sensor 611 outputs the measured acceleration to the control unit 613.

[0176] The angular velocity sensor 612 has the same configuration as the angular velocity sensor 512 of the fifth embodiment. Surroundings The angular velocity sensor 612 measures the angular velocity (also called spatial angular velocity) of the object. The angular velocity sensor 612 outputs the measured angular velocity to the control unit 613.

[0177] The control unit 613 has the same configuration as the control unit 513 of the fifth embodiment. The control unit 613 acquires the acceleration in three axial directions and the angular velocity around three axes from each of the acceleration sensor 611 and the angular velocity sensor 612. The control unit 613 converts the acquired acceleration and angular velocity into digital data, and outputs the converted digital data (also called sensor data) to the encoding unit 615 and the feature extraction unit 616.

[0178] The encoding unit 615 acquires sensor data from the control unit 613. The encoding unit 615 includes an encoding model 651. The encoding model 651 is an encoding model constructed by the learning device of the second embodiment. For example, model parameters set by the learning device of the second embodiment are set in the encoding model 651. The encoding unit 615 inputs the acquired sensor data to the encoding model 651 and encodes the data. The encoding unit 615 outputs the code obtained by encoding the sensor data to the transmission unit 617.

[0179] The feature amount extraction unit 616 has the same configuration as the feature amount calculation unit 12 in the first embodiment. The feature amount extraction unit 616 acquires sensor data from the control unit 613. The feature amount extraction unit 616 calculates a feature amount using the acquired sensor data. The feature amount extraction unit 616 outputs the calculated feature amount to the transmission unit 617.

[0180] The transmission unit 617 has the same configuration as the transmission unit 517 in the fifth embodiment. The transmission unit 617 acquires a code from the encoding unit 615. The transmission unit 617 also acquires a feature from the feature extraction unit 616. The transmission unit 617 transmits the acquired code and feature to the estimation device 63.

[0181] [Estimation device] 26 is a block diagram showing an example of the configuration of the estimation device 63. The estimation device 63 includes a receiving unit 631, an estimation unit 635, and an output unit 637.

[0182] The receiving unit 631 has the same configuration as the receiving unit 531 of the fifth embodiment. The receiving unit 631 receives the code and the feature amount from the measurement device 61. The receiving unit 631 outputs the received code and the feature amount to the estimating unit 635.

[0183] The estimation unit 635 acquires the code and the feature amount from the receiving unit 631. The estimation unit 635 includes an estimation model 655. The estimation model 655 is an estimation model constructed by the learning device of the second embodiment. For example, the model parameters set by the learning device of the second embodiment are set in the estimation model 655. The estimation unit 635 inputs the acquired code and the feature amount to the estimation model 655, and outputs an estimation result based on the original sensor data (raw data) of the code. For example, the estimation unit 635 estimates a score related to the user's gait. For example, the score is a value that indexes an evaluation related to the user's gait. For example, the estimation unit 635 estimates the user's physical condition. For example, the physical condition includes the degree of pronation / supination of the foot, the progress of hallux valgus, the progress of knee osteoarthritis, muscle strength, balance ability, body flexibility, and the like. There is no particular limitation on the estimation process by the estimation unit 635 as long as it is an estimation process related to gait. The estimation unit 635 outputs the estimation result to the output unit 637.

[0184] The output unit 637 has the same configuration as the output unit 537 in the fifth embodiment. The output unit 637 acquires the estimation result by the estimation unit 635. The output unit 637 outputs the estimation result by the estimation unit 635.

[0185] FIG. 27 is a conceptual diagram for explaining the processing by the estimation system 60. In the example of FIG. 27, the estimation device 63 and the learning device 65 are implemented in a cloud or a server. FIG. 27 shows a state where a user is walking while carrying a mobile terminal 660. A measurement device 61 is installed in a shoe 600 worn by the user. The measurement device 61 is wirelessly connected to the mobile terminal 660. The mobile terminal 660 is connected to the estimation device 63 implemented in the cloud or a server via a network 670. In addition, a learning device 65 similar to the learning device of the second embodiment is implemented in the cloud or a server. For example, at the timing of setting the model parameters, such as at the time of initial setting or update, the learning device 65 transmits update data of the model parameters to the measurement device 61 and the estimation device 63.

[0186] The measurement device 61 measures sensor data related to foot movement such as acceleration and angular velocity as the user walks. The encoding unit 615 of the measurement device 61 inputs the measured sensor data to the encoding model 651 and encodes it. In addition, the feature extraction unit 616 of the measurement device 61 calculates the feature using the sensor data. The measurement device 61 transmits the code and the feature obtained by encoding the sensor data to the mobile terminal 660. The code and the feature transmitted from the measurement device 61 are transmitted to the estimation device 63 via the mobile terminal 660 carried by the user and the network 670. When the measurement device 61 obtains update data for the model parameters of the encoding model 651 from the learning device 65, it updates the model parameters of the encoding model 651.

[0187] The estimation device 63 receives the code and the feature amount from the measurement device 61 via the network 670. The estimation unit 635 of the estimation device 63 inputs the received code and the feature amount into the estimation model 655 to perform estimation. The estimation device 63 outputs the estimation result by the estimation model 655. The estimation result output from the estimation device 63 is transmitted to the mobile terminal 660 carried by the user via the network 670. When the estimation device 63 obtains update data for the model parameters of the estimation model 655 from the learning device 65, it updates the model parameters of the estimation model 655.

[0188] (operation) Next, the operation of the estimation system 60 of this embodiment will be described with reference to the drawings. In the following, the operations of the measurement device 61 and the estimation device 63 will be described individually.

[0189] [Measuring equipment] Fig. 28 is a flowchart for explaining an example of the operation of the measuring device 61. In the explanation following the flowchart of Fig. 28, the measuring device 61 will be described as the subject of the operation.

[0190] 28, first, the measurement device 61 measures a physical quantity related to the movement of the foot (step S611). For example, the physical quantity related to the movement of the foot is an acceleration in three axial directions or an angular velocity around three axes.

[0191] Next, the measuring device 61 converts the measured physical quantity into digital data (sensor data) (step S612).

[0192] Next, the measuring device 61 inputs the sensor data (raw data) to the encoding model 651 to calculate a code (step S613).

[0193] Next, the measuring device 61 extracts features from the sensor data (step S614). Steps S613 and S614 may be interchanged in order, or may be performed in parallel.

[0194] Next, the measuring device 61 transmits the code and the feature amount to the estimation device 63 (step S615).

[0195] When the measuring device 61 receives the update data (Yes in step S616), it updates the model parameters of the encoding model 651 (step S617). When the measuring device 61 has not received the update data (No in step S616), it proceeds to step S618 without updating the model parameters of the encoding model 651.

[0196] If the measurement is to be stopped (Yes in step S618), the process according to the flowchart in Fig. 28 ends. The measurement may be stopped at a preset timing, or may be stopped in response to an operation by the user. If the measurement is not to be stopped (No in step S618), the process returns to step S611.

[0197] [Measuring equipment] Fig. 29 is a flowchart for explaining an example of the operation of the estimating device 63. In the explanation following the flowchart of Fig. 29, the estimating device 63 will be described as the subject of the operation.

[0198] In FIG. 29, first, the estimation device 63 receives the code and the feature amount from the measurement device 61 (step S621).

[0199] Next, the estimation device 63 inputs the code and the feature amount to the estimation model 655 and calculates the estimation result (step S622).

[0200] Next, the estimation device 63 outputs the calculated estimation result (step S623).

[0201] When the estimation device 63 receives the update data (Yes in step S624), it updates the model parameters of the estimation model 655 (step S625). When the estimation device 63 has not received the update data (No in step S624), it proceeds to step S626 without updating the model parameters of the estimation model 655.

[0202] If the estimation is to be stopped (Yes in step S626), the process according to the flowchart in Fig. 29 ends. The estimation may be stopped at a preset timing, or may be stopped in response to an operation by the user. If the estimation is not to be stopped (No in step S626), the process returns to step S621.

[0203] As described above, the estimation system of the present embodiment includes a measurement device and an estimation device. The measurement device includes a coding model constructed by the learning device of the second embodiment and at least one measuring instrument. The measurement device calculates a feature amount using raw data. The measurement device inputs raw data measured by the measuring instrument to the coding model. The measurement device transmits to the estimation device a code output from the coding model in response to the input of the raw data and a feature amount calculated using the raw data. The estimation device includes an estimation model constructed by the learning device of the second embodiment. The estimation device receives the code and the feature amount transmitted from the measurement device. The estimation device inputs the received code and feature amount to the estimation model. The estimation device outputs an estimation result output from the estimation model in response to the input of the code.

[0204] In this embodiment, the coding model constructed by the learning device of the second embodiment is implemented in a measurement device, and the estimation model constructed by the learning device of the second embodiment is implemented in an estimation device. According to this embodiment, as in the fifth embodiment, by using a coding model and an estimation model trained based on the relationship between the code and the feature amount, it is possible to perform highly accurate estimation while correctly grasping the causal relationship. Furthermore, according to this embodiment, since estimation is performed using the feature amount in addition to the code, it is possible to perform more accurate estimation than the fifth embodiment.

[0205] Seventh embodiment Next, an estimation system according to the seventh embodiment will be described with reference to the drawings. The estimation system according to the present embodiment includes an encoding model, a reconstructed model, and an estimation model constructed by the learning device according to the first embodiment. This embodiment differs from the fifth embodiment in that estimation is performed using a reconstructed feature reconstructed by the reconstructed model.

[0206] (composition) 30 is a block diagram showing an example of the configuration of an estimation system 70 of this embodiment. The estimation system 70 includes a measurement device 71 and an estimation device 73. The measurement device 71 and the estimation device 73 may be connected by wire or wirelessly. The measurement device 71 and the estimation device 73 may be configured as a single device. The arrangement of the measurement device 71 and the estimation device 73 is the same as in the fifth embodiment. In the following, the same points as in the fifth embodiment may be omitted.

[0207] [Measuring equipment] 31 is a block diagram showing an example of a detailed configuration of the measurement device 71. The measurement device 71 has an acceleration sensor 711, an angular velocity sensor 712, a control unit 713, an encoding unit 715, and a transmission unit 717. The measurement device 71 also includes a real-time clock and a power supply, which are not shown.

[0208] The acceleration sensor 711 has the same configuration as the acceleration sensor 511 of the fifth embodiment. The acceleration sensor 711 measures acceleration in three axial directions (also called spatial acceleration). The acceleration sensor 711 outputs the measured acceleration to the control unit 713.

[0209] The angular velocity sensor 712 has the same configuration as the angular velocity sensor 512 of the fifth embodiment. The angular velocity sensor 712 measures angular velocities in three axial directions (also called spatial angular velocities). The angular velocity sensor 712 outputs the measured angular velocities to the control unit 713.

[0210] The control unit 713 has the same configuration as the control unit 513 of the fifth embodiment. The control unit 713 acquires the acceleration in three axial directions and the angular velocity around the three axes from each of the acceleration sensor 711 and the angular velocity sensor 712. The control unit 713 converts the acquired acceleration and angular velocity into digital data, and outputs the converted digital data (also called sensor data) to the encoding unit 715.

[0211] The encoding unit 715 acquires sensor data from the control unit 713. The encoding unit 715 includes an encoding model 751. The encoding model 751 is an encoding model constructed by the learning device of the first embodiment (variation 1). For example, model parameters set by the learning device of the first embodiment (variation 1) are set in the encoding model 751. The encoding unit 715 inputs the acquired sensor data to the encoding model 751 and encodes the data. The encoding unit 715 outputs the code obtained by encoding the sensor data to the transmission unit 717.

[0212] The transmitting unit 717 has the same configuration as the transmitting unit 517 in the fifth embodiment. The transmitting unit 717 obtains the code from the encoding unit 715. The transmitting unit 717 transmits the obtained code to the estimation device 73.

[0213] [Estimation device] 32 is a block diagram showing an example of the configuration of the estimation device 73. The estimation device 73 includes a receiving unit 731, a reconstructing unit 733, an estimation unit 735, and an output unit 737.

[0214] The receiving unit 731 has the same configuration as the receiving unit 531 of the fifth embodiment. The receiving unit 731 receives a code from the measuring device 71. The receiving unit 731 outputs the received code to the reconstructing unit 733.

[0215] The reconstruction unit 733 acquires the code from the receiving unit 731. The reconstruction unit 733 includes a reconstructed model 753. The reconstructed model 753 is an estimation model constructed by the learning device of the first embodiment (variation 1). For example, the reconstructed model 753 is set with model parameters set by the learning device of the first embodiment (variation). The reconstruction unit 733 inputs the acquired code to the reconstructed model 753 and calculates a reconstructed feature. The reconstruction unit 733 outputs the calculated reconstructed feature to the estimation model 755.

[0216] The estimation unit 735 acquires the reconstructed feature from the reconstruction unit 733. The estimation unit 735 includes an estimation model 755. The estimation model 755 is an estimation model constructed by the learning device of the first embodiment (modification 1). For example, the estimation model 755 is set with model parameters set by the learning device of the first embodiment (modification 1). The estimation unit 735 inputs the acquired reconstructed feature to the estimation model 755, and outputs an estimation result based on the original sensor data (raw data) of the reconstructed feature. For example, the estimation unit 735 estimates a score related to the user's gait. For example, the score is a value that indexes an evaluation related to the user's gait. For example, the estimation unit 735 estimates the user's physical condition. For example, the physical condition includes the degree of pronation / supination of the foot, the progress of hallux valgus, the progress of knee osteoarthritis, muscle strength, balance ability, body flexibility, and the like. There is no particular limitation on the estimation process by the estimation unit 735 as long as it is an estimation process related to gait. The estimation unit 735 outputs the estimation result to the output unit 637.

[0217] The output unit 737 has the same configuration as the output unit 537 in the fifth embodiment. The output unit 737 acquires the estimation result by the estimation unit 735. The output unit 737 outputs the estimation result by the estimation unit 735.

[0218] FIG. 33 is a conceptual diagram for explaining the processing by the estimation system 70. In the example of FIG. 33, the estimation device 73 and the learning device 75 are implemented in a cloud or a server. FIG. 33 shows a state where a user is walking while carrying a mobile terminal 760. A measurement device 71 is installed in shoes 700 worn by the user. The measurement device 71 is wirelessly connected to the mobile terminal 760. The mobile terminal 760 is connected to the estimation device 73 implemented in the cloud or a server via a network 770. In addition, a learning device 75 similar to the learning device of the first embodiment (variation 1) is implemented in the cloud or a server. For example, at the timing of setting the model parameters, such as during initial setting or update, the learning device 75 transmits update data of the model parameters to the measurement device 71 and the estimation device 73.

[0219] The measurement device 71 measures sensor data related to foot movement such as acceleration and angular velocity as the user walks. The encoding unit 715 of the measurement device 71 inputs the measured sensor data to an encoding model 751 and encodes it. The measurement device 71 transmits the code obtained by encoding the sensor data to a mobile terminal 760. The code transmitted from the measurement device 71 is transmitted to the estimation device 73 via the mobile terminal 760 carried by the user and a network 770. When the measurement device 71 obtains update data for the model parameters of the encoding model 751 from the learning device 75, it updates the model parameters of the encoding model 751.

[0220] The estimation device 73 receives the code from the measurement device 71 via the network 770. The reconstruction unit 733 of the estimation device 73 inputs the received code to the reconstructed model 753 to calculate a reconstructed feature. The estimation unit 735 of the estimation device 73 inputs the reconstructed feature calculated by the reconstruction unit 733 to the estimation model 755 to perform estimation. The estimation device 73 outputs an estimation result. The estimation result output from the estimation device 73 is transmitted to a mobile terminal 760 carried by a user via the network 770. When the estimation device 73 obtains update data for the model parameters of the reconstructed model 753 and the estimation model 755 from the learning device 75, the estimation device 73 updates the model parameters of the reconstructed model 753 and the estimation model 755.

[0221] (operation) Next, the operation of the estimation system 70 of this embodiment will be described with reference to the drawings. In the following, the operations of the measurement device 71 and the estimation device 73 will be described individually.

[0222] [Measuring equipment] Fig. 34 is a flowchart for explaining an example of the operation of the measuring device 71. In the explanation following the flowchart of Fig. 34, the measuring device 71 will be described as the subject of the operation.

[0223] 34, first, the measurement device 71 measures a physical quantity related to the movement of the foot (step S711). For example, the physical quantity related to the movement of the foot is an acceleration in three axial directions or an angular velocity around three axes.

[0224] Next, the measuring device 71 converts the measured physical quantity into digital data (sensor data) (step S712).

[0225] Next, the measuring device 71 inputs the sensor data (raw data) to the encoding model 751 to calculate a code (step S713).

[0226] Next, the measuring device 71 transmits the calculated code to the estimation device 73 (step S714).

[0227] When the measuring device 71 receives the update data (Yes in step S715), it updates the model parameters of the encoding model 751 (step S716). When the measuring device 71 does not receive the update data (No in step S715), it proceeds to step S717 without updating the model parameters of the encoding model 751.

[0228] If the measurement is to be stopped (Yes in step S717), the process according to the flowchart in Fig. 34 ends. The measurement may be stopped at a preset timing, or may be stopped in response to an operation by the user. If the measurement is not to be stopped (No in step S717), the process returns to step S711.

[0229] [Measuring equipment] Fig. 35 is a flowchart for explaining an example of the operation of the estimating device 73. In the explanation following the flowchart of Fig. 35, the estimating device 73 will be described as the subject of the operation.

[0230] In FIG. 35, first, the estimation device 73 receives a code from the measurement device 71 (step S721).

[0231] Next, the estimation device 73 inputs the code to the reconstructed model 753 and calculates the reconstructed feature (step S722).

[0232] Next, the estimation device 73 inputs the reconstructed feature to the estimation model 755 and calculates the estimation result (step S723).

[0233] Next, the estimation device 73 outputs the calculated estimation result (step S724).

[0234] When the estimation device 73 receives the update data (Yes in step S725), it updates the model parameters of the reconstructed model 753 and the estimation model 755 (step S726). When the estimation device 73 has not received the update data (No in step S725), it proceeds to step S727 without updating the model parameters of the reconstructed model 753 and the estimation model 755.

[0235] If the estimation is to be stopped (Yes in step S727), the process according to the flowchart in Fig. 35 ends. The estimation may be stopped at a preset timing, or may be stopped in response to an operation by the user. If the estimation is not to be stopped (No in step S727), the process returns to step S721.

[0236] As described above, the estimation system of this embodiment includes a measurement device and an estimation device. The measurement device includes a coding model constructed by the learning device of the first embodiment (variation) and at least one measuring instrument (sensor). The measurement device inputs raw data measured by the measuring instrument to the coding model. The measurement device transmits a code output from the coding model in response to the input of raw data to the estimation device. The estimation device includes a reconstruction model that outputs a reconstructed feature corresponding to an original feature of the code in response to the input of the code. The estimation device also includes an estimation model that outputs an estimation result corresponding to the original raw data of the code in response to the input of the reconstructed feature. The estimation device receives the code transmitted from the measurement device. The estimation device inputs the received code to the reconstructed model. The estimation device inputs the reconstructed feature output from the reconstructed model in response to the input of the code to the estimation model. The estimation device outputs an estimation result output from the estimation model in response to the input of the reconstructed feature.

[0237] In this embodiment, the coding model constructed by the learning device of the first embodiment (modification) is implemented in a measurement device. Also, the reconstructed model constructed by the learning device of the first embodiment (modification) and an estimation model that outputs an estimation result corresponding to raw data in response to an input of a reconstructed feature output by the reconstructed model are implemented in an estimation device. According to this embodiment, as in the fifth and sixth embodiments, by using a coding model and an estimation model trained based on the relationship between a code and a feature, it is possible to perform highly accurate estimation while correctly grasping the causal relationship. Also, according to this embodiment, since estimation is performed using a reconstructed feature reconstructed from a code, it is possible to perform more accurate estimation than the fifth embodiment.

[0238] (Verification example) Next, a verification example in which a model constructed by the learning device of each embodiment is verified will be described. In this verification example, a model constructed by the learning device according to the first embodiment is verified using a dataset of Modified National Institute of Standards and Technology (Fashoin-MNIST). Fashoin-MNIST is a dataset disclosed in Non-Patent Document 1 (Non-Patent Document 1: H. Xiao, K. Rasul, R. Vollgraf, “Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms”, arXiv preprint arXiv:1708.07747 (2017).). Fashoin-MNIST is a dataset of clothing images. The Fashoin-MNIST dataset includes images of 28×28 pixels and labels of the images. The labels include 10 types of clothing classes from 0 to 9. Descriptions of the labels and classes will be omitted.

[0239] In this verification example, the image size of a 28×28 pixel image (raw data) was converted to a reduced image of 7×7 pixels. The reduced image of 7×7 pixels corresponds to the feature amount of the 28×28 pixel image (raw data). The reduced image (feature amount) contains the features of the raw data because the appearance of the raw data image remains. Furthermore, the reduced image has a reduced data volume compared to the raw data. In other words, the reduced image is suitable as a feature amount including knowledge that is handled in this disclosure.

[0240] Fig. 36 is a conceptual diagram for explaining the configurations of the coding model, the reconstruction model, and the estimation model in this verification example. Fig. 37 is a table summarizing the details of the configurations of the coding model, the reconstruction model, and the estimation model in this verification example. In this verification example, the coding model and the reconstruction model were trained so that the error between the reconstruction result (reconstruction feature amount) calculated by the reconstruction model and the feature amount calculated from the original raw data of the reconstruction feature amount is minimized.

[0241] The coding model includes an input layer E1, a first hidden layer E2, a second hidden layer E3, and an output layer E4. The number of units in the input layer E1 is 28×28. Raw data of 28×28 (=784) pixels is input to the input layer E1. When raw data of 28×28 pixels is input in units of 28 pixels, the number of units in the input layer E1 may be 28. The number of units in the first hidden layer E2 is 300. The activation function of the first hidden layer E2 is a rectified linear function (ReLU: Rectified Linear Unit). Each unit in the first hidden layer E2 is fully connected to each unit in the input layer E1 and the second hidden layer E3. The number of units in the second hidden layer E3 is 300. The activation function of the second hidden layer E3 is a rectified linear function (ReLU). Each unit in the second hidden layer E3 is fully connected to each unit in the first hidden layer E2 and the output layer E4. The number of units in the output layer E4 is 49. The activation function of the output layer E4 is a sigmoid. The encoding model outputs a code, which is a vector having 49 numerical values ​​as elements, in response to the input of raw data of 28×28 pixels.

[0242] The reconstructed model includes an input layer R1, a first hidden layer R2, a second hidden layer R3, and an output layer R4. The number of units in the input layer R1 is 49. A code, which is a vector having 49 numerical values ​​as elements, is input to the input layer R1. The number of units in the first hidden layer R2 is 300. The activation function of the first hidden layer R2 is a rectified linear function (ReLU). Each unit in the first hidden layer R2 is fully connected to each unit in the input layer R1 and the second hidden layer R3. The number of units in the second hidden layer R3 is 300. The activation function of the second hidden layer R3 is a rectified linear function (ReLU). Each unit in the second hidden layer R3 is fully connected to each unit in the first hidden layer R2 and the output layer R4. The number of units in the output layer R4 is 7 × 7. The activation function of the output layer R4 is a sigmoid. When outputting a 7×7 pixel reconstruction result (reconstructed feature) for every 7 pixels, the number of units in the output layer R4 may be 7. The reconstruction model outputs a 7×7 pixel reconstruction result (reconstructed feature) in response to an input of a code, which is a vector having 49 numerical values ​​as elements.

[0243] The estimation model includes an input layer C1, a first hidden layer C2, a second hidden layer C3, and an output layer C4. The number of units in the input layer C1 is 49. A code, which is a vector having 49 numerical values ​​as elements, is input to the input layer C1. The number of units in the first hidden layer C2 is 300. The activation function of the first hidden layer C2 is a rectified linear function (ReLU). Each unit in the first hidden layer C2 is fully connected to each unit in the input layer C1 and the second hidden layer C3. The number of units in the second hidden layer C3 is 300. The activation function of the second hidden layer C3 is a rectified linear function (ReLU). Each unit in the second hidden layer C3 is fully connected to each unit in the first hidden layer C2 and the output layer C4. The number of units in the output layer C4 is 10. The activation function of the output layer C4 is a softmax. The estimation model outputs 10 classes as estimation results in response to the input of a code, which is a vector with 49 numerical elements.

[0244] 38 is a diagram for explaining the configuration of an estimation model of Comparative Example 1. The estimation model of Comparative Example 1 outputs 10 types of classes as estimation results in response to input of feature amounts of 7×7 pixels calculated using raw data of 28×28 pixels. Comparative Example 1 is an example for evaluating the classification accuracy that should be achieved at a minimum.

[0245] The estimation model of Comparative Example 1 includes an input layer A1, a first hidden layer A2, a second hidden layer A3, and an output layer A4. The number of units in the input layer A1 is 7×7. A feature of 7×7 (=49) pixels is input to the input layer A1. When the feature of 7×7 pixels is input 7 pixels at a time, the number of units in the input layer A1 may be 7. The number of units in the first hidden layer A2 is 300. The activation function of the first hidden layer A2 is a rectified linear function (ReLU). Each unit in the first hidden layer A2 is fully connected to each unit in the input layer A1 and the second hidden layer A3. The number of units in the second hidden layer A3 is 300. The activation function of the second hidden layer A3 is a rectified linear function (ReLU). Each unit in the second hidden layer A3 is fully connected to each unit in the first hidden layer A2 and the output layer A4. The number of units in the output layer A4 is 10. The activation function of the output layer A4 is softmax. The estimation model of Comparative Example 1 outputs 10 types of classes as estimation results in response to an input of 7×7 pixel features.

[0246] 39 is a diagram for explaining the configuration of an estimation model of Comparative Example 2. The estimation model of Comparative Example 2 outputs 10 types of classes as estimation results in response to input of raw data of 28×28 pixels. Comparative Example 2 is an example for evaluating the maximum expected classification accuracy.

[0247] The estimation model of Comparative Example 2 includes an input layer B1, a first hidden layer B2, a second hidden layer B3, and an output layer B4. The number of units in the input layer B1 is 28×28. Raw data of 28×28 (=784) pixels is input to the input layer B1. When raw data of 28×28 pixels is input in units of 28 pixels, the number of units in the input layer B1 may be 28. The number of units in the first hidden layer B2 is 300. The activation function of the first hidden layer B2 is a rectified linear function (ReLU). Each unit in the first hidden layer B2 is fully connected to each unit in the input layer B1 and the second hidden layer B3. The number of units in the second hidden layer B3 is 300. The activation function of the second hidden layer B3 is a rectified linear function (ReLU). Each unit in the second hidden layer B3 is fully connected to each unit in the first hidden layer B2 and the output layer B4. The number of units in the output layer B4 is 10. The activation function of the output layer B4 is softmax. The estimation model of Comparative Example 2 outputs 10 types of classes as estimation results in response to an input of features of 28×28 pixels.

[0248] In this verification example, of the 70,000 datasets of Fashoin-MNIST, 55,000 datasets were used as training data, 5,000 datasets were used as validation data, and 10,000 datasets were used as test data. Different datasets were used for the training data, validation data, and test data. In this verification example, Adam optimization was used as the optimization algorithm, and the epoch was set to 30 times. As the loss of the estimation result, a multi-class cross entropy error was calculated. As the loss of the reconstruction result (reconstruction feature), a mean square error was calculated. In this verification example, the weighted average of the loss of the estimation result and the reconstruction result was used as the loss function. With respect to the weighted average of the loss of the estimation result and the reconstruction result, the weight of the estimation result is denoted as α, and the weight of the reconstruction result is denoted as β. A multi-class cross entropy error function was used as the loss function in Comparative Example 1-2. As the evaluation index, a value obtained by dividing the total number of test data by the number of correct answers (classification accuracy) was used.

[0249] FIG. 40 is a table summarizing the classification accuracy of this verification example and comparative examples 1-2. For this verification example, the weight α of the estimation result was set to 0.9, the weight β of the reconstruction result was set to 0.1, and the weighted average of the losses of the estimation result and the reconstruction result was set as the loss function. In this verification example (α=0.9, β=0.1), the classification accuracy was 89.43% (percent). The classification accuracy of comparative example 1 was 85.53%. The classification accuracy of comparative example 2 was 89.06%. That is, according to this verification example, while the amount of data used for estimation was the same as that of comparative example 1, a classification accuracy equivalent to that of comparative example 2 was obtained.

[0250] FIG. 41 is a diagram for comparing the communication load (data size) and classification accuracy of this verification example and comparative examples 1-2. The communication load of this verification example is as light as that of comparative example 1 in which estimation is performed using feature amounts. Moreover, the classification accuracy of this verification example is as high as that of comparative example 2 in which estimation is performed using raw data. That is, according to this verification example, high classification accuracy comparable to that of comparative example 2 in which estimation is performed using raw data was achieved while the communication load is as light as that of comparative example 1 in which estimation is performed using feature amounts.

[0251] Next, regarding this verification example, an example is shown in which the weight α of the estimation result and the weight β of the reconstruction result are changed to verify the reconstruction result (reconstructed feature amount) by the reconstruction model. FIG. 42 is a conceptual diagram for explaining the change in the reconstructed feature amount according to the weight α of the estimation result and the weight β of the reconstruction result. FIG. 42 shows an image of 7×7 pixels, but does not accurately represent the actual experimental results, but represents them conceptually. FIG. 42 also shows the feature amount (reference example) calculated using raw data. The feature amount of the reference example is designed based on knowledge.

[0252] In reconstruction example 1 (α = 0.9, β = 0.1), the proportion of weight β in the reconstruction result is large (10%). The classification accuracy of the estimation result according to the input of the code to the estimation model constructed under the conditions of reconstruction example 1 was 89.43%. It can be intuitively understood that the reconstructed features of reconstruction example 1 retain the characteristics of the features of the reference example.

[0253] In reconstruction example 2 (α = 0.9999, β = 0.0001), the proportion of weight β in the reconstruction result is smaller (0.01%) than in reconstruction example 1. The classification accuracy of the estimation result according to the code input to the estimation model constructed under the conditions of reconstruction example 2 was 88.73%. It can be intuitively understood that the reconstructed features of reconstruction example 2 also retain the characteristics of the features of the reference example, but compared to reconstruction example 1, there are many aspects that differ from the reference example.

[0254] In the reconstruction example 3 (α=1, β=0), the weight β of the reconstruction result is 0. In other words, in the reconstruction example 3, the reconstruction result (reconstruction feature amount) is not reflected in the model. To The classification accuracy of the estimation results according to the input of the sign was 89.56%. It is not intuitively clear that the reconstructed features of reconstruction example 3 retain the features of the reference example.

[0255] The feature of the reference example is designed based on knowledge, and therefore includes features that can be intuitively understood. According to reconstruction examples 1 to 3, by increasing the weight β of the reconstruction result, a reconstructed feature closer to the reference example was reconstructed from the code. That is, in this verification example, the code contains sufficient information corresponding to knowledge. On the other hand, when the weight β of the reconstruction result is reduced, the knowledge constraint is weakened, and it becomes difficult to reconstruct the feature from the code. For example, when the reliability of the feature is low, it may be possible to improve the performance in practice by reducing the value of the weight β of the reconstruction result to weaken the knowledge constraint and to make the code contain more information from the data origin.

[0256] For example, in the model learning stage, a reconstructed feature reconstructed using a code calculated from raw data may be displayed on a screen. By displaying the reconstructed feature on a screen, the features contained in the original code of the reconstructed feature are visualized, so that the quality of the model in the learning stage can be intuitively understood. For example, as shown in FIG. 42, by displaying the feature and the reconstructed feature side by side on the screen, the features contained in the original code of the reconstructed feature can be more intuitively understood.

[0257] For example, by calculating the similarity between the feature and the reconstructed feature, the feature included in the original code of the reconstructed feature can be evaluated numerically. For example, by setting a threshold value for the similarity between the feature and the reconstructed feature, the feature included in the original code of the reconstructed feature can be evaluated numerically according to the relationship between the similarity and the threshold. For example, the similarity between the feature and the reconstructed feature may be displayed on a screen.

[0258] (Hardware) Here, a hardware configuration for executing the processing of the learning device or estimation device according to each embodiment of the present disclosure will be described using an information processing device 90 in Fig. 43 as an example. Note that the information processing device 90 in Fig. 43 is an example of a configuration for executing the processing of the learning device or estimation device according to each embodiment, and does not limit the scope of the present disclosure.

[0259] As shown in Fig. 43, an information processing device 90 includes a processor 91, a main storage device 92, an auxiliary storage device 93, an input / output interface 95, and a communication interface 96. In Fig. 43, the interface is abbreviated as I / F (Interface). The processor 91, the main storage device 92, the auxiliary storage device 93, the input / output interface 95, and the communication interface 96 are connected to each other via a bus 98 so as to be able to communicate data with each other. In addition, the processor 91, the main storage device 92, the auxiliary storage device 93, and the input / output interface 95 are connected to a network such as the Internet or an intranet via the communication interface 96.

[0260] The processor 91 loads a program stored in the auxiliary storage device 93 or the like into the main storage device 92. The processor 91 executes the program loaded into the main storage device 92. In this embodiment, a software program installed in the information processing device 90 may be used. The processor 91 executes processing by the learning device and the estimation device according to this embodiment.

[0261] The main memory device 92 has an area in which a program is loaded. The processor 91 loads a program stored in the auxiliary memory device 93 or the like in the main memory device 92. The main memory device 92 is realized by a volatile memory such as a dynamic random access memory (DRAM). Furthermore, a non-volatile memory such as a magnetoresistive random access memory (MRAM) may be configured / added to the main memory device 92.

[0262] The auxiliary storage device 93 stores various data such as programs. The auxiliary storage device 93 is realized by a local disk such as a hard disk or a flash memory. Note that it is also possible to configure the main storage device 92 to store various data and omit the auxiliary storage device 93.

[0263] The input / output interface 95 is an interface for connecting the information processing device 90 to peripheral devices based on standards and specifications. The communication interface 96 is an interface for connecting to external systems and devices through a network such as the Internet or an intranet based on standards and specifications. The input / output interface 95 and the communication interface 96 may be a common interface for connecting to external devices.

[0264] Input devices such as a keyboard, a mouse, and a touch panel may be connected to the information processing device 90 as necessary. These input devices are used to input information and settings. When a touch panel is used as an input device, the display screen of the display device may also serve as an interface for the input device. Data communication between the processor 91 and the input devices may be mediated by an input / output interface 95.

[0265] The information processing device 90 may be equipped with a display device for displaying information. When the display device is equipped, the information processing device 90 is preferably equipped with a display control device (not shown) for controlling the display of the display device. The display device may be connected to the information processing device 90 via the input / output interface 95.

[0266] The information processing device 90 may also be provided with a drive device. The drive device mediates between the processor 91 and a recording medium (program recording medium) for reading data and programs from the recording medium, writing the processing results of the information processing device 90 to the recording medium, and the like. The drive device may be connected to the information processing device 90 via an input / output interface 95.

[0267] The above is an example of a hardware configuration for enabling the learning device or the estimation device according to each embodiment of the present invention. The hardware configuration in FIG. 43 is an example of a hardware configuration for executing the arithmetic processing of the learning device or the estimation device according to each embodiment, and does not limit the scope of the present invention. In addition, a program for causing a computer to execute processing related to the learning device or the estimation device according to each embodiment is also included in the scope of the present invention. Furthermore, a program recording medium on which the program according to each embodiment is recorded is also included in the scope of the present invention. The recording medium can be realized, for example, by an optical recording medium such as a CD (Compact Disc) or a DVD (Digital Versatile Disc). The recording medium may be realized by a semiconductor recording medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) card. In addition, the recording medium may be realized by a magnetic recording medium such as a flexible disk or other recording medium. When the program executed by the processor is recorded on a recording medium, the recording medium corresponds to a program recording medium.

[0268] The components of the learning device and the estimation device of each embodiment may be combined in any manner. Furthermore, the components of the learning device and the estimation device of each embodiment may be realized by software or a circuit.

[0269] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0270] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) An acquisition unit for acquiring a dataset of raw data and ground truth data; a feature amount calculation unit that calculates a feature amount using the raw data; a model construction unit that constructs an encoding model that outputs a code corresponding to the feature amount in response to input of the raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input of the code; a learning processing unit that trains the encoding model and the estimation model based on the relationship between the code and the feature amount so that the estimation result matches the ground truth data. (Appendix 2) The model construction unit includes: 2. The learning device according to claim 1, which constructs the estimation model using the code output from the encoding model in response to input of the raw data and the feature calculated using the raw data. (Appendix 3) The model construction unit includes: constructing a reconstruction model that outputs a reconstructed feature corresponding to the original feature of the code in response to the input of the code; The learning processing unit: training the encoding model and the reconstruction model such that the reconstructed features match the features; 3. The learning device according to claim 1, further comprising: a training unit for training the encoding model and the estimation model based on a relationship between the code and the feature so that the estimation result matches the ground truth data. (Appendix 4) The model construction unit includes: 4. The learning device according to claim 3, which constructs the estimation model using the reconstructed feature output from the reconstructed model in response to an input of the code and the code. (Appendix 5) The learning processing unit: 5. The learning device according to claim 1, wherein the estimation model is trained so that a mutual information amount between the code output from the encoding model in response to input of the raw data and the feature amount calculated using the raw data is reduced. (Appendix 6) The model construction unit includes: constructing a first pre-learning model that outputs a first conversion value corresponding to the feature amount in response to input of the raw data; The learning processing unit: training the first pre-trained model so that the first transformation value matches the feature; Setting model parameters of the first pre-learning model after training as initial values ​​of the encoding model; 3. The learning device according to claim 1, further comprising: a training unit for training the encoding model and the estimation model so that the estimation result matches the correct answer data. (Appendix 7) The model construction unit includes: constructing a second pre-learning model that outputs a second conversion value corresponding to the raw data in response to input of the feature amount calculated using the raw data; The learning processing unit: training the second pre-trained model so that the second transformation value fits the estimation result; Setting model parameters of the second pre-learned model after training as initial values ​​of the estimation model; 7. The learning device of claim 6, which trains the encoding model and the estimation model so that the estimation result matches the correct answer data. (Appendix 8) An estimation device in which an encoding model and an estimation model constructed by the learning device according to any one of Supplementary Note 1 to 7 are implemented, a measurement device including at least one measuring instrument and the coding model, inputting raw data measured by the measuring instrument into the coding model, and transmitting a code output from the coding model in response to the input of the raw data; an estimation device that includes the estimation model, receives the code transmitted from the measurement instrument, inputs the received code to the estimation model, and outputs an estimation result that is output from the estimation model in response to the input of the code. (Appendix 9) The measuring device includes: Calculating features using the raw data; Transmitting the code output from the encoding model in response to input of the raw data and the feature amount calculated using the raw data to the estimation device; The estimation device comprises: receiving the code and the feature amount transmitted from the measurement device; inputting the received code and the feature quantity into the estimation model; 9. The estimation system according to claim 8, which outputs the estimation result output from the estimation model in response to an input of the code and the feature amount. (Appendix 10) The estimation device comprises: a reconstruction model that outputs a reconstructed feature corresponding to the original feature of the code in response to an input of the code; A reconstructed feature output from the reconstructed model in response to the input of the code is input to the estimation model; 10. The estimation system according to claim 8, further comprising: a processor configured to output the estimation result from the estimation model in response to an input of the reconstructed feature. (Appendix 11) The measuring device includes: a sensor device that is worn by a user, measures sensor data related to the user's movements, and transmits the code output from the encoding model in response to the input of the measured sensor data to the estimation device; The estimation device comprises: receiving the code transmitted from the measuring device; 11. The estimation system according to any one of claims 8 to 10, further comprising: a mobile terminal having a display unit viewable by the user, the mobile terminal transmitting information regarding the estimation result. (Appendix 12) The computer Obtain a dataset of raw data and ground truth data, Calculating features using the raw data; constructing an encoding model that outputs a code corresponding to the feature amount in response to input of the raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input of the code; A learning method for training the encoding model and the estimation model based on a relationship between the code and the feature amount so that the estimation result matches the ground truth data. (Appendix 13) Obtaining a dataset of raw and ground truth data; A process of calculating a feature amount using the raw data; A process of constructing an encoding model that outputs a code corresponding to the feature amount in response to input of the raw data, and an estimation model that outputs an estimation result corresponding to the raw data in response to input of the code; and training the encoding model and the estimation model based on the relationship between the code and the feature amount so that the estimation result matches the ground truth data. [Explanation of symbols]

[0271] 10, 20, 30, 40 Learning Device 11, 21, 31, 41 Acquisition section 12, 22, 32, 42 Feature calculation section 13, 23, 33, 43 Model Construction Section 15, 25, 35, 45 Learning processing section 50, 60, 70 Estimation System 51, 61, 71 Measuring equipment 53, 63, 73 Estimation device 131, 231, 331 Encoding section 133 Reconstruction Department 135, 235, 335 Estimation part 151, 251, 351 coding models 153 Reconstruction Model 155, 255, 355 Estimation Model 337 First Pre-Learning Section 338 Second Pre-Learning Section 357 First Pre-Trained Model 358 Second Pre-Trained Model 511, 611, 711 Acceleration Sensor 512, 612, 712 Angular Rate Sensor 513, 613, 713 Control section 515, 615, 715 Encoding section 517, 617, 717 Transmitter 531, 631, 731 Receiver 535, 635, 735 Estimation part 537, 637, 737 Output section 551, 651, 751 coding models 555, 655, 755 Estimation Models 616 Feature Extraction Unit 733 Reconstruction Department 753 Reconstruction Model

Claims

1. An acquisition means for acquiring a data set of raw data, which is sensor data relating to foot movement, and correct answer data, which is a physical condition corresponding to the raw data; a feature calculation means for calculating a feature used for estimating the physical condition based on medical knowledge from the raw data; a model construction means for constructing an encoding model which receives the raw data as an input and outputs a code corresponding to the feature amount, and an estimation model which receives the code as an input and outputs an estimation result corresponding to the raw data; A learning processing means for training the encoding model and the estimation model based on a relationship between the code and the feature amount so that the estimation result matches the correct answer data, The coding model is A learning device that outputs the code corresponding to the feature amount using constraint conditions based on the medical knowledge.

2. The learning processing means The learning device according to claim 1 , wherein the raw data is used as input, and the estimation model is trained so as to reduce mutual information between the code output from the encoding model and the feature calculated using the raw data.

3. The model construction means includes: constructing a first pre-learning model that receives the raw data as input and outputs a first conversion value corresponding to the feature; The learning processing means training the first pre-trained model so that the first transformation value matches the feature; Setting model parameters of the first pre-learning model after training as initial values ​​of the encoding model; The learning device according to claim 1 or 2, wherein the encoding model and the estimation model are trained so that the estimation result matches the ground truth data.

4. The model construction means includes: constructing a second pre-learning model that outputs a second conversion value corresponding to the raw data in response to input of the feature amount calculated using the raw data; The learning processing means training the second pre-trained model so that the second transformation value matches the estimation result; Setting model parameters of the second pre-learned model after training as initial values ​​of the estimation model; The learning device according to claim 3 , wherein the encoding model and the estimation model are trained so that the estimation result matches the ground truth data.

5. An estimation device in which an encoding model and an estimation model constructed by the learning device according to any one of claims 1 to 4 are implemented, at least one measuring instrument and the coding model; inputting raw data measured by the measuring device into the encoding model; a measurement device that receives the raw data as an input and transmits a code output from the encoding model; and the estimation model that receives the code transmitted from the measurement device; inputting the received code into the estimation model; an estimation device that receives the code as an input and outputs an estimation result output from the estimation model.

6. The measuring device includes: Calculating features using the raw data; The raw data is input, and the code output from the encoding model and the feature amount calculated using the raw data are transmitted to the estimation device; The estimation device includes: receiving the code and the feature amount transmitted from the measurement device; inputting the received code and the feature quantity into the estimation model; The estimation system according to claim 5 , wherein the code and the feature amount are input, and the estimation system outputs the estimation result output from the estimation model.

7. The estimation device includes: a reconstruction model that receives the code as an input and outputs a reconstructed feature corresponding to the original feature of the code; The code is used as an input, and a reconstructed feature output from the reconstructed model is input to the estimation model; The estimation system according to claim 5 or 6, wherein the reconstructed feature is input, and the estimation result output from the estimation model is output.

8. The measuring device includes: The device is worn by a user and measures sensor data related to the user's movements; The measured sensor data is input, and the code output from the encoding model is transmitted to the estimation device; The estimation device includes: receiving the code transmitted from the measuring device; The estimation system according to claim 6 or 7, wherein information relating to the estimation result is transmitted to a mobile terminal having a display means viewable by the user.

9. The computer Acquire a data set of raw data, which is sensor data related to foot movement, and ground truth data, which is a physical condition corresponding to the raw data; Calculating a feature amount used for estimating the physical condition based on medical knowledge from the raw data; constructing an encoding model that receives the raw data as an input and outputs a code corresponding to the feature amount, and an estimation model that receives the code as an input and outputs an estimation result corresponding to the raw data; training the encoding model and the estimation model based on a relationship between the code and the feature amount so that the estimation result matches the ground truth data; The coding model is A learning method that outputs the code corresponding to the feature amount by using constraint conditions based on the medical knowledge.

10. A computer comprising: A process of acquiring raw data, which is sensor data related to foot movement, and a data set of ground truth data, which is a physical condition corresponding to the raw data; A process of calculating, from the raw data, feature amounts used for estimating the physical condition based on medical knowledge; A process of constructing an encoding model that receives the raw data as an input and outputs a code corresponding to the feature amount, and an estimation model that receives the code as an input and outputs an estimation result corresponding to the raw data; training the encoding model and the estimation model based on a relationship between the code and the feature amount so that the estimation result matches the ground truth data; The coding model is A program that outputs the code corresponding to the feature amount by using constraints based on the medical knowledge.

Images