Information estimation device and information estimation method

Abstract

To reduce the load of calculation processing for a neural network, while maintaining an accuracy rate of estimation results without dropping the rate.SOLUTION: After a neural network is trained, the neural network comprising at least a convolution layer 12, a pooling layer 13, and a fully-connected layer 14, to set a trained neural network, a connection is made so that data output from the convolution layer 12 are supplied to the fully-connected layer 14, and learning data is input to the trained neural network to acquire data output from the fully-connected layer 14, as a heat map. A collection list creation unit 30 creates a collection list by extracting a ground-truth heat map corresponding to a ground-truth label. A correlation value calculation unit 50 calculates a correlation value which indicates correlation between model elements from the collection list. A principal model element extraction unit 60 extracts a plurality of principal model elements having correlation values equal to or smaller than a predetermined threshold and having low correlation with other model elements.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information estimation device and an information estimation method for estimating the state of a model to be estimated from time-series data of the model to be estimated, which is composed of a plurality of model elements, using a neural network. [Background technology]

[0002] In recent years, rapid advances in neural networks have made it possible to estimate a variety of things. Among these, technologies have been developed that use neural networks to estimate the behavior of humans, animals, and objects, or the state of an entire system, based on changes in the movements of humans, animals, and objects, or changes in individual observations from sensors installed in the system.

[0003] This neural network technology is configured such that, in an estimation target model composed of a plurality of model elements, time-series data including time changes of parameters associated with the plurality of model elements constituting the estimation target model is input to the neural network, and the state of the entire estimation target model is output as an estimation result.

[0004] A skeleton model is known as an example of an estimation target model.

[0005] In a skeletal model, as shown in Figure 1(a), the joint points (including individual parts such as the eyes, nose, and mouth) of a person, animal, or object are set as model elements, and the spatial coordinates of each joint point are set as parameters of the model elements. Skeletal models use time-series data containing temporal changes in the coordinates of each skeletal joint point, obtained by, for example, photographing a human using a camera. Skeletal model-based estimation technology inputs time-series data representing the movement of each skeletal joint point into a neural network, which outputs a score for each predefined skeletal behavior classification. The behavior with the highest score can be estimated as the behavior corresponding to the input data. The temporal changes in the coordinates of each skeletal joint point of a person, animal, or object represent the movement of each joint point over time, and the neural network can estimate the specific behavior of the entire skeletal model (the overall state of the model being estimated), such as running, eating, or sleeping, from the temporal changes in each joint point.

[0006] The positions and number of joint points (also called key points) of a skeletal model are not particularly limited, and are generally defined in various ways in data, devices, etc. Specifically, Open Pose, which defines 18 or 25 positions as joint points of a skeletal model, and Kinect (registered trademark), which defines 32 positions, are known.

[0007] As neural networks that handle skeletal models, the techniques described in Patent Document 1 and Non-Patent Documents 1 to 4, for example, are known.

[0008] A graph model is known as an example of an estimation target model.

[0009] In a graph model, as shown in Figure 1(b), objects and events are defined as model elements, and coordinates containing specific observation values ​​of the objects and events are defined as parameters of the model elements. A graph model uses time-series data containing temporal changes in the observation values ​​of objects and events, acquired, for example, by sensors implemented in the system. In estimation techniques using graph models, model elements such as objects and events are represented as nodes, and time-series data showing how the observation values ​​related to each node change is input to a neural network. A score is output for each predetermined classification of the overall system state, and the state with the highest score can be estimated as the state corresponding to the input data. Specifically, a graph model is established in which nodes representing individual phenomena such as temperature, time, and speed are connected. The specific state of the entire system (the overall state of the model being estimated), such as safety or danger, can be estimated from the temporal changes in the observation values ​​of each node.

[0010] In estimation techniques using neural networks, time-series data of the parameters of multiple model elements constituting a model to be estimated is prepared as input data to the neural network. Then, using the time-series data as input data, the neural network is trained to output an estimation result indicating which of the pre-set classifications the state of the entire model to be estimated falls into. For example, supervised learning is performed in which the neural network is trained with correct answers given to the training data.

[0011] For example, in the skeletal model shown in Figure 1(a), time-series data represented by a tensor consisting of "number of joint points" x "number of joint point coordinates (number of spatial coordinates such as 2D or 3D)" x "number of time frames" is prepared as input data for learning, and correct label data indicating which of a pre-determined list of actions is correct is prepared in correspondence with these input data.

[0012] Note that the coordinates referred to here are, for example, (x, y) or (x, y, likelihood) in two dimensions. x and y are the coordinates of the joint point. The likelihood is the likelihood of the coordinates, which takes values ​​between 0 and 1. If a joint point is not visible due to occlusion, the likelihood becomes zero, and the coordinates x and y are set to some appropriate value, for example, zero. In three dimensions, the coordinates are (x, y, z) or (x, y, z, likelihood).

[0013] Also, for example, in the graph model shown in Figure 1(b), time series data represented by a tensor consisting of "number of nodes" x "number of node parameters" x "number of time frames" is prepared as input data for learning, and correct answer label data indicating which of a pre-determined state list is correct is prepared corresponding to this input data. In the case of a graph model representing states, the parameters held by each node are state value (observation value) x or (x, likelihood). That is, a temperature node representing temperature t has t or (t, likelihood), and a humidity node representing humidity m has m or (m, likelihood), etc.

[0014] Then, the training data is input to the neural network, and training is performed so that correct output data is output based on the correct label data.

[0015] Furthermore, a test is usually conducted to input test data to evaluate the performance of the trained neural network. The test data includes time-series data and correct label data, just like the training data. During the test, it is confirmed whether the trained neural network can output correct estimation results.

[0016] By actually using a trained neural network that has been trained and tested as described above, it is possible to perform estimation processing on time series data obtained from an estimation target model in an unknown state, and estimate which of the predetermined classifications the state of the entire estimation target model falls into.

[0017] For example, in the case of the skeletal model shown in Figure 1(a), the movement of human joints can be observed in real time with a camera, and the observed data can be input into a trained neural network, allowing human behavior to be estimated at any time.

[0018] In addition, for example, in the case of the graph model shown in Figure 1(b), the state of the entire system can be estimated at any time by observing temperature, humidity, weather, air conditioning electricity costs, etc. in real time using sensors and inputting the observed data into a trained neural network. [Prior art documents] [Patent documents]

[0019] [Patent Document 1] Special Publication No. 2022-501103 [Non-patent literature]

[0020] [Non-Patent Document 1] Sijie Yan, Yuanjun Xiong and Dahua Lin. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition, AAAI, 2018 [Non-patent document 2] Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Junliang Xing, Jianru Xue, Nanning Zheng. Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition, CVPR2020 [Non-patent document 3] Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, Weiming Hu. Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition, ICCV2021 [Non-patent document 4] Yi-Fan Song, Zhang Zhang, Caifeng Shan, Liang Wang. Richly Activated Graph Convolutional Network for Robust Skeleton-based Action Recognition, IEEE T-CSVT, 2020 Summary of the Invention [Problem to be solved by the invention]

[0021] In estimation techniques using neural networks, a huge amount of data is input to an information estimation device (estimator) equipped with a neural network, so the neural network must perform considerable calculations. In particular, the calculation load on the neural network tends to increase as the number of model elements increases. For example, the more model elements constituting the model to be estimated and the more massive the input data, the greater the calculation load on the neural network becomes, resulting in problems such as long calculation times and the need for a computer with a large memory capacity.

[0022] For example, in the case of a skeletal model, if multiple people are being observed, or if even small elements such as the number of finger joints, facial expressions, and eye movements are included in the skeletal model, the number of joint points in the skeletal model and the number of parameters set for each joint point will increase significantly, posing a problem of increasing the computational load on the neural network.

[0023] Similarly, in the case of graph models, if the number of state factors that represent the system environment (e.g., temperature, humidity, brightness, speed, etc.) is increased, the number of nodes in the graph model and the number of parameters set for each node will increase significantly, resulting in a problem of increased computational load on the neural network.

[0024] In view of the problem that an increase in the number of model elements and parameters increases the computational load of the neural network, one possible solution is to remove some of the model elements and leave only the main model elements that affect the estimation of the model state.

[0025] For example, if the behavioral pattern to be estimated using a skeletal model is simple, such as when classifying limited behaviors, such as the behavioral patterns of automobile occupants, a full-body skeleton is unnecessary, and only the joint points of a part of the body or the upper body are expected to be important for behavior estimation. In this case, it is possible to reduce the computational load of the neural network by omitting some of the joint points of the skeletal model, which is input data.

[0026] However, it is not easy to actually determine which model elements (joint points) have a large impact and which have a small impact, and a guideline is needed to determine which model elements (joint points) are optimal to remove and which to keep.

[0027] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information estimation device and an information estimation method that can reduce the load of calculation processing on a neural network while maintaining the accuracy rate of the estimation results without reducing the accuracy rate. [Means for solving the problem]

[0028] In order to achieve the above object, according to the present invention, there is provided an information estimation device including a neural network capable of performing learning based on learning data in which time-series data in which parameters of a target model to be estimated, the time-series data being changed over time and a correct label assigned to the time-series data are paired, the device comprising: The neural network a convolution layer that converts the supplied data into data that represents the temporal and spatial influences of the parameters of the plurality of model elements on each other; A pooling layer that converts the supplied data into data consisting of specific feature dimensions; a fully connected layer that converts the supplied data into data having dimensions equal to the number of classifications of the state of the model to be estimated; and a heat map storage unit that connects the convolution layer so that data output from the convolution layer is supplied to the pooling layer and data output from the pooling layer is supplied to the fully connected layer, and then trains the neural network using the training data to set a trained neural network, and then connects the convolution layer so that data output from the convolution layer is supplied to the fully connected layer, inputs the training data to the trained neural network, and stores the data output from the fully connected layer as a heat map; a collection list creation unit that creates a collection list by extracting and storing correct heat maps corresponding to correct labels from the heat maps stored in the heat map storage unit; a correlation value calculation unit that calculates a correlation value indicating a correlation between each model element from the collection list created by the collection list creation unit; a main model element extraction unit that extracts, as a plurality of main model elements, a plurality of model elements whose correlation values ​​calculated by the correlation value calculation unit are equal to or less than a predetermined threshold and whose correlations with other model elements are low; and There is provided an information estimation device configured to reconstruct an estimation target model consisting of the plurality of main model elements extracted by the main model element extraction unit, and to re-learn the neural network using the reconstructed estimation target model to perform estimation processing.

[0029] Furthermore, in order to achieve the above object, according to the present invention, there is provided an information estimation method in an information estimation device including a neural network capable of performing learning based on learning data in which time-series data in which parameters of a target model to be estimated consisting of a plurality of model elements change over time and a correct label assigned to the time-series data are paired, the method comprising: The neural network a convolution layer that converts the supplied data into data that represents the temporal and spatial influences of the parameters of the plurality of model elements on each other; A pooling layer that converts the supplied data into data consisting of specific feature dimensions; a fully connected layer that converts the supplied data into data having dimensions equal to the number of classifications of the state of the model to be estimated; and a step of connecting the convolution layer so that data output from the convolution layer is supplied to the pooling layer and data output from the pooling layer is supplied to the fully connected layer, and training the neural network using the training data to set a trained neural network; a heat map storage step of, after setting the trained neural network, connecting the convolution layer so that data output from the fully connected layer is supplied to the fully connected layer, inputting training data into the trained neural network, and storing data output from the fully connected layer as a heat map in a heat map storage unit; a collection list creation step of creating a collection list in which correct heat maps corresponding to correct labels are extracted and stored from the heat maps stored in the heat map storage step; a correlation value calculation step of calculating a correlation value indicating a correlation between each model element from the collection list created in the collection list creation step; a main model element extraction step of extracting, as a plurality of main model elements, a plurality of model elements whose correlation values ​​calculated in the correlation value calculation step are equal to or less than a predetermined threshold and have low correlations with other model elements; an estimation step of reconstructing an estimation target model including the plurality of main model elements extracted in the main model element extraction step, and re-learning the neural network using the reconstructed estimation target model to perform estimation processing; An information estimation method is provided having: [Effects of the Invention]

[0030] According to the present invention, for an estimation target model composed of multiple model elements, the influence (contribution) of each model element on the estimation result of a neural network can be determined, major model elements that affect the estimation result of the neural network can be extracted, and model elements that have little effect on the estimation result of the neural network can be appropriately omitted. As a result, by performing neural network training and estimation using lightweight and simplified time-series data that includes only the parameters of the extracted major model elements, it is possible to obtain highly reliable estimation results with almost no decrease in the accuracy rate of the estimation results, and to reduce the computational load on the neural network (for example, by shortening the calculation time and reducing the memory usage). [Brief explanation of the drawings]

[0031] [Figure 1] 1A and 1B are diagrams showing examples of estimation target models applied to neural networks in the present invention and the prior art, where FIG. 1A is a diagram showing an example of a human skeletal model, and FIG. 1B is a diagram showing an example of a graph model representing an environment. [Figure 2] 1 is a block diagram showing an example of a configuration of an information estimation device according to a first embodiment of the present invention. [Figure 3] FIG. 3 is a diagram for explaining the processing in the learning stage in the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram for explaining the processing up to the creation of a collection list at the main model element extraction stage in the first embodiment of the present invention. [Figure 5] FIG. 3 is a diagram showing an example of a collection list created in the first embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing an example of a correlation table (left half) obtained by calculation processing by a correlation value calculation unit at the main model element extraction stage in the first embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an example of a correlation table (right half) obtained by calculation processing by a correlation value calculation unit at the main model element extraction stage in the first embodiment of the present invention. [Figure 8] 10 is a flowchart showing an example of processing executed by a main model element extraction unit at a main model element extraction stage in the first embodiment of the present invention. [Figure 9] 1 is a table showing the experimental results of examples and comparative examples in experiments carried out in relation to the present invention. [Figure 10] FIG. 3 is a diagram for explaining an estimation process in the first embodiment of the present invention. [Figure 11] FIG. 10 is a diagram for explaining a non-response period related to the second embodiment of the present invention. [Figure 12] 10 is a flowchart showing an example of a no-response period deletion process executed by a collection list creation unit in the second embodiment of the present invention. [Figure 13] FIG. 10 is a diagram for explaining a collection list created by a collection list creation unit in the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] Hereinafter, first and second embodiments of the present invention will be described with reference to the drawings.

[0033] <Summary of the Invention> The present invention relates to a technology for estimating the state of an entire model to be estimated, which is composed of multiple model elements, using a neural network. The present invention determines the influence (contribution) of each model element on the estimation result of the neural network, extracts the main model elements that affect the estimation result of the neural network, and appropriately omits model elements that have little influence on the estimation result of the neural network. As a result, by performing neural network training and estimation using reduced and simplified time-series data that includes only the parameters of the extracted main model elements, it is possible to obtain highly reliable estimation results with almost no drop in the accuracy rate of the estimation results, and also to reduce the computational load on the neural network (e.g., shortening the calculation time and reducing memory usage).

[0034] The estimation target model used in the present invention is composed of multiple model elements, each of which has one or more parameters set thereto. Although not particularly limited, for example, the skeletal model shown in FIG. 1(a) or the graph model shown in FIG. 1(b) described above can be used.

[0035] In the case of a skeletal model, the model elements are the joint points of the human skeleton, and the parameters set for each model element are the spatial coordinates of each joint point. For example, by capturing human movement with a camera or other device, skeletal model data can be obtained from the image. Furthermore, the state of the entire estimation target model obtained as an estimation result represents human behavior (running, eating, sleeping, etc.).

[0036] In the case of a graph model, model elements are nodes that represent events in a system environment, and the parameters set for each model element are information related to each event. For example, graph model data can be obtained by using the observed values ​​(sensor data) of each event detected by a sensor or other device installed in the system as information related to each event. Furthermore, the state of the entire estimation target model obtained as an estimation result is the state of the entire system environment (safe, dangerous, etc.).

[0037] In the following, a skeleton model will be mainly described as an example of a model to be estimated, but the model to be estimated used in the present invention is not limited to a skeleton model.

[0038] First Embodiment A first embodiment of the present invention will be described.

[0039] <Configuration of information estimation device> First, the overall configuration of the information estimation device 1 in the first embodiment of the present invention will be described. Fig. 2 is a functional block diagram showing an example of the overall configuration of the information estimation device 1 in the first embodiment of the present invention.

[0040] 2 merely represents functions related to the present invention, and in actual implementation, the functions may be realized by hardware, software, firmware, or any combination thereof. Functions implemented in software are stored on any computer-readable medium as one or more instructions or codes, and these instructions or codes can be executed by a hardware-based processing unit such as a CPU (Central Processing Unit). Furthermore, the functions related to the present invention may be realized by various devices including ICs (Integrated Circuits), IC chipsets, etc.

[0041] As an example, the information estimation device 1 in the first embodiment of the present invention is realized by a computer capable of executing a program in which a processing procedure is written in a predetermined programming language. The computer realizing the information estimation device 1 in the first embodiment of the present invention is configured to include, as hardware, a processor such as a CPU that executes the program, storage such as an HDD (Hard Disk Drive) or an SDD (Solid State Drive) that stores the program and data to be processed, memory such as a RAM (Random Access Memory) that temporarily stores the program and data being processed, operation input devices such as a mouse and keyboard that accept operation inputs by the user, and a display that provides visual information to the user. The storage may be a portable storage medium such as a USB (Universal Serial Bus) memory, or may be network storage connected via a network.

[0042] The information estimation device 1 shown in FIG. 2 is configured to include an estimation processing unit 10, a heat map storage unit 20, a collection list creation unit 30, a collection list storage unit 40, a correlation value calculation unit 50, and a main model element extraction unit 60.

[0043] The estimation processing unit 10 includes a neural network with a hierarchical structure in which multiple layers are arranged, and is configured to perform calculations on input data based on a predetermined mathematical model and output the results obtained by the calculations. Specifically, the estimation processing unit 10 is a classifier that performs supervised learning, and includes a neural network with a hierarchical structure including any layer 11 including an input layer and an intermediate layer, a convolution layer 12, a pooling layer 13, a fully connected layer 14, and an output layer 15 including a softmax layer.

[0044] 2 represents one or more layers from the input layer of the neural network to the layer preceding the convolution layer 12. The structure of the optional layer 11 is not particularly limited, and may be, for example, a structure in which multiple convolution layers or pooling layers are arranged.

[0045] A convolutional layer 12 is placed after any layer 11. The convolutional layer 12 is a layer that expresses the effect of the parameters of multiple model elements influencing each other temporally and spatially, and is configured to perform convolution of data supplied from the layer placed in the previous stage.

[0046] In the convolution layer 12, the coordinate dimensions of the supplied data, i.e., two or three dimensions, are expanded into vectors with even more dimensions called features. In the convolution layer 12, these features are added to the spatial neighborhood, i.e., the feature of a given joint point is added to the feature of nearby joint points connected by bones, or the feature of a joint point at a certain time is added to the feature of the same joint point at a previous or subsequent time on the time axis. In other words, in the convolution layer 12, processing is performed to express the effect that the feature of a given joint point affects the feature of other joint points located in its spatial neighborhood, and the effect that the feature of a given joint point affects the feature of the same joint point located in its temporal neighborhood.

[0047] A pooling layer 13 is arranged after the convolution layer 12. The pooling layer 13 is a layer that combines the temporal and spatial influences of the parameters of multiple model elements into a vector consisting of a specific feature dimension, and is configured to map the data supplied from the convolution layer 12 arranged before it.

[0048] A fully connected layer 14 is arranged after the pooling layer 13. The fully connected layer 14 is a layer that converts a vector consisting of specific feature dimensions into a vector consisting of the dimensions of the number of classifications of the state of the estimation model, and is configured to reflect the correlation of all variables in the vector consisting of specific feature dimensions and output the correlation as a vector consisting of the dimensions of the number of classifications.

[0049] An output layer 15 is arranged after the fully connected layer 14. The output layer 15 is configured, for example, by a softmax layer, and is configured to normalize and emphasize the data supplied from the fully connected layer 14 and output a score for each pre-set classification as an estimation result in the neural network. The score (estimation result) output from the softmax layer may be stored in, for example, a storage device or an external secondary storage device, may be processed for display and displayed on a display screen, or may be transmitted to another device connected via a network.

[0050] The neural network itself provided in the estimation processing unit 10 has a configuration similar to that of a hierarchical-structured neural network in the prior art. That is, similar to the prior art, the estimation processing unit 10 is configured to input time-series data including time changes of parameters associated with a plurality of model elements constituting an estimation target model to the neural network, in the estimation target model composed of the plurality of model elements, and to output the state of the entire estimation target model as an estimation result.

[0051] However, in the main model element extraction stage described later, the pooling layer 13 is deleted (skipped), and as shown by data line L1 in Fig. 2, the data output by the convolution layer 12 is supplied to the fully connected layer 14. Also, in the main model element extraction stage described later, as shown by data line L2 in Fig. 2, the data output by the fully connected layer 14 is stored as a heatmap in the heatmap storage unit 20.

[0052] The estimation processing unit 10 is realized, for example, by the processor of the information estimation device 1 executing a predetermined neural network program stored in advance in storage. The estimation processing unit 10 can realize a neural network desired by the user by setting the neural network settings, hierarchical structure, etc., in advance.

[0053] The heat map storage unit 20 has a function of collecting and storing, as a heat map, data output by the fully connected layer 14 in a main model element extraction stage described later. The heat map storage unit 20 is configured by, for example, a storage of the information estimation device 1.

[0054] The collection list creation unit 30 has a function of extracting a heat map (correct heat map) corresponding to a correct label from the heat map stored in the heat map storage unit 20 in a main model element extraction stage described below. The collection list creation unit 30 is realized, for example, by the processor of the information estimation device 1 executing a predetermined collection list creation program stored in advance in storage. Note that the collection list creation unit 30 may be configured to directly acquire the heat map, which is data output by the fully connected layer 14, and extract a heat map (correct heat map) corresponding to a correct label from this heat map. Furthermore, the collection list creation unit 30 may be configured to acquire the collection list stored in the collection list storage unit 40 and recreate the collection list based on this collection list.

[0055] The collection list storage unit 40 has a function of storing the collection list created by the collection list creation unit 30 in the main model element extraction stage described later. The collection list storage unit 40 is configured by a storage of the information estimation device 1, for example.

[0056] The correlation value calculation unit 50 has a function of calculating correlation values ​​between the model elements of the estimation target model from the collection list stored in the collection list storage unit 40 in the main model element extraction stage described below. The correlation value calculation unit 50 is realized, for example, by the processor of the information estimation device 1 executing a predetermined correlation value calculation program stored in advance in storage.

[0057] The main model element extraction unit 60 has a function of extracting main model elements that affect the estimation results of the neural network by referring to the correlation values ​​between the model elements calculated by the correlation value calculation unit 50 in the main model element extraction stage described below. In this specification, "main" means that the main model elements have a high degree of influence (contribution) on the estimation results of the neural network. The main model elements output from the main model element extraction unit 60 may be stored in, for example, a storage device or an external secondary storage device, may be processed for display and displayed on a display screen, or may be transmitted to another device connected via a network.

[0058] <Processing in the information estimation device> The information estimation device 1 in the first embodiment of the present invention is set to a state in which the state of the estimation target model can actually be estimated by executing processes in three stages: a learning stage, a main model element extraction stage, and a re-learning stage.

[0059] (Learning stage) In the information estimation device 1 according to the first embodiment of the present invention, a learning stage is first performed. For example, data to which correct labels have been assigned is divided into learning data and test data, and supervised learning is performed in which an untrained neural network is trained with the learning data assigned with correct answers. The learning data includes the time-series data of the skeletal model described above, i.e., input data expressed as a tensor consisting of "number of time frames" x "number of articulation points" x "number of coordinates of articulation points (number of spatial coordinates, such as two-dimensional or three-dimensional)", and correct-answer label data indicating which of a predetermined behavior list is correct corresponding to this input data. The learning data is then input to the untrained neural network, and learning is performed so that correct output data is output.

[0060] Furthermore, as with the prior art, a test is conducted by inputting test data to evaluate the performance of the trained neural network. The test data, like the training data, includes time-series data and correct label data. During the test, it is confirmed whether the trained neural network can output correct estimation results, and it is possible to evaluate whether the training has been performed appropriately.

[0061] Here, the process of the learning stage in the first embodiment of the present invention will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining the process of the learning stage in the first embodiment of the present invention.

[0062] In the learning stage, learning data having the dimensions of "number of time frames" x "number of joint points" x "number of coordinates (number of spatial coordinates such as 2D or 3D)" is input as input data to an unlearned neural network. When the learning data is input, calculation processing is performed by an optional layer 11, and data having the dimensions of "number of time frames" x "number of joint points" x "number of features" is input to the convolution layer 12.

[0063] The data input to the convolution layer 12, which has the dimensions of "time frame" x "number of joint points" x "number of features," is converted by the convolution process in the convolution layer 12 into data having the dimensions of "time frame" x "number of joint points" x "number of features," and then output.

[0064] The data output by the convolution layer 12 is input to the pooling layer 13. In the pooling layer 13, the dimensions of the number of joint points and time are collapsed by calculating the average value or extracting the maximum value of the data output by the convolution layer 12, and the data is compiled into a feature vector having the number of dimensions of a single feature, and data having the dimension of the "number of features" is output.

[0065] The data output by the pooling layer 13 is input to the fully connected layer 14. In the fully connected layer 14, the supplied data is converted into data having dimensions equal to the number of classifications of the state of the model to be estimated, and output. In the fully connected layer 14, for example, the data output by the pooling layer 13 is organized into a vector having dimensions equal to the number of single feature quantities, and in the fully connected layer 14, the data output by the pooling layer 13 is converted into data having dimensions equal to the "number of estimated actions" (number of classifications) that reflects the correlation of each value, and output.

[0066] The data output by the fully connected layer 14 is input to the output layer 15, which is composed of a softmax layer. In the output layer 15, for example, the data output by the fully connected layer 14 is normalized and emphasized, converted into data having the dimension of "estimated number of actions" that includes the estimated probability (score) of each action, and output. By referencing the score, which is the estimated probability of each action included in this data having the dimension of "estimated number of actions," it is possible to infer what action the input data, having the dimension of "time frame" x "number of joint points" x "number of coordinates," was obtained from.

[0067] During training, backpropagation is performed to maximize the score output from the correct classification, and the parameters inside the neural network are updated. This allows training to be performed so that the correct output data is output.

[0068] In the learning stage, the same processing as that performed in a normal neural network is performed. In the neural network according to the prior art, when the learning stage is completed, a trained neural network is set, and an estimation process is actually performed using this trained neural network, and the classification with the maximum score can be estimated as the state (behavior) of the skeleton model.

[0069] In estimation processing using a trained neural network according to the prior art, data having the dimensions of "number of time frames" x "number of joint points" x "number of coordinates" is used as input data. However, the larger the dimension of "number of time frames" x "number of joint points" x "number of coordinates," the greater the problem of the load on the neural network's calculation processing. The present invention has been made in consideration of this problem, and instead of using the trained neural network obtained in the above-mentioned training stage as is for estimation processing, a main model element extraction stage and a re-training stage are further performed to reduce the number of model elements (number of joint points) and the number of dimensions of the input data, thereby reducing the load on the neural network's calculation processing.

[0070] (Major model element extraction stage) In the information estimation device 1 according to the first embodiment of the present invention, a main model element extraction step is then performed on the trained neural network set up in the above-described training step. In this main model element extraction step, the trained neural network performs a process of visualizing which parts of the neural network are activated in response to input data and contribute to the correct estimated output.

[0071] For example, Non-Patent Document 4 above proposes a calculation method for neural network processing that visualizes, as a heatmap value, which joints in input data at which times have a significant effect on the estimation score in the neural network. In the main model element extraction stage, the method proposed in Non-Patent Document 4 is used to set the neural network to output a heatmap, thereby confirming which joint points are important joint points (main model elements) for which action estimation and which joint points can be omitted.

[0072] In the main model element extraction stage, the pooling layer 13 in the estimation processing unit 10 shown in Fig. 2 is deleted (skipped), and the output from the convolution layer 12 is set to be supplied to the fully connected layer 14. The pooling layer 13 arranged after the convolution layer 12 performs processing to reduce the number of dimensions by collapsing the dimensions for each joint point and each time for the data output by the convolution layer 12, but by deleting this pooling layer 13, data can be supplied from the convolution layer 12 to the fully connected layer 14 without collapsing the dimensions for each joint point and each time included in the data output by the convolution layer 12.

[0073] The fully connected layer 14 outputs a heat map value, which is data that reflects the correlation of all data output by the convolution layer 12 for each input joint point and for each time. The data output by the fully connected layer 14 is not a one-dimensional array of behavior classification scores, but data with the dimensions of "number of time frames" x "number of joint points" x "number of estimated behaviors." This can be seen as a score that contributes to the output of a certain estimated behavior for a certain input data, for a certain given joint point or time. In other words, the heat map, which is data output by the fully connected layer 14, represents the degree of contribution to the correct estimation score for a certain input data. By referring to this heat map, it is possible to see which joint points contribute to the score for which behavior for a certain snapshot at a certain time.

[0074] <Creating a collection list> Hereinafter, the processing up to the creation of a collection list at the main model element extraction stage in the first embodiment of the present invention will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining the processing up to the creation of a collection list at the main model element extraction stage in the first embodiment of the present invention.

[0075] As shown in Figure 4, in the main model element extraction stage, training data or test data (data with a given correct answer) with dimensions of "number of time frames" x "number of joint points" x "number of coordinates" is input as input data to the trained neural network. When the training data or test data is input, calculation processing is performed by an optional layer 11, and data with dimensions of "number of time frames" x "number of joint points" x "number of features" is input to the convolution layer 12.

[0076] In the convolution layer 12, convolution processing is performed on data having the dimensions of "number of time frames" x "number of joint points" x "number of features" that is input to the convolution layer 12. As a result, data having the dimensions of "number of time frames" x "number of joint points" x "number of features" is output from the convolution layer 12.

[0077] The data output by the convolution layer 12 skips the pooling layer 13 and is input to the fully connected layer 14. In the fully connected layer 14, the data output by the convolution layer 12 is converted into data (heat map) having the dimensions of "number of time frames" x "number of joint points" x "number of estimated actions" that reflects the correlation between each value, and is output. The data having the dimensions of "time frame" x "number of joint points" x "number of estimated actions" is stored in the heat map storage unit 20 as a heat map.

[0078] Next, the extraction list creation unit 30 extracts data on actions corresponding to the correct labels assigned to the input data from the heat maps having the dimensions of “number of time frames” × “number of joint points” × “number of estimated actions,” i.e., the correct heat map having the dimensions of “number of time frames” × “number of joint points” × “1 dimension (action corresponding to the correct label),” adds it to the collection list, and stores it in the collection list storage unit 40.

[0079] In this way, the collection list stored in the collection list storage unit 40 accumulates data having the dimension of "number of time frames" x "number of joint points" as data representing actions corresponding to the correct labels.

[0080] Fig. 5 is a diagram showing an example of a collection list created in the first embodiment of the present invention. In Fig. 5, N columns (total number of joint points N) containing all joint points are arranged along the horizontal axis, and T rows x number of data (total number of time frames T_total) containing snapshots of the heat map values ​​of all N joint points at each time are arranged along the vertical axis.

[0081] The collected list is updated as rows, representing the correct heatmaps for each joint point at various times. In other words, in the table shown in Figure 5, as data is collected at each time, rows are added downward in the collected list shown in Figure 5.

[0082] For example, suppose there are 100 sets (100 data points) of data with 120 time frames T and 25 joint points N. This means that 100 sets of time-series data (1 set) have been prepared, obtained by photographing a person with 25 joint points for 4 seconds at 30 frames per second (30 bps). In this case, the collection list is a two-dimensional table with 120 time frames x 100 data sets = 12,000 snapshots (total number of data points T_total = 12,000) arranged vertically, and the number of all joint points (25) arranged horizontally. The heat map values ​​corresponding to each joint point and time are stored in this table.

[0083] <Creating a correlation table> The information estimation device 1 according to the first embodiment of the present invention includes a correlation value calculation unit 50. The correlation value calculation unit 50 is configured to read a collection list stored in the collection list storage unit 40 and calculate a correlation value between N joint points in a column from the collection list. The correlation value can be calculated as a value for each pair of any two of the N points. For example, a correlation value r between a heat map value x of a certain joint point X and a heat map value y of another joint point Y collected for T time periods can be calculated as follows:

[0084] The average of the heat map value x of joint X in T_total snapshots can be calculated using the following formula: T_total is the total number of data, and is T x number of data.

[0085]

number

[0086] The average of the heat map value y of joint Y in T_total snapshots can be calculated using the following formula:

[0087]

number

[0088] At this time, the correlation value r between the heat map values ​​of joint point X and joint point Y taking into account the data for all T_total time periods is xy can be calculated by the following formula:

[0089]

number

[0090] Regarding the method of calculating the correlation value for the number of input data, for example, one correlation value for each input data for the number T of time frames may be calculated, and then the correlation values ​​for all input data may be calculated by summing them up later. Alternatively, [number of input data] x [number of time frames] may be collected at once and the correlation values ​​within them may be calculated.

[0091] The following describes the correlation values ​​obtained by the calculation process by the correlation value calculation unit 50. Figures 6 and 7 are diagrams showing the left and right halves of a correlation table obtained by the calculation process by the correlation value calculation unit 50 at the stage of extracting main model elements in the first embodiment of the present invention. Here, due to space limitations, one correlation table is divided into left and right halves, and one correlation table is obtained by combining the left half of the correlation table shown in Figure 6 and the right half of the correlation table shown in Figure 7.

[0092] The correlation values ​​obtained by the calculation process by the correlation value calculation unit 50 can be expressed in the form of a correlation table including the correlation values ​​between each joint point, as shown in FIGS. 6 and 7. The horizontal and vertical axes of the correlation table each list all of the joint points. The field where the horizontal and vertical axes intersect stores the correlation value between the heat map value of the joint point on the horizontal axis and the heat map value of the joint point on the vertical axis (the correlation value r obtained by the above-mentioned formula). In the correlation tables shown in FIGS. 6 and 7, the values ​​on the diagonal line from the upper left to the lower right are the autocorrelation value = "1.00", and the values ​​on the upper right and lower left sides of this diagonal line are symmetrical and have the same value.

[0093] <Main model element extraction process> The information estimation device 1 according to the first embodiment of the present invention includes a main model element extraction unit 60. The main model element extraction unit 60 is configured to extract joint points that are weakly correlated with other joint points, i.e., joint points that are independent and not linked to other joint points, from a correlation table including correlation values ​​calculated by the correlation value calculation unit 50. In the correlation table, for example, when a joint point in a specific column is focused on, the correlation values ​​between this joint point in the specific column and multiple other joint points are listed side by side. If all of the correlation values ​​between the joint point in the specific column and the other joint points are lower than a predetermined threshold, the joint point in the specific column has a low correlation with any of the other joint points, and can be determined to be a main joint point that functions independently of all the other joint points.

[0094] Hereinafter, the main model element extraction processing executed in the main model element extraction unit 60 of the information estimation device 1 in the first embodiment of the present invention will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the main model element extraction processing executed in the main model element extraction unit 60 of the information estimation device 1 in the first embodiment of the present invention.

[0095] As shown in FIG. 8, the main model element extracting unit 60 first acquires the collection list stored in the collection list storage unit 40 (step S1).

[0096] Next, the main model element extraction unit 60 selects a specific joint point i (for example, i = 1 in the initial processing) (step S2), and checks whether there are any other joint points for which correlation has not yet been checked (step S3).

[0097] If another joint point exists, the joint point j (j≠i) is selected (step S4). The main model element extraction unit 60 refers to the correlation value between the heat map value of the specific joint point i and the heat map value of another joint point j (the correlation value between the heat map values ​​stored in a specific field of the correlation table shown in FIGS. 6 and 7), and compares the magnitude relationship of the correlation value with a predetermined threshold value set in advance (step S5). In step S5, for example, it is confirmed whether the correlation value is equal to or less than the threshold value (correlation value≦threshold value).

[0098] If the correlation value is equal to or less than the threshold value in step S5 (correlation value≦threshold value), there is a possibility that a specific joint point i is a main model element. The main model element extraction unit 60 returns to step S3 again to perform the process and checks whether there are any other joint points for which correlation has not yet been checked (joint points for which the correlation value has not been checked in step S5) (step S3). If there are any such other joint points, it selects that joint point j (j≠i) (step S4) and performs a process of comparing the correlation value with a predetermined threshold value (step S5).

[0099] That is, in the processing of steps S3 to S5, attention is focused on a specific joint point i, and it is confirmed whether all correlation values ​​between the heat map value of the specific joint point i and the heat map values ​​of other joint points j (j≠i) are equal to or less than a threshold. If the correlation values ​​between the heat map value of the specific joint point i and the heat map values ​​of all other joint points j (j≠i) are equal to or less than a threshold, the specific joint point i is determined to be a main model element independent of the other joint points j (step S6), and the processing for the specific joint point i is terminated.

[0100] On the other hand, if a correlation value that is not less than the threshold value exists in step S5 (correlation value>threshold value), it is determined that the specific joint point i is not a major model element at that time (step S7), and processing for the specific joint point i is terminated.

[0101] Next, when the processing for specific joint point i is completed in steps S6 and S7, the main model element extraction unit 60 then checks whether there is another joint point that has not been selected as a specific joint point (step S8). If there is another joint point that has not been selected as a specific joint point, it selects another joint point (for example, i=i+1) as a specific joint point (step S9) and performs the processing from step S2 onwards again. On the other hand, if there is no other joint point that has not been selected as a specific joint point, that is, if all joint points have been selected as specific joint points, the main model element extraction processing ends.

[0102] The threshold value referenced in step S5 can be set by the user before executing the main model component extraction process. By appropriately adjusting the value set as the threshold value, it is possible to increase or decrease the number of main model elements extracted in the main model component extraction process. If the number of main model elements ultimately extracted is fewer than the desired number, the threshold value is decreased and the main model component extraction process is executed again. If the number of main model elements ultimately extracted is more than the desired number, the threshold value is increased and the main model component extraction process is executed again. This makes it possible to bring the number of main model elements extracted in the main model component extraction process closer to the desired number.

[0103] As an example, suppose that the main model element extraction process is performed on the correlation tables shown in Figures 6 and 7. In this case, if the threshold value, which is the reference value for determining whether an element is a main model element, is set to "0.87," the ten joint points marked with "■" among the joint points shown in Figures 6 and 7, namely, "0: Nose," "1: Neck," "3: RElbow," "4: RWrist," "6: LElbow," "7: LWrist," "15: REye," "16: LEye," "17: REar," and "18: LEar," will be extracted as main model elements.

[0104] (Relearning stage) The main model elements extracted in the main model element extraction process have low correlation with other joint points, and can be considered to be independent elements that greatly contribute to the state estimation of the entire estimation target model. In the information estimation device 1 in the first embodiment of the present invention, the trained neural network set in the above-mentioned training stage is temporarily reset to an untrained neural network, and elements other than the main model elements that were not extracted in the main model element extraction process are removed, and training of the untrained neural network is performed again using only the main model elements extracted in the main model element extraction process.

[0105] Specifically, elements other than the main model elements (main joint points) are thinned out from the data with the correct labels used in the training stage, leaving only the parameters related to the main model elements. This data is then divided into training data and test data, and supervised learning is performed, in which an untrained neural network is trained with the training data provided with the correct answers. The training data includes time-series data of a skeletal model composed of main joint points, i.e., input data expressed as a tensor consisting of "number of time frames" x "number of main joint points" x "number of coordinates of the main joint points (spatial coordinate number, such as 2D or 3D)," and correct label data indicating which of a predetermined list of actions is correct for this input data. The training data is then input to an untrained neural network, and training is performed so that correct output data is output.

[0106] In the re-learning stage, the neural network undergoes the same training process as in the training stage. However, in the training stage, input data with dimensions proportional to the "number of joint points" is used. In contrast, in the re-learning stage, the "number of joint points" is reduced to the "number of major joint points," significantly reducing the number of parameters included in the input data, resulting in lighter and simpler input data compared to the training stage. Specifically, while the training stage had 25 joint points, the re-learning stage uses only the 10 major joint points (major model elements) shown in the correlation tables in Figures 6 and 7. In this way, by leaving only the major model elements that have a significant influence (contribution) on the estimation results and thinning out the other model elements, the computational load on the neural network can be significantly reduced.

[0107] Similar to the training stage described above, a test is conducted by inputting test data to evaluate the performance of the trained neural network. The test data, like the training data, includes time-series data and correct label data. During the test, it is confirmed whether the trained neural network can output correct estimation results, making it possible to evaluate whether training has been carried out appropriately. Note that this test data has also been made lighter and simplified by reducing the "joint points" to "major joint points," which significantly reduces the computational load on the neural network.

[0108] <Example> As an experiment to confirm the effect of the present invention, the inventor constructed a neural network in a computer and performed training and evaluation of the neural network. In this experiment, the CTR-GCN described in Non-Patent Document 3 was used as the neural network, and training and evaluation of the neural network were performed using a total of 120 time frames for a skeletal model of 25 joint points with two-dimensional coordinates. Figure 9 shows the experimental results of the example and comparative example in the experiment conducted in relation to the present invention.

[0109] In this experiment, as an example, a learning stage and a main model element extraction stage were carried out, and 10 main joint points were extracted from 25 joint points as shown in the correlation tables of Figures 6 and 7, and the neural network was trained and evaluated in the re-learning stage. Also, as a comparative example to the present invention, only the above-mentioned learning stage was carried out, and the neural network was trained and evaluated without thinning out the 25 joint points.

[0110] In the comparative example, the number of joint points in the input data is 25, whereas in the example, the number of joint points in the input data is reduced to 10 (the number of major joint points). In the CTR-GCN used in this experiment, calculation processing is required in a certain layer with a memory amount proportional to the square of the number of joint points, and the calculation processing in the comparative example requires a memory amount proportional to the square of the number of joint points (25 2 ), whereas the calculation process in this embodiment requires a memory capacity of 10 times the number of joint points (10 2 ) can be reduced to a memory amount of 1. That is, as shown in FIG. 9, if the maximum memory required for the neural network in the example is 1, the calculation processing in the comparative example is 6.25, and the maximum memory required for the neural network in the example can be reduced to approximately 16% of the maximum memory required for the neural network in the comparative example, thereby realizing memory saving. Furthermore, the calculation amount in the example can be significantly reduced compared to the calculation amount in the comparative example, and a reduction in calculation time can also be realized.

[0111] 9, the accuracy rate of behavior estimation was 78.3% in the comparative example, while it was 76.18% in the example. Despite the large reduction in the number of joint points, the example was able to obtain a value almost the same as the accuracy rate of the comparative example.

[0112] Here, the estimation process in the first embodiment of the present invention will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining the estimation process in the first embodiment of the present invention.

[0113] When performing estimation processing using the information estimation device 1 in the first embodiment of the present invention, estimation target data having the dimensions of "number of time frames" x "number of major joint points" x "number of coordinates of major joint points (number of spatial coordinates, such as two-dimensional or three-dimensional)" is input as input data to the trained neural network set in the above-mentioned re-learning stage. When the estimation target data is input, calculation processing is performed by an arbitrary layer 11, and data having the dimensions of "number of time frames" x "number of major joint points" x "number of feature amounts" is input to the convolution layer 12.

[0114] The data input to the convolution layer 12, which has the dimensions of "time frame" x "number of major joint points" x "number of features," is converted by the convolution process in the convolution layer 12 into data having the dimensions of "time frame" x "number of major joint points" x "number of features," and is then output.

[0115] The data output by the convolution layer 12 is input to the pooling layer 13. In the pooling layer 13, the dimensions of the number of joint points and time are collapsed by calculating the average value or extracting the maximum value of the data output by the convolution layer 12, and the data is converted into data having the dimension of the "number of features" and then output.

[0116] The data output by the pooling layer 13 is input to the fully connected layer 14. In the fully connected layer 14, the data output by the pooling layer 13 is converted into data having the dimension of "estimated number of actions" (number of categories) that reflects the correlation of each value, and is output.

[0117] The data output by the fully connected layer 14 is input to the output layer 15, which is composed of a softmax layer. In the softmax layer, for example, the data output by the fully connected layer 14 is normalized and converted into data having the dimension of "estimated number of actions" including the estimated probability (score) of each action, and then output. By referencing the score, which is the estimated probability of each action included in this data having the dimension of "estimated number of actions," it is possible to infer what kind of action the estimation target data, which has the dimension of "number of time frames" x "number of major joint points" x "number of coordinates," was obtained from.

[0118] As described above, the information estimation device 1 according to the first embodiment of the present invention uses the trained neural network established by the re-learning stage described above to perform an estimation process on time-series data obtained from an estimation target model in an unknown state, thereby estimating which of the predetermined classifications the state of the entire estimation target model falls into. In the estimation process, data expressed as a tensor consisting of the number of time frames x the number of major joint points x the number of coordinates of the major joint points (two-dimensional, three-dimensional, or other spatial coordinate numbers) is used as input data, and an estimation result specifying the state (behavior) of the skeletal model is output based on this input data. In other words, the information estimation device 1 according to the first embodiment of the present invention can reduce the computational load of the neural network while maintaining almost no drop in the accuracy rate of the estimation result by inputting lighter and simplified data to the trained network established by the re-learning stage described above.

[0119] Second Embodiment A second embodiment of the present invention will now be described.

[0120] In the first embodiment of the present invention described above, in order to investigate the correlation between model elements such as joint points and nodes, a collection list is created that collects the heat map values ​​of all model elements in time-series data given a correct answer, such as training data or test data, and correlation values ​​are calculated from the correlation list that includes the heat map values ​​of all model elements.

[0121] In contrast, in the second embodiment of the present invention, in order to further improve the accuracy of the correlation value, it is proposed to create a collection list containing only the heat map values ​​of significant data in the time series data and calculate the correlation value.

[0122] In the first embodiment of the present invention described above, as shown in FIG. 4, time-series data to which correct answers have been assigned, such as training data or test data, is input to a trained neural network, and the data output by the fully connected layer 14 is collected as a heat map.

[0123] However, the quality of training data and test data varies. When creating training data and test data, time-series data is obtained by capturing images of a person performing a predetermined target behavior using a camera or the like for a predetermined period of time. However, the person does not always perform the predetermined target behavior throughout the entire period of time. For example, the start or end of the recording may include a moment when no action is being performed (hereinafter referred to as a non-reaction period).

[0124] The unresponsive period will now be described. FIG. 11 is a diagram illustrating the "unresponsive period" that is the focus of the second embodiment of the present invention. The center of FIG. 11 shows a two-dimensional map of heat map values ​​for each joint point obtained by capturing a total of 120 time frames by filming a human's pointing behavior at 30 bps for 4 seconds and using a skeletal model that defines 35 joint points (three-dimensional spatial coordinates) from these 120 time frames. In this map, the vertical axis represents time and the horizontal axis represents joint points. At the point where the vertical axis represents time and the horizontal axis represents joint points, the heat map value of a specific joint point at a specific time is represented by brightness (darker means more inactive, brighter means more active).

[0125] As shown in FIG. 11, even if a pointing behavior lasts for, for example, four seconds, the pointing behavior is not necessarily performed for the entire four seconds. For example, in the map shown in FIG. 11, the first second includes an unresponsive period during which no behavior occurs. The learning data and test data are data that include such unresponsive periods. In the first embodiment described above, heat map values ​​are collected for all values ​​within the time period (for example, four seconds) of the learning data and test data. As a result, a collection list is created that includes heat map values ​​during the unresponsive periods described above, and a correlation value is calculated using the collected list.

[0126] Therefore, in the second embodiment, we propose to remove time frames of unresponsive periods from the time series data and extract only time frames in which the target behavior to be estimated is actually occurring, and collect heat map values ​​for each joint point. Specifically, in certain time series data, the sum of the heat map values ​​for each joint point is calculated for each time frame, and if the sum of the heat map values ​​for all joint points in that time frame is lower than a predetermined threshold, that time frame is deemed to be an unresponsive period and deleted. As a result, only time frames that contribute highly to the score of the correct behavior are retained as significant data in the time series data, and a collection list containing the heat map values ​​of all joint points for each joint point is created, and correlation values ​​between each joint point are calculated.

[0127] Hereinafter, the no-response period deletion process executed in the collection list creation unit 30 of the information estimation device 1 in the second embodiment of the present invention will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the no-response period deletion process executed in the collection list creation unit 50 of the information estimation device 1 in the second embodiment of the present invention.

[0128] 12, the collection list creation unit 30 first acquires the collection list stored in the collection list storage unit 40 (step S11). Note that, as an example, a case will be described here in which the collection list creation unit 30 reacquires the collection list stored in the collection list storage unit 40 in the first embodiment described above and recreates the collection list, but the collection list creation unit 30 may perform the following process when performing the process of adding the heat map values ​​acquired from the heat map storage unit 20 as rows in the collection list, so as to add only significant data as rows in the collection list.

[0129] Next, the collection list creation unit 30 selects a specific time frame t (for example, t=1 in the initial process) (step S12), and calculates the sum of the heat map values ​​of all joint points included in the selected specific time frame t (step S13). Then, the collection list creation unit 30 compares the sum calculated in step S13 with a predetermined threshold value (step S14). In step S14, it is confirmed, for example, whether the sum is equal to or greater than the threshold value (sum value≧threshold value).

[0130] If the sum is greater than or equal to the threshold value in step S14 (sum ≧ threshold value), it is determined that this particular time frame does not belong to the unresponsive period and is significant data, and this particular time frame is retained (step S15).On the other hand, if the sum is less than or equal to the threshold value in step S14 (sum < threshold value), it is determined that this particular time frame belongs to the unresponsive period, and this particular time frame is deleted (step S16).

[0131] After determining whether a specific time frame belongs to an unresponsive period, the collection list creation unit 30 checks whether there is another time frame for which it has not yet been determined whether it belongs to an unresponsive period (step S17). If there is another time frame that has not been selected as a specific time frame, it selects another time frame (e.g., t=t+1) as a specific time frame (step S18) and performs the processes from step S12 onwards again. On the other hand, if there is no other time frame as a time frame, that is, if it has been determined whether all time frames belong to an unresponsive period, the unresponsive period deletion process ends.

[0132] Fig. 13 is a diagram showing an example of a collection list created in the second embodiment of the present invention. The configuration of the collection list shown in Fig. 13 is the same as the collection list shown in Fig. 5, and therefore a description thereof will be omitted here.

[0133] 13, the sum of the heat map values ​​of all joint points is smaller than the threshold (sum < threshold) for the second-highest time frame 2. The collection list creation unit 30 determines that this time frame 2 belongs to the unresponsive period, and deletes this time frame 2 from the collection list.

[0134] By deleting the time frames belonging to the unresponsive period from the collection list in this way, only significant data that do not belong to the unresponsive period remains in the collection list recreated by the collection list creation unit 30. As in the first embodiment described above, the correlation value calculation unit 50 performs processing to create a correlation table from this collection list, and the main model element extraction unit 60 performs processing to extract main model elements from this correlation table. At this time, in the second embodiment, since the data belonging to the unresponsive period has been deleted from the collection list, it is possible to extract main model elements with higher accuracy. Furthermore, there is no need to perform calculation processing for insignificant data that belong to the unresponsive period, thereby shortening the calculation time. [Industrial Applicability]

[0135] The present invention determines the influence (contribution) of each model element on the estimation result of a neural network for an estimation target model composed of multiple model elements, extracts the main model elements that affect the estimation result of the neural network, and appropriately omits model elements that have little effect on the estimation result of the neural network, making it applicable to neural network technologies in general. Furthermore, the present invention can achieve faster calculation processing (shortening calculation time) and lighter hardware for the computing device (reducing memory usage) while maintaining the accuracy rate of estimation by the neural network, and therefore is highly effective in environments requiring high-speed processing and high reliability, such as estimation of moving objects such as automobiles and pedestrians. [Explanation of symbols]

[0136] 1. Information estimation device 10 Estimation processing unit 11 Optional layers (input layer + intermediate layer) 12 Convolutional Layers 13 Pooling Layer 14 Fully connected layer 15 Output layer (softmax layer) 20 Heat map storage section 30 Collection List Creation Department 40 Collection list storage section 50 Correlation value calculation unit 60 Main model element extraction section

Claims

1. An information estimation device including a neural network capable of performing learning based on learning data in which time-series data in which parameters of a model to be estimated, each of which changes over time, are paired with a correct label assigned to the time-series data, in an estimation target model consisting of a plurality of model elements, The neural network a convolution layer that converts the supplied data into data that represents the temporal and spatial influences of the parameters of the plurality of model elements on each other; A pooling layer that converts the supplied data into data consisting of specific feature dimensions; a fully connected layer that converts the supplied data into data having dimensions equal to the number of classifications of the state of the model to be estimated; and a heat map storage unit that connects the convolution layer so that data output from the convolution layer is supplied to the pooling layer and data output from the pooling layer is supplied to the fully connected layer, and then trains the neural network using the training data to set a trained neural network, and then connects the convolution layer so that data output from the convolution layer is supplied to the fully connected layer, inputs the training data to the trained neural network, and stores the data output from the fully connected layer as a heat map; a collection list creation unit that creates a collection list by extracting and storing correct heat maps corresponding to correct labels from the heat maps stored in the heat map storage unit; a correlation value calculation unit that calculates a correlation value indicating a correlation between each model element from the collection list created by the collection list creation unit; a main model element extraction unit that extracts, as a plurality of main model elements, a plurality of model elements whose correlation values ​​calculated by the correlation value calculation unit are equal to or less than a predetermined threshold and whose correlations with other model elements are low; and an information estimation device configured to reconstruct an estimation target model consisting of the plurality of main model elements extracted by the main model element extraction unit, and to re-learn the neural network using the reconstructed estimation target model to perform estimation processing.

2. The information estimation device according to claim 1, wherein the collection list creation unit is configured to calculate a sum of the heat map values ​​included in each time frame and create a collection list including only the heat map values ​​of each time frame whose sum is equal to or greater than a predetermined threshold.

3. 3. The information estimation device according to claim 1, wherein the estimation target model is a skeletal model in which skeletal joint points are set as the model elements, or a graph model in which nodes representing events obtained from a system environment are set as the model elements.

4. An information estimation method for an information estimation device including a neural network capable of performing learning based on learning data in which time-series data in which parameters of a model to be estimated, which model is made up of a plurality of model elements, change over time, and a correct label assigned to the time-series data are paired, the method comprising: The neural network a convolution layer that converts the supplied data into data that represents the temporal and spatial influences of the parameters of the plurality of model elements on each other; A pooling layer that converts the supplied data into data consisting of specific feature dimensions; a fully connected layer that converts the supplied data into data having dimensions equal to the number of classifications of the state of the model to be estimated; and a step of connecting the convolution layer so that data output from the convolution layer is supplied to the pooling layer and data output from the pooling layer is supplied to the fully connected layer, and training the neural network using the training data to set a trained neural network; a heat map storage step of, after setting the trained neural network, connecting the convolution layer so that data output from the fully connected layer is supplied to the fully connected layer, inputting training data into the trained neural network, and storing data output from the fully connected layer as a heat map in a heat map storage unit; a collection list creation step of creating a collection list in which correct heat maps corresponding to correct labels are extracted and stored from the heat maps stored in the heat map storage step; a correlation value calculation step of calculating a correlation value indicating a correlation between each model element from the collection list created in the collection list creation step; a main model element extraction step of extracting, as a plurality of main model elements, a plurality of model elements whose correlation values ​​calculated in the correlation value calculation step are equal to or less than a predetermined threshold and have low correlations with other model elements; an estimation step of reconstructing an estimation target model including the plurality of main model elements extracted in the main model element extraction step, and re-learning the neural network using the reconstructed estimation target model to perform estimation processing; An information estimation method having the following.

5. The information estimation method according to claim 4, wherein in the collection list creation step, the sum of the heat map values ​​included in each time frame is calculated, and a collection list is created that includes only the heat map values ​​of each time frame whose sum of the heat map values ​​is equal to or greater than a predetermined threshold.

6. 6. The information estimation method according to claim 4, wherein a skeletal model in which skeletal joint points are set as the model elements, or a graph model in which nodes representing events obtained from a system environment are set as the model elements, is used as the estimation target model.

Images