Recognition device, recognition method, and program

Abstract

To improve identification accuracy of a neural network used for class identification.SOLUTION: A recognition device according to an embodiment includes a feature quantity generation unit, a feature quantity conversion unit, an important feature quantity generation unit, a feature quantity integration unit and an identification unit. The feature quantity generation unit generates a first feature quantity having a feature of sensor data on the basis of the sensor data. The feature quantity conversion unit converts the first feature quantity to a second feature quantity having a feature to contribute to class identification of the sensor data. The important feature quantity generation unit generates an important feature quantity having an important feature in class identification on the basis of a cross correlation between the first feature quantity and the second feature quantity. The feature quantity integration unit generates an integrated feature quantity while considering each feature of the first feature quantity and the second feature quantity on the basis of the second feature quantity and the important feature quantity. The identification unit identifies the class on the basis of the integrated feature quantity.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] FIELD An embodiment of the present invention relates to a recognition device, a recognition method, and a program. [Background technology]

[0002] In recent years, with the advancement of processor and sensor performance and the miniaturization and low power consumption of these devices, low-cost wearable devices have become commonplace, and healthcare services that log daily physical activities (life logs) have been actively proposed. There is also an accelerating trend to expand the scope of use of these devices to the social infrastructure field. For example, there is a known technology that uses sensors such as wearable devices to obtain the physical activities of workers in manufacturing, logistics, and inspection work sites, and recognizes the workers' work activities from the sensor data using neural networks.

[0003] As one of the above techniques, for example, there is a technique for processing features extracted from sensor data using attention information generated based on the class classification contribution, and performing classification. However, in this technique, the class classification contribution is common to all sensor data and does not reflect the different characteristics of each sensor data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-190960 A Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide a recognition device, a recognition method, and a program capable of improving the classification accuracy of a neural network used for class classification. [Means for solving the problem]

[0006] A recognition device according to an embodiment includes a feature generating unit, a feature transforming unit, an important feature generating unit, a feature integrating unit, and a classification unit. The feature generating unit generates a first feature having a feature of the sensor data based on the sensor data. The feature transforming unit transforms the first feature into a second feature having a feature that contributes to classifying the sensor data. The important feature generating unit generates an important feature having a feature that is important in classifying the class based on a cross-correlation between the first feature and the second feature. The feature integrating unit generates an integrated feature that takes into account the respective features of the first feature and the second feature based on the second feature and the important feature. The classification unit identifies the class based on the integrated feature. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an example of the configuration of a recognition system including a recognition device according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of the arrangement of a recognition device according to a first embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of the configuration of a network model corresponding to the recognition device according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining a first specific example using the network model of FIG. 3. [Diagram 5] FIG. 4 is a diagram for explaining a second specific example using the network model of FIG. 3. [Figure 6] 4 is a flowchart showing an example of the operation of the recognition device according to the first embodiment. [Figure 7] FIG. 11 is a block diagram showing an example of the arrangement of a recognition device according to a second embodiment. [Figure 8] 10 is a flowchart showing an example of the operation of a recognition device according to a second embodiment. [Figure 9] FIG. 13 is a block diagram showing an example of the arrangement of a recognition device according to a third embodiment. [Figure 10] 13 is a flowchart showing an example of the operation of a recognition device according to a third embodiment. [Figure 11]FIG. 1 is a block diagram illustrating a hardware configuration of a computer according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of a recognition device will be described in detail with reference to the drawings.

[0009] (First embodiment) FIG. 1 is a block diagram showing a configuration example of a recognition system 1 including a recognition device 100 according to a first embodiment. The recognition system 1 in FIG. 1 includes the recognition device 100 and one or more sensors. In FIG. 1, sensors 110-1, 110-2, ..., 110-n are illustrated as the one or more sensors. The recognition device 100 and the sensors 110-1, 110-2, ..., 110-n are connected via a network 120. The communication method of the network 120 may be a wired method or a wireless method. The communication method may be a combination of a wired method and a wireless method. In the following, when the sensors 110-1, 110-2, ..., 110-n are not distinguished from each other, they are simply referred to as sensors 110.

[0010] The sensor 110 acquires sensor data indicating information of a recognition target, and transmits the sensor data to the recognition device 100. The sensor data is, for example, acceleration data, angular velocity data, geomagnetic data, atmospheric pressure data, temperature and humidity data, body temperature data, myoelectric potential data, and pulse wave data. The sensor 110 is, for example, a wearable sensor. When a worker wears the sensor 110, the information of the recognition target is, for example, information on the behavior of the sensor wearer (behavior information).

[0011] The sensor 110 may be an imaging device worn by the worker, or may be an imaging device that images the worker. When the sensor 110 is an imaging device, the sensor data is, for example, an image and three-dimensional skeleton data.

[0012] When estimating (recognizing) the behavior of a person wearing a wearable sensor, the target actions and tasks are different and diverse, such as in a factory, field, home, office, and fitness center. In this specification, a work environment that is grouped by the type of work (task type) is called a work domain. In a work domain such as a factory, it can be assumed that the main and common work actions are "cart movement," "transport," "touch panel operation," "writing operation," and "switch operation." On the other hand, in a logistics site such as a warehouse, in addition to "transportation" of luggage, the main work types are "barcode reading," "picking," "packing," and even "forklift operation." When reusing a learning model between different work domains, it is necessary to respond to different work types depending on the work domain. It is also necessary to consider the influence of different sensor configurations. For example, in order to prioritize recognition accuracy, it is possible to attach sensors to multiple parts of the body such as the legs and trunk in addition to attaching sensors to the arms.

[0013] Fig. 2 is a block diagram showing an example of the configuration of the recognition device 100 according to the first embodiment. The recognition device 100 in Fig. 2 acquires sensor data from a sensor 110, and recognizes information of a recognition target (e.g., human behavior information, etc.) from the acquired sensor data. The recognition device 100 includes a feature generation unit 210, a feature conversion unit 220, an important feature generation unit 230, a feature integration unit 240, and a classification unit 250.

[0014] The feature generation unit 210 generates a first feature having a feature of the sensor data based on the sensor data acquired from the sensor 110. The feature generation unit 210 outputs the first feature to the feature conversion unit 220 and the important feature generation unit 230.

[0015] The feature conversion unit 220 receives a first feature from the feature generation unit 210. The feature conversion unit 220 converts the first feature into a second feature having a feature that contributes to classifying the sensor data. Specifically, the feature conversion unit 220 generates a second feature by activating a feature that is of interest in class classification for the first feature. The feature conversion unit 220 outputs the second feature to the important feature generation unit 230, the feature integration unit 240, and the classification unit 250. The above classes will be described later.

[0016] The important feature generation unit 230 receives the first feature from the feature generation unit 210 and receives the second feature from the feature conversion unit 220. The important feature generation unit 230 generates an important feature having an important feature in class identification based on the cross-correlation between the first feature and the second feature. Specifically, the important feature generation unit 230 calculates a cross-correlation value between the first feature and the second feature, and generates an important feature that is important in class identification. In addition, this important feature has a high correlation with the signal features of the sensor data.

[0017] The feature integration unit 240 receives the second feature from the feature conversion unit 220 and receives the important feature from the important feature generation unit 230. The feature integration unit 240 generates an integrated feature based on the second feature and the important feature, taking into account the respective features of the first feature and the second feature. Specifically, the feature integration unit 240 generates an integrated feature by adding or multiplying and integrating the important feature and the second feature. The feature integration unit 240 outputs the integrated feature to the classification unit 250.

[0018] The identification unit 250 receives the integrated feature from the feature integration unit 240. The identification unit 250 identifies a class based on the integrated feature. Specifically, the identification unit 250 identifies a class of sensor data corresponding to the integrated feature by using a neural network that receives the integrated feature as input and outputs a class.

[0019] Furthermore, the identification unit 250 may identify a class based on the second feature amount. Specifically, the identification unit 250 receives the second feature amount from the feature amount conversion unit 220. The identification unit 250 identifies the class of the sensor data corresponding to the second feature amount by using a neural network that receives the second feature amount as an input and outputs a class.

[0020] The classes identified by the identification unit 250 may be defined arbitrarily. The classes are, for example, behavior, work type, and work motion. Specifically, classes identified in a work domain such as a factory are, for example, "cart movement," "transport," "touch panel operation," "writing operation," and "switch operation." Thus, the identification unit 250 can identify the behavior of the sensor wearer.

[0021] The class identification results by the identification unit 250 may be displayed on a display not shown in Figures 1 and 2, may be stored in a storage device, or may be transmitted to another device connected wirelessly or via a wired connection.

[0022] The above has described the configurations of the recognition device 100 and the recognition system 1 according to the first embodiment. Next, a network model corresponding to the recognition device 100 according to the first embodiment will be described.

[0023] Fig. 3 is a block diagram showing an example of the configuration of a network model corresponding to the recognition device 100 according to the first embodiment. The network model in Fig. 3 is an example in which each unit of the recognition device 100 in Fig. 2 is embodied. In the following, each unit of the recognition device 100 in Fig. 2 will be specifically described.

[0024] First, a description will be given of the sensor data input to the recognition device 100. The input sensor data is time-series data acquired at sampling intervals of the sensor 110, and is data in which time-series data included in a specific interval section (a section defined by a predetermined number of samplings) is compiled by a predetermined number of intervals. Therefore, the processing unit of the sensor data in the recognition device 100 is temporally continuous sensor data defined by a predetermined number of intervals.

[0025] The sensor data included in the above-mentioned interval section may be a time-series sensor value, or may be a frequency feature (signal intensity value for each frequency) obtained by converting the time-series sensor value into the frequency domain using an algorithm such as FFT (Fast Fourier Transform). For example, multiple sensors may acquire time-series data at different sampling intervals. In this case, it is necessary to handle the sensor data at different sampling intervals in a unified manner. Therefore, in the following description, the sensor data is assumed to be a frequency feature obtained by performing a frequency transformation using FFT or the like.

[0026] The feature generating unit 210 includes one or more neural networks (NNs). For example, a convolutional neural network (CNN) is used as the NN. Specifically, the feature generating unit 210 includes a feature extraction CNN 211 and a feature fusion CNN 212.

[0027] The feature extraction CNN 211 is a CNN for extracting features. The feature extraction CNN 211 has, for example, an input layer according to the number of sensor data. The feature extraction CNN 211 outputs features for each sensor data. Hereinafter, it is assumed that data is acquired from three types of modalities (for example, an accelerometer, a gyroscope, and a geomagnetometer) for each of three sensors attached to different locations.

[0028] The feature fusion CNN 212 is a CNN for fusing features. The feature fusion CNN 212 fuses a plurality of features output from the feature extraction CNN 211. Feature fusion includes, for example, fusion regarding sensor modality and fusion regarding sensor attachment site. In the fusion regarding sensor modality, for example, sensor data of different sensor attachment sites but the same sensor modality are fused. This fusion makes it possible to efficiently extract sensor signal features specific to the sensor modality. In addition, in the fusion regarding sensor attachment site, for example, sensor data of different sensor modalities but the same sensor attachment site are fused. This fusion makes it possible to efficiently extract sensor signal features specific to the sensor attachment site.

[0029] It is to be noted that both the sensor modality-related fusion and the sensor attachment site-related fusion may be performed, which makes it possible to extract signal features that take into account both the sensor modality and the sensor attachment site.

[0030] The feature conversion unit 220 includes one or more NNs. For example, a CNN is used as the NN. Specifically, the feature conversion unit 220 includes a feature conversion CNN 221.

[0031] The feature conversion CNN 221 is a CNN for converting features. The feature conversion CNN 221 has a convolution layer using a kernel of 1×1 size, for example, with the number of output filters set to the number of classes to be identified. The convolution layer used here is configured to activate the features of the classes assigned to the filters. The feature conversion CNN 221 generates second features by passing the first features through the convolution layer. Therefore, the second features may be called class activation features (CAM: Class Activation Map).

[0032] In general, the feature transforming unit 220 transforms a first feature into a second feature so as to activate a class assigned to a filter.

[0033] The important feature generation unit 230 includes one or more NNs. For example, a CNN is used as the NN. Specifically, the important feature generation unit 230 includes an adjustment CNN 231, a first multiplication unit 232, and a second multiplication unit 233.

[0034] The adjustment CNN 231 is a CNN for adjusting features. The adjustment CNN 231 has a convolution layer in which the number of output filters is set to the number of classes to be identified, using a kernel of 1×1 size, for example. The convolution layer used here is configured to activate the features of the sensor data. In addition, the number of filters output from the adjustment CNN 231 is the same as the number of filters output from the feature conversion CNN 221. The adjustment CNN 231 generates generic features by passing the first features through the above-mentioned convolution layer. Therefore, the generic features have the same number of filters as the second features.

[0035] The first multiplication unit 232 generates feature importance by multiplying the generic feature and the second feature. Specifically, when the generic feature and the second feature are expressed by matrices of the same size, the first multiplication unit 232 calculates a matrix product between the generic feature and the transposed matrix of the second feature, and generates feature importance by applying a softmax function on a filter-by-filter basis.

[0036] The second multiplication unit 233 generates an important feature by multiplying the generic feature by the feature importance. Specifically, when the generic feature and the feature importance are expressed by matrices of the same size, the second multiplication unit 233 generates an important feature by calculating a matrix product between the transposed matrix of the generic feature and the feature importance.

[0037] In general, the important feature generating unit 230 generates an important feature having an important feature in class discrimination, based on the cross-correlation between the first feature and the second feature.

[0038] The feature integration unit 240 includes an adder 241. The adder 241 generates an integrated feature by adding the second feature and the important feature.

[0039] The classification unit 250 has a classification section for each type of feature to be classified.

[0040] The first identification unit 251 identifies a class of a recognition target based on the integrated feature. Specifically, the first identification unit 251 includes an RNN (Recurrent Neural Network) such as an LSTM (Long Short Term Memory) and a GRU (Gated Recurrent Unit), and a softmax layer that applies a softmax function. The first identification unit 251 uses the RNN and the softmax layer to configure a neural network that outputs a class by inputting the integrated feature. The identification result in the first identification unit 251 reflects the learning data in time units. The output data from the softmax layer may be called a class identification likelihood.

[0041] The second classification unit 252 classifies the class of the recognition target based on the second feature. Specifically, the second classification unit 252 includes a GAP (Global Average Pooling) layer and a softmax layer. The second classification unit 252 uses the GAP layer and the softmax layer to configure a neural network that outputs a class by inputting the second feature. The classification result in the second classification unit 252 reflects the entire training data. The output data from the softmax layer may be called a class classification likelihood.

[0042] The configuration of the network model corresponding to the recognition device 100 according to the first embodiment has been described above. Next, specific examples using this network model will be described. In the first specific example, the use of feature importance calculated in units of two-dimensional feature elements will be described, and in the second specific example, the use of feature importance calculated in units of three-dimensional feature elements will be described.

[0043] Fig. 4 is a diagram for explaining a first specific example using the network model of Fig. 3. Fig. 4 shows a network model (feature conversion CNN 221, adjustment CNN 231, first multiplication unit 232, second multiplication unit 233, and addition unit 241) related to the feature conversion unit 220, important feature generation unit 230, and feature integration unit 240, and intermediate generated data (first feature, second feature, generic feature, feature importance, important feature, and integrated feature). Also, Fig. 4 deals with features that have been fused regarding sensor modality for sensor data.

[0044] The feature conversion unit 220 receives the first feature CNN_t from the feature generation unit 210 (not shown). The first feature CNN_t is obtained by performing fusion regarding the sensor modality for the sensor data. Therefore, the feature elements of the first feature CNN_t are time, sensor modality, and frequency features.

[0045] When the first feature CNN_t is input, the feature conversion CNN 221 converts the first feature A_t (filter CAM ,elements).

[0046] The important feature generator 230 receives the first feature CNN_t from the feature generator 210 (not shown), and receives the second feature A_t (filter CAM ,elements).

[0047] When the first feature CNN_t is input, the adjustment CNN 231 calculates the general feature B_t (filter CNN , elements) is output. At this time, the generic feature B_t (filter CNN The number of filters for the second feature A_t(filter CAM , elements) is adjusted to be equal to the number of filters.

[0048] The first multiplication unit 232 multiplies the generic feature B_t(filter CNN , elements) and the second feature A_t (filter CAM ,elements) to obtain the feature importance W_t(filter CNN ,filter CAM Specifically, the feature importance W_t(filter CNN ,filter CAM ) is expressed by the following equation (1).

[0049]

number

[0050] Here, Softmax() indicates a softmax function, and transpose() indicates transposition. According to the above formula (1), the first multiplication unit 232 generates filter-based feature importance W_t by applying a softmax function to the matrix product of the generic feature B_t and the transposed matrix of the second feature A_t, in units of filters. Note that in FIG. 4, since the feature importance W_t is generated in units of time, the feature importance W_t includes two-dimensional feature elements (sensor modality and frequency feature).

[0051] The second multiplication unit 233 multiplies the generic feature B_t(filter CNN ,elements) and feature importance W_t(filter CNN ,filter CAM ) and the important features Att_t(elements,filter CAM Specifically, we generate the important features Att_t(elements, filter CAM ) is expressed by the following equation (2).

[0052]

number

[0053] According to the above formula (2), the second multiplication unit 233 generates the important feature Att_t by calculating the matrix product of the transposed matrix of the generic feature B_t and the feature importance W_t.

[0054] In general, the important feature generating unit 230 generates an important feature having an important feature in class discrimination, based on the cross-correlation between the first feature and the second feature.

[0055] The feature integration unit 240 receives the second feature A_t(filter CAM , elements) and receives the important feature Att_t(elements, filter CAM )

[0056] The adder 241 calculates the second feature amount A_t(filter CAM ,elements) and important features Att_t(elements,filter CAM ) to obtain the integrated feature C_t(filter CNN Specifically, we generate the integrated feature C_t(filter CNN , elements) is expressed by the following equation (3).

[0057]

number

[0058] According to the above formula (3), the adder 241 generates an integrated feature C_t by calculating the sum of the second feature A_t and the transposed matrix of the important feature Att_t.

[0059] Fig. 5 is a diagram for explaining a second specific example using the network model of Fig. 3. Similar to Fig. 4, Fig. 5 shows a network model relating to the feature conversion unit 220, the important feature generation unit 230, and the feature integration unit 240, and intermediate generated data (first feature, second feature, generic feature, feature importance, important feature, and integrated feature).

[0060] The second specific example in Fig. 5 differs from the first specific example in Fig. 4 in that time is included in the feature elements when calculating the feature importance. That is, in Fig. 5, the feature importance W_t includes three-dimensional feature elements (time, sensor modality, and frequency feature). When time is included in the feature elements, in the part where the feature elements are integrated in each time unit in the description of Fig. 4, time is also included in the feature elements and the feature importance and integrated feature are generated collectively in filter units.

[0061] In addition, the feature elements used for calculating the feature importance may include only frequency features. In the case where the feature elements include only frequency features, the feature elements related to time and the feature elements related to sensor modality are separated, and the feature importance and the integrated feature are generated separately for each of them.

[0062] In the above description, the feature quantity for which fusion regarding the sensor modality is performed for the sensor data is handled, but this is not limited thereto. For example, the feature quantity for which fusion regarding the sensor mounting site is performed for the sensor data may be handled. When fusion regarding the sensor mounting site is performed, for example, the feature elements include the sensor mounting site and frequency features. Furthermore, when fusion regarding both the sensor modality and the sensor mounting site are performed for the sensor data, for example, the feature elements include the sensor modality, the sensor mounting site, and frequency features.

[0063] A specific example using a network model corresponding to the recognition device 100 according to the first embodiment has been described above. Next, the operation of the recognition device 100 will be described with reference to the flowchart of FIG.

[0064] Fig. 6 is a flowchart showing an example of the operation of the recognition device 100 according to the first embodiment. The flowchart in Fig. 6 shows, for example, a series of flows from input of sensor data to output of a class identification result.

[0065] (Step ST110) The recognition device 100 acquires sensor data from a sensor 110 .

[0066] (Step ST120) After acquiring the sensor data, the feature generator 210 generates a first feature based on the sensor data.

[0067] (Step ST130) After generating the first feature amount, the feature amount transforming unit 220 transforms the first feature amount into a second feature amount.

[0068] (Step ST140) After converting the first feature amount into the second feature amount, the important feature amount generating section 230 calculates the importance based on the first feature amount and the second feature amount.

[0069] (Step ST150) After calculating the importance, the important feature generating section 230 generates an important feature based on the first feature and the importance.

[0070] (Step ST160) After generating the important feature amounts, the feature amount integration unit 240 generates an integrated feature amount based on the important feature amounts and the second feature amounts.

[0071] (Step ST170) After generating the integrated features, the classifier 250 classifies classes based on the integrated features.

[0072] As described above, the recognition device of the first embodiment generates a first feature having a characteristic of the sensor data based on sensor data, converts the first feature into a second feature having a characteristic that contributes to identifying a class of the sensor data, generates an important feature having a characteristic that is important in identifying the class based on the cross-correlation between the first feature and the second feature, generates an integrated feature that takes into account the characteristics of each of the first feature and the second feature based on the second feature and the important feature, and identifies the class based on the integrated feature.

[0073] Therefore, the recognition device according to the first embodiment can improve the classification accuracy of the neural network used for class classification by taking into account both the features of the sensor data and the features of the class classification.

[0074] For example, the recognition device according to the first embodiment may combine sensor data from a plurality of sensors with different modalities. Specifically, the sensor data may be the same image, but may combine images with different sensing frequency bands, such as a visible light image and an infrared image. In this case, for example, when detecting a human body, the recognition device integrates features to emphasize the infrared image and suppress the visible light image.

[0075] Also, the combination of different sensor data may be, for example, a time-series sensor signal output from an inertial sensor and an acoustic signal acquired using a microphone. For example, the recognition device acquires an acoustic signal from the work site from a microphone attached to the worker's head or arm, or a microphone installed on a machine tool, and at the same time acquires a sensor signal associated with the worker's movements using an inertial sensor attached to the worker's arm or wrist. This allows the recognition device to simultaneously input not only the movements during work but also acoustic information at the work site as sensor data, and enables the integration of features focusing on sounds specific to the work (for example, acoustic signals emitted from a machine tool), which is expected to have the effect of improving the recognition accuracy of work movements.

[0076] Furthermore, the combination of different sensor data may be a combination of signals output from a biosensor such as an electrooculography sensor or an electromyography sensor and a signal output from an inertial sensor. This makes it possible to obtain eye movements, blinks, and muscle contraction activity, and improves recognition accuracy by using unique sensor information that cannot be obtained by an inertial sensor.

[0077] Second Embodiment In the first embodiment, a recognition device that identifies classes from sensor data using a network model has been described. In the second embodiment, on the other hand, a description will be given of switching of a network model when a condition for class identification is changed.

[0078] Fig. 7 is a block diagram showing an example of the configuration of a recognition device 700 according to a second embodiment. The recognition device 700 in Fig. 7 includes a feature generation unit 210, a feature conversion unit 220, an important feature generation unit 230, a feature integration unit 240, a classification unit 250, a storage unit 710, and a model switching unit 720. In the following description, the feature generation unit 210, the feature conversion unit 220, the important feature generation unit 230, the feature integration unit 240, and the classification unit 250 are collectively referred to as a classification model 200.

[0079] The storage unit 710 stores information about the network model used in the identification model 200. For example, the storage unit 710 stores a work domain and a network model in association with each other. For example, the storage unit 710 stores a sensor modality or a sensor attachment site and a network model in association with each other. For example, the storage unit 710 stores a worker and parameters of the network model in association with each other.

[0080] The model switching unit 720 switches the network model of the identification model based on model switching data including information about the network model of the identification model 200. The information about the network model includes, for example, information about the work domain, information about the sensor modality of the input sensor data or information about the sensor attachment site, and information about the worker.

[0081] Note that switching of a network model includes both switching of the network model itself and switching of parameters of the network model. Therefore, the model switching unit 720 switches at least one of the network model and the parameters of the network model.

[0082] Fig. 8 is a flowchart showing an example of the operation of the recognition device 700 according to the second embodiment. The flowchart in Fig. 8 includes a model switching process and a class identification process. Note that the class identification process is the same as the series of processes in the flowchart in Fig. 6, and therefore a description thereof will be omitted.

[0083] (Step ST210) The recognition device 700 acquires model switching data.

[0084] (Step ST220) After acquiring the model switching data, the model switching unit 720 switches the network model based on the model switching data. Specifically, the model switching unit 720 reads out the network model associated with the information on the network model included in the model switching data from the storage unit 710, and switches the network model of the identification model 200.

[0085] (Step ST230) After switching the network model, the recognition device 700 executes a class identification process using the switched network model.

[0086] As described above, the recognition device according to the second embodiment can switch at least one of the network model and the parameters of the network model.

[0087] Therefore, the recognition device according to the second embodiment can perform appropriate recognition processing in accordance with various situations by switching the network model or the parameters of the network model in accordance with the sensor data.

[0088] For example, the recognition device according to the second embodiment can switch the neural network identification model when applied to different work domains, such as a work domain in a factory or a work domain in a logistics site. Furthermore, even in the same work domain, this recognition device can flexibly perform recognition processing with the same system configuration by switching the neural network identification model even if the sensor configuration is changed (for example, the sensor modality or the sensor mounting position is different, or a new sensor is added later).

[0089] (Third embodiment) In the first and second embodiments, a recognition device that identifies a class from sensor data using a pre-trained network model has been described. On the other hand, in the third embodiment, learning of a network model when changing the conditions for class identification or the sensor data to be recognized will be described.

[0090] 9 is a block diagram showing an example of the configuration of a recognition device 900 according to the third embodiment. The recognition device 900 in FIG.

[0091] The learning unit 910 learns the network model using a learning data set in which sensor data is associated with class data (correct answer data) that is the correct answer. The learning unit 910 reflects the learning result of the network model in the model parameters of the discrimination model 200. The learning unit 910 may also store the learning result in the storage unit 710.

[0092] 9 further stores a learning data set used by the learning unit 910. The storage unit 710 may also store the learning results obtained by the learning unit 910.

[0093] Fig. 10 is a flowchart showing an example of the operation of the recognition device 900 according to the third embodiment. The flowchart in Fig. 10 includes a model switching process, a class identification process, and a learning process. Note that the model switching process and the class identification process are similar to the series of processes in the flowchart in Fig. 8, and therefore detailed explanations thereof will be omitted.

[0094] (Step ST310) The recognition device 700 acquires model switching data.

[0095] (Step ST320) After acquiring the model switching data, the model switching unit 720 switches the network model based on the model switching data. In the following, learning will be performed on the switched network model.

[0096] (Step ST330) After switching the network model, the recognition device 900 executes a class identification process. When training a network model, the recognition device 900 executes a class identification process using sensor data included in a training data set.

[0097] (Step ST340) After executing the class classification process, the learning unit 910 learns the network model based on the classification result. Specifically, the learning unit 910 updates the model parameters of the network model so that the error between the classification result and the correct answer data becomes zero. More specifically, the learning unit 910 calculates the cross-entropy error between the classification result and the correct answer data, and updates the model parameters based on the error backpropagation method.

[0098] 3, when the classification unit 250 is composed of two networks (first classification unit 251 and second classification unit 252), the output of each of the two networks has a distribution corresponding to the class classification likelihood (class classification likelihood distribution). Therefore, the learning unit 910 may treat the distribution distance between the class classification likelihood distributions of the two networks as a loss function of the network in addition to the cross-entropy error.

[0099] The distance between the two class classification likelihood distributions can be calculated using, for example, the Kullback-Leibler divergence, but is not limited to this. If the output distribution of the network that classifies the class using the second feature is P, the output distribution of the network that classifies the class using the integrated feature is Q, and the target class variable is i, then the Kullback-Leibler divergence D KL can be calculated using the following equation (4).

[0100]

number

[0101] Therefore, the learning unit 910 may learn the neural network model so that the distribution of the class identification likelihood output based on the second feature is equal to the distribution of the class identification likelihood output based on the integrated feature. By using this configuration, the identification unit 250 can correct the current identification result by using the identification results from a certain period of the past by storing the past identification results in the storage unit 710. For example, the identification unit 250 may output the most frequent identification result among the identification results recorded over a certain period of the past. In this way, it is possible to avoid a state in which the recognition result temporarily changes due to the influence of some abnormality in the sensor value, and it is possible to build a recognition system that is robust against noise factors such as sensor abnormalities.

[0102] As described above, the recognition device according to the third embodiment is capable of learning a neural network model.

[0103] Therefore, the recognition device according to the third embodiment can perform appropriate recognition processing even in a new environment by learning the neural network model.

[0104] For example, according to the recognition device of the third embodiment, even in the same work domain, when a target worker is added or changed, the network model can be trained and updated by providing a small amount of sensor data and a correct answer class. This makes it possible to develop a recognition system that is adapted to the worker, and can be developed as a recognition system that is more flexible and robust against noise.

[0105] (Hardware configuration) 11 is a block diagram illustrating a hardware configuration of a computer according to an embodiment. The computer 1100 includes, as hardware, a central processing unit (CPU) 1110, a random access memory (RAM) 1120, a program memory 1130, an auxiliary storage device 1140, and an input / output interface 1150. The CPU 1110 communicates with the RAM 1120, the program memory 1130, the auxiliary storage device 1140, and the input / output interface 1150 via a bus 1160.

[0106] The CPU 1110 is an example of a general-purpose processor. The RAM 1120 is used by the CPU 1110 as a working memory. The RAM 1120 includes a volatile memory such as a Synchronous Dynamic Random Access Memory (SDRAM). The program memory 1130 stores various programs including a parameter update processing program. As the program memory 1130, for example, a Read-Only Memory (ROM), a part of the auxiliary storage device 1140, or a combination thereof is used. The auxiliary storage device 1140 stores data non-temporarily. The auxiliary storage device 1140 includes a non-volatile memory such as an HDD or SSD.

[0107] The input / output interface 1150 is an interface for connecting to other devices, and is used, for example, to connect to a sound collection device and an output device.

[0108] Each program stored in the program memory 1130 includes computer-executable instructions. When executed by the CPU 1110, the program (computer-executable instructions) causes the CPU 1110 to execute a predetermined process. For example, when executed by the CPU 1110, a class identification process program causes the CPU 1110 to execute the series of processes described with respect to each part in FIG. 2, FIG. 7, and FIG. 9.

[0109] The program may be provided to the computer 1100 in a state where it is stored in a computer-readable storage medium. In this case, for example, the computer 1100 further includes a drive (not shown) for reading data from the storage medium, and acquires the program from the storage medium. Examples of the storage medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. Also, the program may be stored in a server on a communication network, and the computer 1100 may download the program from the server using the input / output interface 1150.

[0110] The processing described in the embodiment is not limited to being performed by a general-purpose hardware processor such as the CPU 1110 executing a program, but may be performed by a dedicated hardware processor such as an ASIC (Application Specific Integrated Circuit). The term processing circuit (processing unit) includes at least one general-purpose hardware processor, at least one dedicated hardware processor, or a combination of at least one general-purpose hardware processor and at least one dedicated hardware processor. In the example shown in FIG. 11, the CPU 1110, the RAM 1120, and the program memory 1130 correspond to the processing circuit.

[0111] Therefore, according to each of the above embodiments, it is possible to improve the classification accuracy of the neural network used for class classification.

[0112] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0113] 1...recognition system, 100, 700, 900...recognition device, 110-1, 110-2, 110-n...sensor, 120...network, 200...discrimination model, 210...feature generation unit, 211...CNN for feature extraction, 212...CNN for feature fusion, 220...feature conversion unit, 221...CNN for feature conversion, 230...important feature generation unit, 231...CNN for adjustment, 232...first multiplication unit, 233...second multiplication unit, 240...feature Feature integration unit, 241...addition unit, 250...classification unit, 251...first classification unit, 252...second classification unit, 710...storage unit, 720...model switching unit, 910...learning unit, 1100...computer, 1130...program memory, 1140...auxiliary storage device, 1150...input / output interface, 1160...bus, A_t...second feature, Att_t...important feature, B_t...general feature, C_t...integrated feature, CNN_t...first feature.

Claims

1. a feature generating unit that generates a first feature having a feature of the sensor data based on the sensor data; a feature conversion unit that converts the first feature into a second feature having a feature that contributes to identifying a class of the sensor data; an important feature generating unit that generates an important feature having a feature that is important in classifying the class based on a cross-correlation between the first feature and the second feature; a feature integration unit that generates an integrated feature based on the second feature and the important feature, the integrated feature being generated by taking into consideration features of the first feature and the second feature; a classification unit for classifying the class based on the integrated feature; A recognition device comprising:

2. the feature generating unit generates the first feature by at least one of fusing features related to a type of the sensor data and fusing features related to a mounting site of the sensor; The recognition device according to claim 1 .

3. the feature transformation unit transforms the first feature into the second feature such that the number of the classes to be identified is the same as the number of the filters. The recognition device according to claim 1 or 2.

4. the feature transformation unit transforms the first feature into the second feature so as to activate the class assigned to the filter; The recognition device according to claim 3.

5. The important feature generation unit generating a generic feature having the same number of filters as the second feature based on the first feature; calculating a feature importance by multiplying the generic feature by the second feature; generating the important feature by multiplying the generic feature by the feature importance; A recognition device according to any one of claims 1 to 4.

6. the important feature generation unit generates the feature importance in units of feature elements including at least a frequency feature of the sensor data and further including at least one of time, a type of the sensor data, and an attachment site of the sensor; The recognition device according to claim 5.

7. the feature integration unit generates the integrated feature by adding or multiplying the second feature and the important feature. A recognition device according to any one of claims 1 to 6.

8. The recognition device according to claim 1 , wherein the classifying unit identifies the class by using a past classification result.

9. The recognition device according to claim 1 , wherein the identification unit identifies a behavior of a person wearing the sensor.

10. The recognition device according to claim 1 , wherein the classifier is configured to classify a class of the entire sensor data based on the second feature amount.

11. a storage unit that stores information about a neural network model that constitutes the feature generating unit, the feature transforming unit, the important feature generating unit, the feature integrating unit, and the classifying unit; Further comprising: A recognition device according to any one of claims 1 to 10.

12. A switching unit that switches at least one of the neural network model and the parameters of the neural network model. The recognition device of claim 11 further comprising:

13. A learning unit for learning the neural network model. Further comprising: The recognition device according to claim 11 or 12.

14. The recognition device according to claim 13, wherein the learning unit learns the neural network model so that a distribution of the class identification likelihood output based on the second feature and a distribution of the class identification likelihood output based on the integrated feature are equal to each other.

15. The learning unit updates the neural network model using a learning result. A recognition device according to claim 13 or 14.

16. The learning unit stores the learning result in the storage unit. The recognition device according to claim 15.

17. generating a first feature amount having a feature of the sensor data based on the sensor data; converting the first feature amount into a second feature amount having a feature that contributes to identifying a class of the sensor data; generating an important feature having an important feature in classifying the class based on a cross-correlation between the first feature and the second feature; generating an integrated feature amount based on the second feature amount and the important feature amount, the integrated feature amount taking into consideration respective features of the first feature amount and the second feature amount; identifying the class based on the integrated features; A recognition method comprising:

18. Computer, means for generating a first feature amount having a feature of the sensor data based on the sensor data; a means for converting the first feature amount into a second feature amount having a feature that contributes to identifying a class of the sensor data; a means for generating an important feature having an important feature in classifying the class based on a cross-correlation between the first feature and the second feature; a means for generating an integrated feature based on the second feature and the important feature, the integrated feature taking into consideration respective features of the first feature and the second feature; means for identifying the class based on the integrated feature amount; A program to function as a

Images