Learning device, learning method, program, and inference device

Abstract

It is desirable to reduce the development costs of models for identifying the types of time series data and determining the degree of anomaly in time series data, as well as the processing load imposed by such models. [Solution] A learning device is provided, comprising: a model generation unit that generates a generation model by connecting an identification model for identifying the type of time series data to a determination model for determining the degree of anomaly of the time series data; and a model learning unit that learns the generation model based on training data, first training data which is training data for the type of time series data identified from the training data, and second training data which is training data for the degree of anomaly of the time series data identified from the training data.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, a program, and an inference device. [Background technology]

[0002] <000000> In recent years, techniques have been developed to identify the type of time series data and to determine the degree of anomaly in that time series data. Behavior can be considered an example of time series data. For example, Non-Patent Documents 1 and 2 disclose techniques for identifying the type of behavior and determining the degree of anomaly in that behavior. Generally, the identification model for identifying the type of time series data and the determination model for determining the degree of anomaly in that time series data are not connected. In other words, the identification model and the determination model are separate. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Zehua Sun et al., "Human Action Recognition from Various Data Modalities: A Review", [online], arXiv 2021, [Retrieved January 27, 2020], Internet<https: / / arxiv.org / pdf / 2012.11866v3> [Non-Patent Document 2] Abida Haque et al., "Markov Chain Modeling for Anomaly Detection in High Performance Computing System Logs", [online], Association for Computing Machinery 2017, [Retrieved January 27, 2025], Internet<https: / / dl.acm.org / doi / 10.1145 / 3152493.<3152559> [Overview of the project] [Problems that the invention aims to solve]

[0004] However, if the identification model and the decision model are separate, it is conceivable that the development cost of a model that includes both the identification and decision models will increase, and the processing load on that model will also increase.

[0005] Therefore, it is desirable to have technologies that can reduce the development costs of models for identifying the types of time series data and determining the degree of anomaly in time series data, as well as the processing load imposed by such models. [Means for solving the problem]

[0006] To solve the above problems, according to one aspect of the present invention, a learning device is provided comprising: a model generation unit that generates a generative model by connecting an identification model for identifying the type of time series data to a determination model for determining the degree of anomaly of the time series data; and a model learning unit that learns the generative model based on training data, first training data which is training data for the type of time series data identified from the training data, and second training data which is training data for the degree of anomaly of the time series data identified from the training data. The generative model may be an example of a learning model generated by machine learning.

[0007] The aforementioned time-series data may also represent the actions of the acting entity.

[0008] The aforementioned actor may be a person.

[0009] The discrimination model may be configured to include a neural network, and the decision model may be configured to include neurons.

[0010] The model generation unit may connect at least a portion of the intermediate layers included in the identification model to the identification model, using the neurons as the determination model.

[0011] The model learning unit may acquire a first output value output from the discrimination model corresponding to the training data and a second output value output from the decision model corresponding to the training data, and train the generative model based on a first loss based on the first output value and the first training data, and a second loss based on the second output value and the second training data.

[0012] The model learning unit may calculate the loss by adding the first loss and the second loss, and then train the generative model based on the calculated loss.

[0013] An inference device may be provided that includes an identification unit that identifies the type of time series data from the identification data and determines the degree of abnormality of the time series data, based on the trained model, which is the generated model after training by the learning device, and the identification data.

[0014] Furthermore, in order to solve the above problems, according to another aspect of the present invention, a computer-based learning method is provided, which includes generating a generative model by connecting an identification model for identifying types of time series data to a determination model for determining the degree of anomaly of the time series data, and learning the generative model based on training data, first training data which is training data for the types of time series data identified from the training data, and second training data which is training data for the degree of anomaly of the time series data identified from the training data.

[0015] Further, according to another aspect of the present invention to solve the above problem, there is provided a program that causes a computer to function as: a model generation unit that generates a generative model by connecting a determination model for determining an abnormality degree of time-series data to an identification model for identifying a type of the time-series data; and a model learning unit that learns the generated model based on training data, first teacher data which is teacher data of the type of time-series data identified from the training data, and second teacher data which is teacher data of the abnormality degree of the time-series data identified from the training data. [Effects of the Invention]

[0016] As described above, according to the present invention, there is provided a technique capable of reducing the development cost of a model for identifying the type of time-series data and determining the abnormality degree of time-series data, and reducing the processing load caused by the model. [Brief Description of the Drawings]

[0017] [Figure 1] It is a diagram showing an example of skeleton feature amounts of a person captured in a moving image. [Figure 2] It is a diagram for explaining the outline of identification processing by the identification model 42. [Figure 3] It is a diagram showing a functional configuration example of the identification system 1 according to the embodiment of the present invention. [Figure 4] It is a diagram for explaining the generated model 46. [Figure 5] It is a flowchart showing an example of processing in a learning stage. [Figure 6] It is a flowchart showing an example of processing in an inference stage. [Figure 7] It is a diagram showing the hardware configuration of an information processing apparatus 900 as an example of the identification system 1 according to the embodiment of the present invention. [Figure 8] It is a diagram showing an example of movement feature amounts of a person captured in a moving image. [Figure 9] It is a diagram showing an example of information about a target object captured in a moving image. [Figure 10] It is a diagram showing an example of depth information of an object captured in a moving image. [Mode for Carrying Out the Invention]

[0018] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and the accompanying drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0019] (0. Overview) First, an overview of an embodiment of the present invention will be described.

[0020] In recent years, identification models that perform various types of identification processing have been known. An example of the identification processing includes processing for identifying the behavior of a person appearing in a moving image. For example, feature amounts such as skeletal feature amounts of a person appearing in a moving image can be used for processing of identifying the behavior of a person appearing in a moving image. Here, with reference to FIG. 1, the skeletal feature amount of a person appearing in a moving image will be described.

[0021] FIG. 1 is a diagram showing an example of a skeletal feature amount of a person appearing in a moving image. Referring to FIG. 1, a skeletal feature amount F1 is shown as an example of the skeletal feature amount of a person appearing in a moving image. In the example shown in FIG. 1, from a first partial feature amount f11 to a sixth partial feature amount f16 are assumed as an example of a plurality of partial feature amounts included in the skeletal feature amount F1. However, the number of partial feature amounts included in the skeletal feature amount F1 is not necessarily limited to six.

[0022] Each of the first partial feature amount f11 to the sixth partial feature amount f16 may be a joint position existing in a person. In the example shown in FIG. 1, the partial feature amount f11 is the position of the joint "left wrist", and the second partial feature amount f12 is the position of the joint "right wrist". Further, the third partial feature amount f13 is the position of the joint "left elbow", and the fourth partial feature amount f14 is the position of the joint "right elbow". Furthermore, the fifth partial feature amount f15 is the position of the joint "left shoulder", and the sixth partial feature amount f16 is the position of the joint "right shoulder".

[0023] For example, joint positions can be represented by the two-dimensional coordinates of the joints as seen in the moving image (a combination of the X and Y coordinates of the joint). Note that joint positions are just one example of the positions of parts that make up a person. Therefore, each of the first to sixth partial features f11 through f16 may represent the positions of parts other than joints.

[0024] Figure 2 is a diagram illustrating the overview of the classification process performed by the classification model 42. Referring to Figure 2, the classification model 42 is shown. The classification model 42 may include a neural network. Alternatively, the classification model 42 may be a pre-trained model. That is, the classification model 42 is composed of a pre-trained neural network, and the weight parameters of the classification model 42 may be set to values ​​acquired through training. Alternatively, the weight parameters of the classification model 42 may be set to random values.

[0025] Here, we assume that the discrimination model 42 is composed of a TCN (Temporal Convolutional Network). However, the discrimination model 42 may be composed of a neural network other than a TCN. For example, the discrimination model 42 may be composed of an RNN (Recurrent Neural Network), a CNN (Convolutional Neural Network), or a GCN (Graph Convolutional Network).

[0026] In the example shown in Figure 2, t is the time axis, and t=0,1,2,3...,T-1,T are shown along the time axis. The discrimination model 42 has input layer processing units x(0),x(1),x(2),x(3),...,x(T-1),x(T) corresponding to the times t=0,1,2,3,...,T-1,T. Each of the times t=0,1,2,3,...,T-1,T corresponds to each frame contained in the video. For example, the number of elements from 0 to T may be 100 or more, but it may be any number.

[0027] Each processing unit x(0), x(1), x(2), x(3), ..., x(T-1), x(T) in the input layer consists of 12 neurons. However, the number of neurons in each processing unit x(0), x(1), x(2), x(3), ..., x(T-1), x(T) in the input layer is not limited to 12. For example, these 12 neurons may be input with six partial features, from the first partial feature f11 to the sixth partial feature f16 (i.e., the X and Y coordinates of each of the six joint positions).

[0028] Similarly, the discrimination model 42 comprises processing units y(0), y(1), y(2), y(3), ..., y(T-1), y(T). Each of these processing units y(0), y(1), y(2), y(3), ..., y(T-1), y(T) consists of five neurons. However, the number of neurons in each of the processing units y(0), y(1), y(2), y(3), ..., y(T-1), y(T) is not limited to five. Furthermore, this primarily assumes that the number of layers in the intermediate layer y is one. However, the number of layers in the intermediate layer y is not limited to one.

[0029] The discrimination model 42 comprises output layer processing units z(0), z(1), z(2), z(3), ..., z(T-1), z(T). Each output layer processing unit z(0), z(1), z(2), z(3), ..., z(T-1), z(T) consists of four neurons. However, the number of neurons contained in each output layer processing unit z(0), z(1), z(2), z(3), ..., z(T-1), z(T) is not limited to four.

[0030] Skeletal features are extracted from the frame corresponding to time t=0 among the multiple frames contained in the video. For example, the skeletal features extracted from the frame corresponding to time t=0 are input to the processing unit x(0) of the input layer corresponding to time t=0.

[0031] Similarly, the skeletal features extracted from the frame corresponding to time t=1 are input to the processing unit x(1) of the input layer corresponding to time t=1. The skeletal features extracted from the frame corresponding to time t=2 are input to the processing unit x(2) of the input layer corresponding to time t=2. The skeletal features extracted from the frame corresponding to time t=3 are input to the processing unit x(3) of the input layer corresponding to time t=3.

[0032] The outputs based on the skeletal features input to the processing units x(0) to x(3) of these input layers are input to the processing units y(1) to y(3) of the hidden layer. The outputs based on the input to the processing units y(1) to y(3) of the hidden layer are input from the processing units y(1) to y(3) of the hidden layer to the processing unit z(3) of the output layer. Based on the input to the processing unit z(3) of the output layer, the processing unit z(3) of the output layer produces an output. The number of processing units x of the input layer and processing units y of the hidden layer corresponding to the processing unit z of the output layer are not limited.

[0033] For example, each of the four neurons constituting the output layer processing unit z(3) corresponds to a type of action such as "standby," "attach part A," "attach part B," or "attach part C," and the type of action corresponding to the neuron with the highest output value is identified as the type of action performed by the person in the video. In other words, when t is an integer from 3 to T, the type of action at time t is identified based on the output value output from the output layer processing unit z(t) based on the skeletal features input to the input layer processing units x(t-3) to x(t).

[0034] The embodiments of the present invention have been described above.

[0035] (1. Details of the Embodiment) Next, we will describe the details of embodiments of the present invention.

[0036] (1-1. Explanation of Structure) First, an example of the configuration of the identification system 1 according to an embodiment of the present invention will be described. Figure 3 is a diagram showing an example of the functional configuration of the identification system 1 according to an embodiment of the present invention. As shown in Figure 3, the identification system 1 according to an embodiment of the present invention comprises a learning unit 10 and an inference unit 20. The learning unit 10 outputs a trained model 50 to the inference unit 20.

[0037] The identification system 1 is executed by a computer. The learning unit 10 is implemented by a learning device, and the inference unit 20 is implemented by an inference device. The learning device and the inference device may be implemented by the same computer, or they may be implemented by distributing them across multiple computers.

[0038] (Learning Section 10) The learning unit 10 comprises a data unit 110, a model generation unit 120, and a model learning unit 130.

[0039] The model generation unit 120 and the model learning unit 130 include a processor such as a CPU (Central Processing Unit), and their functions can be realized by the processor loading a program stored in memory (not shown) into RAM (Random Access Memory) and executing it. In this case, a computer-readable recording medium on which the program is stored may also be provided. Alternatively, the model generation unit 120 and the model learning unit 130 may be composed of dedicated hardware, or they may be composed of a combination of multiple hardware components.

[0040] (Data section 110) The data unit 110 may be composed of memory. The data unit 110 stores training videos as examples of training data used in the training phase. The data unit 110 also stores training data (first training data) of the types of behavior identified from the training videos. Furthermore, the data unit 110 stores training data (second training data) of the degree of abnormality of the behavior identified from the training videos. The data unit 110 also stores the identification model 42.

[0041] Sensors for obtaining video footage for training are installed in a predetermined environment. For example, the environment in which the sensors are installed may be a workplace where people perform tasks. In this case, the type of action can be identified as the type of work performed by the people. However, the environment in which the sensors are installed is not limited to this example.

[0042] For example, the environment in which the sensor is installed may be a store where purchasing behavior is performed by people. In this case, instead of the type of work performed by people, the type of purchasing behavior performed by people (for example, the action of taking an item from a shelf, the action of returning an item to a shelf, etc.) is identified as the type of behavior. For example, video footage previously obtained by a sensor installed in such a predetermined environment may be acquired by the learning unit 10 via a network and stored by the data unit 110.

[0043] On the other hand, training data for the types of actions may be manually assigned to each frame included in the training video. For example, the training data for the types of actions may be represented by a matrix in which vectors are arranged for each frame, with each vector having an element representing the type of action "standby," "attaching part A," "attaching part B," and "attaching part C," and the correct element being set to 1 and the non-correct elements being set to zero.

[0044] Furthermore, training data for behavioral abnormality may be manually added to the training video in advance. For example, training data for behavioral abnormality may be expressed as the difference between the order of behavioral types identified from the training video and the normal order of behavioral types. For example, the more behavioral types identified from the training video that differ from the normal order, the larger the training data for behavioral abnormality may be.

[0045] For example, let's assume that it is normal for the types of actions to transition in the following order: "Standby," "Attach Component A," "Attach Component B," and "Attach Component C."

[0046] In this case, suppose that in the training video, the person reverses the order in which they attach "part A" and "part B," and the types of actions transition in the order of "standby," "attaching part B," "attaching part A," and "attaching part C." In such a case, the sequence of actions identified from the training video contains two instances where the sequence differs from the normal sequence.

[0047] On the other hand, suppose that in the training video, the person attaches the parts in the correct order, and the types of actions transition in the order of "standby," "attaching part A," "attaching part B," and "attaching part C." In such a case, the sequence of actions identified from the training video does not contain any sequence that differs from the normal sequence.

[0048] As in this example, if a person reverses the order in which they attach "part A" and "part B," the training data for the degree of abnormality of the behavior may be larger than if the person had not made a mistake in the order of attaching the parts.

[0049] (Model generation unit 120) The model generation unit 120 generates the generated model 46 (Figure 4). The generation of the generated model 46 by the model generation unit 120 will be explained with reference to Figure 4.

[0050] Figure 4 is a diagram illustrating the generative model 46. Referring to Figure 4, an identification model 42 for identifying types of behavior is shown. Also referring to Figure 4, a judgment model 44 for determining the degree of abnormality of behavior is shown. The judgment model 44 is composed of neurons w(T). The model generation unit 120 generates the generative model 46 by connecting the judgment model 44 to the identification model 42.

[0051] More specifically, the model generation unit 120 only needs to connect neurons w(T) as judgment models 44 to at least a portion of the intermediate layer y included in the discrimination model 42. As a result, the output value of the intermediate layer y connected to neurons w(T) becomes the input value to neurons w(T). Then, the output value corresponding to the input value to neurons w(T) is output from neurons w(T) as the degree of abnormality of the behavior.

[0052] In the example shown in Figure 4, three processing units of the hidden layer y, y(T-2), y(T-1), and y(T), are connected to neuron w(T). However, the number of processing units of the hidden layer y connected to neuron w(T) does not have to be three. For example, there may be five processing units of the hidden layer y connected to neuron w(T). In this case, five processing units of the hidden layer y, y(T-4), y(T-3), y(T-2), y(T-1), and y(T), may be connected to neuron w(T).

[0053] (Model Learning Section 130) The model learning unit 130 obtains training video footage, training data for behavior types, and training data for behavioral abnormality from the data unit 110. The model learning unit 130 also obtains the generated model 46 from the model generation unit 120. Then, the model learning unit 130 trains the generated model 46 based on the training video footage, training data for behavior types, and training data for behavioral abnormality.

[0054] More specifically, the model learning unit 130 extracts skeletal features from each frame contained in the training video using a feature extractor. For example, the feature extractor may be implemented using a CNN (Convolutional Neural Network). The model learning unit 130 then inputs the skeletal features corresponding to each frame into the generation model 46. Based on the input of the skeletal features corresponding to each frame into the generation model 46, the discrimination model 42 outputs an output value (first output value), and the judgment model 44 outputs an output value (second output value).

[0055] For example, if the training video contains frames corresponding to each time point from time t=0 to T, the model training unit 130 only needs to extract the skeletal features corresponding to each time point from the frames corresponding to each time point. The model training unit 130 then inputs the skeletal features corresponding to each time point into the processing units x(0) to x(T) of the input layer. As a result, output values ​​are output from the processing units z(3) to z(T) of the output layer of the discrimination model 42, and output values ​​are also output from the neuron w(T) of the judgment model 44.

[0056] The model learning unit 130 calculates a first loss (hereinafter also referred to as "loss L1") based on the output values ​​output from the discrimination model 42 corresponding to each frame of the training video and the training data of the type of action. For example, the model learning unit 130 may calculate the loss L1 as the Softmax Cross Entropy error between the output values ​​output from the processing units z(T-m+1)~z(T) of the output layer corresponding to the skeletal features input to the processing units x(0)~x(T) of the input layer and the training data of the type of action attached to each frame.

[0057] Note that m is the number of output layer processing units z used to calculate the loss L1, and may be 100, for example. However, the number of output layer processing units z used to calculate the loss L1 may be any number.

[0058] The model learning unit 130 calculates a second loss (hereinafter also referred to as "loss L2") based on the output value output from the judgment model 44 corresponding to each frame of the training video and the training data for behavioral abnormality. For example, the model learning unit 130 may calculate the loss L2 as the error between the output value output from the neuron w(T) corresponding to the skeletal features input to the processing units x(0) to x(T) of the input layer and the training data for behavioral abnormality. This error may also be the absolute value of the difference between the output value output from the neuron w(T) and the training data for behavioral abnormality.

[0059] The model learning unit 130 learns the generative model 46 based on losses L1 and L2. For example, the model learning unit 130 calculates the loss (hereinafter also referred to as "loss L") by adding losses L1 and L2, and learns the generative model 46 based on the calculated loss L. For example, if the weight of loss L2 is α, the loss L can be calculated as shown in equation (1) below.

[0060] L = L1 + αL2 ... (1)

[0061] However, α may be a predetermined constant. The model learning unit 130 can learn the generative model 46 by updating the weight parameters of the generative model 46 using a backpropagation method based on the loss L calculated in this way. The learning algorithm may be gradient descent. The weight parameters updated by the model learning unit 130 may include both the weight parameters of the discriminative model 42 and the weight parameters of the decision model 44. The model learning unit 130 outputs the learned generative model 46 as the trained model 50 to the inference unit 20.

[0062] (Inference part 20) The inference unit 20 comprises a data unit 210, an identification unit 220, and an output unit 230.

[0063] The identification unit 220 and the output unit 230 include a processor such as a CPU, and their functions can be realized when a program stored in memory (not shown) is loaded into RAM by the processor and executed. In this case, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, the identification unit 220 and the output unit 230 may be composed of dedicated hardware, or they may be composed of a combination of multiple hardware components.

[0064] (Data section 210) The data unit 210 may consist of memory. The data unit 210 stores identification video as an example of identification data used in the inference stage. For example, video obtained by a sensor may be acquired in real time by the inference unit 20 via a network and stored in the data unit 210.

[0065] (Identification unit 220) The identification unit 220 acquires the video image for identification from the data unit 210. Here, we mainly assume that the video image for identification is stored in the data unit 210, and the video image for identification stored in the data unit 210 is acquired by the identification unit 220. However, the video image for identification may also be input to the identification unit 220 from the sensor without going through the data unit 210. The identification unit 220 also acquires the trained model 50 from the model learning unit 130. Based on the video image for identification and the trained model 50, the identification unit 220 identifies the type of human behavior from the video image and determines the degree of abnormality of the human behavior.

[0066] More specifically, the identification unit 220 extracts skeletal features corresponding to each frame from each frame included in the video image for identification using a feature extractor. For example, the feature extractor used by the identification unit 220 may be the same as the feature extractor used by the model learning unit 130. The identification unit 220 then inputs the skeletal features corresponding to each frame into the trained model 50. Based on the input of the skeletal features corresponding to each frame into the trained model 50, an output value is output from the identification model 42 and an output value is output from the judgment model 44.

[0067] For example, if the video for identification includes frames corresponding to each time point from time t=0 to T, the identification unit 220 only needs to extract the skeletal features corresponding to each time point from the frame corresponding to each time point. The identification unit 220 then inputs the skeletal features corresponding to each time point into the processing units x(0) to x(T) of the input layer. As a result, output values ​​are output from the processing units z(3) to z(T) of the output layer of the identification model 42, and output values ​​are also output from the neuron w(T) of the judgment model 44.

[0068] The processing unit z(T) of the output layer of the discrimination model 42 is composed of four neurons, each of which corresponds to a type of action such as "standby," "attach part A," "attach part B," or "attach part C." The discrimination unit 220 identifies the type of action corresponding to the neuron with the highest output value among the output values ​​from the processing unit z(T) as the type of action at time T. The discrimination unit 220 also determines the abnormality of the action at time T based on the output value from neuron w(T).

[0069] (Output section 230) The output unit 230 outputs the type of action at time T. For example, if the identification system 1 has a display in a position where it can be seen by a person, the output unit 230 may output the type of action at time T to the display. This allows the person to recognize the type of action they are taking by looking at the display. For example, if a person is working, the position where they can see the display may be the work environment.

[0070] Furthermore, the output unit 230 outputs the degree of abnormality of the behavior at time T. For example, if the identification system 1 is equipped with a display in a position where it can be seen by a person, the output unit 230 may output the degree of abnormality of the behavior at time T to the display. This allows the person to recognize the degree of abnormality of their own behavior by looking at the display.

[0071] Alternatively, the output unit 230 may compare the degree of abnormality of the behavior at time T with a threshold. If the degree of abnormality of the behavior at time T is greater than the threshold, the output unit 230 may output to the display that the behavior is abnormal. This allows the person to recognize the error in their behavior by looking at the display.

[0072] The above describes an example of the configuration of the identification system 1 according to an embodiment of the present invention.

[0073] (1-2. Operation Description) Next, an example of processing of the identification system 1 according to an embodiment of the present invention will be described. First, an example of processing of the identification system 1 during the learning phase will be described with reference to Figure 5 (and Figures 1 to 4 as appropriate). Processing during the learning phase is mainly performed by the learning unit 10. Next, an example of processing of the identification system 1 during the inference phase will be described with reference to Figure 6 (and Figures 1 to 4 as appropriate). Processing during the inference phase is mainly performed by the inference unit 20.

[0074] (Learning stage) Figure 5 is a flowchart illustrating an example of processing during the learning phase. As shown in Figure 5, the model generation unit 120 obtains the identification model 42 from the data unit 110 (S11). Subsequently, the model generation unit 120 obtains training data for the types of behaviors identified from the training video from the data unit 110. Furthermore, the model generation unit 120 obtains training data for the abnormality of the behaviors identified from the training video.

[0075] Next, the model generation unit 120 generates a generation model 46 by connecting the determination model 44 to the identification model 42 (S12).

[0076] Next, the model learning unit 130 obtains the generated model 46 from the model generation unit 120. Then, the model learning unit 130 generates a trained model 50 using the obtained generated model 46 (S13). More specifically, the model learning unit 130 generates the trained model 50 by training the generated model 46 based on training video footage, training data for behavior types, and training data for behavior anomaly severity.

[0077] The model learning unit 130 outputs the trained generative model 46 as the trained model 50 to the inference unit 20 (S14).

[0078] The above describes an example of the processing of the discrimination system 1 during the learning phase.

[0079] (Inference stage) Figure 6 is a flowchart illustrating an example of processing during the inference stage. As shown in Figure 6, the identification unit 220 obtains the trained model 50 from the model learning unit 130 (S21). The identification unit 220 also obtains video images for identification from the data unit 210. Based on the video images for identification and the trained model 50, the identification unit 220 identifies the type of human behavior from the video images for identification and determines the degree of abnormality of the human behavior (S22).

[0080] The output unit 230 outputs the type of behavior identified by the identification unit 220 and the degree of abnormality of the behavior determined by the identification unit 220 (S23). For example, if the identification system 1 is equipped with a display that can be viewed by a person, the output unit 230 may output the type of behavior and the degree of abnormality of the behavior to the display. This allows a person to recognize the type of behavior and the degree of abnormality of their behavior by viewing the display.

[0081] The above describes an example of the processing of the identification system 1 during the inference stage.

[0082] (2. Hardware Configuration Example) Next, an example of the hardware configuration of the identification system 1 according to an embodiment of the present invention will be described. In the following, an example of the hardware configuration of the information processing device 900 will be described as an example of the hardware configuration of the identification system 1 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the identification system 1. Therefore, the hardware configuration of the identification system 1 may be modified by removing unnecessary components from the hardware configuration of the information processing device 900 described below, or by adding new components.

[0083] Figure 7 shows the hardware configuration of an information processing device 900 as an example of an identification system 1 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0084] The CPU 901 functions as both an arithmetic processing unit and a control unit, controlling the overall operation of the information processing unit 900 according to various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as needed during its execution. These are interconnected by a host bus 904, which consists of a CPU bus and other components.

[0085] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not always necessary to configure the host bus 904, bridge 905, and external bus 906 separately; these functions may be implemented on a single bus.

[0086] The input device 908 consists of input means for the user to input information, such as a mouse, keyboard, touch panel, buttons, microphone, switches, and levers, and an input control circuit that generates input signals based on the user's input and outputs them to the CPU 901. The user operating the information processing device 900 can input various types of data to the information processing device 900 or instruct it to perform processing operations by operating this input device 908.

[0087] The output device 909 includes, for example, display devices such as CRT (Cathode Ray Tube) display devices, liquid crystal display (LCD) devices, OLED (Organic Light Emitting Diode) devices, lamps, and audio output devices such as speakers.

[0088] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded on the storage medium. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). This storage device 910 drives the hard disk and stores programs executed by the CPU 901 and various data.

[0089] The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network. The communication device 911 may support either wireless or wired communication.

[0090] The hardware configuration example of the identification system 1 according to an embodiment of the present invention has been described above.

[0091] (3. Explanation of effects) According to an embodiment of the present invention, a learning unit 10 is provided, comprising a model generation unit 120 and a model learning unit 130. Here, the model generation unit 120 generates a generation model 46 by connecting an identification model 42 for identifying types of behavior to a determination model 44 for determining the degree of abnormality of behavior. The model generation unit 130 then learns the generation model 46 based on training data, first training data which is training data for the types of behavior identified from the training data, and second training data which is training data for the degree of abnormality of behavior identified from the training data.

[0092] With this configuration, since the judgment model 44 is connected to the identification model 42, the identification model 42 has some of the components necessary to determine the degree of abnormality of the behavior, and the judgment model 44 has the rest. As a result, the size of the judgment model 44 is reduced, which reduces the cost of developing the judgment model 44 separately from the development of the identification model 42, and the processing load of the model including the identification model 42 and the judgment model 44 can be reduced.

[0093] (4. Various variations) Although preferred embodiments of the present invention have been described in detail above with reference to the attached drawings, the present invention is not limited to these examples. It is clear to any person with ordinary skill in the art to which the present invention belongs that various modifications or alterations can be conceived within the scope of the technical idea described in the claims, and these are also understood to fall within the technical scope of the present invention.

[0094] (Sensor variations) The above primarily assumes that the training data and classification data are video images obtained by a camera. However, the training data and classification data may also be still images obtained by a camera. Furthermore, a camera is just one example of a sensor, and video and still images are examples of sensor data. Therefore, the training data and classification data may also be sensor data obtained by sensors other than a camera.

[0095] For example, the training data and the identification data may each be sound data obtained by a microphone. Alternatively, the training data and the identification data may each be a three-dimensional point cloud obtained by LiDAR (Light Detection and Ranging), acceleration obtained by an accelerometer, angular velocity obtained by a gyroscope, or electromyography (EMG) obtained by an EMG sensor.

[0096] (Variations in identification processing) The above primarily described examples in which the generative model 46 and the trained model 50 each identify types of human behavior. However, a human being is just one example of an action subject. Therefore, the action subject whose type of behavior is identified may be something other than a human (e.g., a robot or an animal). For example, the robot may be an industrial robot. Furthermore, the action subject whose degree of abnormality is determined may also be changed as appropriate to match the action subject whose type of behavior is identified.

[0097] Furthermore, the actions of the acting entity are an example of time-series data. Therefore, the generative model 46 and the trained model 50 may each identify types of time-series data other than the actions of the acting entity. For example, if the sensor data is a moving or still image, the type of time-series data may be the type of part. In this case, the generative model 46 and the trained model 50 may each determine the degree of abnormality of the item (i.e., the likelihood that the item is damaged).

[0098] Alternatively, if the sensor data is sound data, the type of time-series data may be the type of sound source. In this case, the judgment model 44 may determine the degree of abnormality of the sound source (i.e., the possibility that the sound source is malfunctioning). The sound source can be any object that emits sound. For example, the sound source may be an engine or the like.

[0099] (Variations of input to the model) The above mainly describes the case in which skeletal features extracted from training data are input to the generated model 46 by the model learning unit 130. However, the training data itself may also be input to the generated model 46 by the model learning unit 130. Similarly, the above mainly describes the case in which skeletal features extracted from classification data are input to the trained model 50 by the classification unit 220. However, the classification data itself may also be input to the trained model 50 by the classification unit 220.

[0100] Furthermore, the above mainly described the case where the skeletal features of a person in a video are input as an example of features to the generative model 46 and the trained model 50, respectively. However, the features input to the generative model 46 and the trained model 50 may be features other than the skeletal features of a person in a video.

[0101] For example, the features input to the generative model 46 and the trained model 50 may be features such as motion features of people in the video, information about objects in the video, or depth information of objects in the video. Below, we will explain motion features of people in the video with reference to Figure 8, information about objects in the video with reference to Figure 9, and depth information of objects in the video with reference to Figure 10.

[0102] Figure 8 shows an example of motion features of a person in a video. Referring to Figure 8, the motion feature F2 is shown as an example of motion features of a person in a video. In the example shown in Figure 8, the motion feature F2 includes the motion vectors of each of the six joints present in the person. However, the number of joint motion vectors included in the motion feature F2 is not limited to six.

[0103] The partial feature f21 is the movement vector of the "left wrist" joint, shown as a representative of the movement vectors of the six joints. For example, the movement vector of a joint can be represented by a two-dimensional vector (a combination of the change in the X coordinate and the change in the Y coordinate of the joint) that shows the direction and magnitude of movement of the joint per unit time as captured in the video. Note that the movement vector of a joint is just one example of the movement vector of a part that makes up a person. Therefore, the movement vector of a part other than a joint may be used instead of the movement vector of a joint.

[0104] Figure 9 shows an example of information about objects captured in a video. Referring to Figure 9, object information F3 is shown as an example of information about objects captured in a video. In the example shown in Figure 9, object information F3 includes information about five objects used by a person for work (one rectangular object, two cylindrical objects, one workbench, and one plate-shaped object). However, the number of objects included in object information F3 is not limited to five.

[0105] The partial feature f31 is representative of the information about the five objects, specifically the information about the "plate-like object." For example, information about an object can be represented by the type of object shown in the video (e.g., the tools used in the work). For instance, if a person is performing assembly work, the tools used in the work might be screws for fastening parts.

[0106] Figure 10 shows an example of depth information for an object in a moving image. Referring to Figure 10, depth information F4 is shown as an example of depth information for an object in a moving image. In the example shown in Figure 10, depth information F4 is composed of the depth of each pixel in the depth image, and the depth of each pixel is indicated by the intensity of the color. Specifically, in depth information F4, pixels with a stronger color intensity are indicated as pixels with lower depth.

[0107] The partial feature f41 is the depth of a single pixel, represented as a representative of the depth of each pixel included in the depth image.

[0108] (Processing capacity of identification system 1) The functions of the learning unit 10 and the inference unit 20 have been described above. Here, it can be assumed that the processing load of the recognition unit 20 is smaller than that of the learning unit 10. Therefore, the processing power of the computer required to implement the inference unit 20 may be lower than that of the computer required to implement the learning unit 10. This allows for the efficient use of the processing power of the discrimination system 1. This effect becomes even greater when the trained model 50 generated by the learning unit 10 is used by multiple inference units 20. [Explanation of Symbols]

[0109] 1. Identification System 10. Learning Department 110 Data Section 120 Model Generation Unit 130 Model Learning Department 20 Reasoning section 210 Data Section 220 Identification unit 230 Output section 42 Discriminant Models 44 Decision Models 46 Generative Models 50 pre-trained models

Claims

1. A model generation unit generates a generation model by connecting an identification model for identifying the type of time series data to a determination model for determining the degree of anomaly of the time series data, A model learning unit that learns the generative model based on training data, first training data which is training data for the types of time series data identified from the training data, and second training data which is training data for the anomaly degree of the time series data identified from the training data. A learning device equipped with the following features.

2. The aforementioned time-series data represents the actions of the acting agent. The learning device according to claim 1.

3. The subject of the aforementioned action is a person. The learning device according to claim 2.

4. The aforementioned discrimination model is configured to include a neural network, The aforementioned decision model is composed of neurons, The learning device according to claim 1.

5. The model generation unit connects at least a portion of the intermediate layers included in the discrimination model to the discrimination model, with the neurons being used as the determination model. The learning device according to claim 4.

6. The model learning unit obtains a first output value output from the discrimination model corresponding to the training data and a second output value output from the judgment model corresponding to the training data, and learns the generative model based on a first loss based on the first output value and the first training data, and a second loss based on the second output value and the second training data. The learning device according to claim 1.

7. The model learning unit calculates the loss by adding the first loss and the second loss, and learns the generative model based on the calculated loss. The learning device according to claim 6.

8. The device includes an identification unit that identifies the type of time series data from the identification data and determines the degree of abnormality of the time series data, based on the trained model, which is the generated model after learning by the learning device described in claim 1, and the identification data. Reasoning device.

9. A generative model is generated by connecting an identification model for identifying the type of time series data to a determination model for determining the degree of anomaly of the time series data. The generative model is trained based on training data, first training data which is training data for the types of time series data identified from the training data, and second training data which is training data for the anomaly degree of the time series data identified from the training data. A learning method performed by a computer, including [specific example].

10. Computers, A model generation unit generates a generation model by connecting an identification model for identifying the type of time series data to a determination model for determining the degree of anomaly of the time series data, A model learning unit that learns the generative model based on training data, first training data which is training data for the types of time series data identified from the training data, and second training data which is training data for the anomaly degree of the time series data identified from the training data. A program that makes it function as such.

Images